Onboarding · Dashboard Guide

Experiments (A/B Testing)

Test two versions of an offer — price, content, or timing — and let data decide which one your players prefer. No guessing. No opinion debates.

What is an A/B experiment?

An A/B experiment splits your players into two groups. Group A (control) sees the current offer. Group B (variant) sees your proposed change. After enough data, gamemantra uses a statistical test to tell you which version earns more — with a confidence score so you know the result isn't random luck.

🅰️

Control

Your current offer — the baseline everyone compares against

🅱️

Variant

Your proposed change — lower price, different items, new copy

How experiments work end-to-end

✍️

You define the hypothesis

Start with a clear question: "Will a $2.99 price point convert better than $4.99 for new players in the puzzle genre?"

🎲

Players are randomly split

gamemantra assigns players to A or B using a deterministic hash of their player_id — the same player always sees the same variant, every session.

📈

Data collects automatically

Every offer impression, click, and purchase is recorded. You need at least 1,000 players in each group before results are reliable.

🔬

Statistical test runs

gamemantra uses the Mann-Whitney U test (built for non-normal IAP data). A p-value below 0.05 means the winner is statistically real.

🏆

You promote the winner

Click "Promote Variant" in the dashboard. The winning offer config replaces the control for 100% of players.

How to create an experiment

Go to Analytics → Experiments in the sidebar.

Click "New Experiment" and name it

A modal opens. Give your experiment a clear name that describes the hypothesis, e.g."Price test: $2.99 vs $4.99 — Puzzle genre newcomers". Add your hypothesis in plain text — this becomes your success criterion.

Choose a trigger event

Pick the in-game moment where the offer fires. This ties the experiment to the offer pipeline — only players who encounter this event are enrolled. Available events:

session_start

App open

first_launch

First install

level_complete

Stage cleared

first_loss

First defeat

first_win

First victory

currency_low

Balance dips

session_end

App closes

retry

Retry attempt

pause

Pause screen

Tip: enable the trigger in Settings → Offer Triggers first — if the event is disabled there, no offers (and no experiment enrollments) will fire for it.

Set the traffic split

Use the Variant B Split slider (10–50%). Group A (control) sees the default AI-selected offer. Group B (variant) sees your override. The same player always sees the same group — gamemantra uses a deterministic hash of the player ID.

Use 20% for high-risk tests (large price increase). Use 50% for standard tests where you want results faster.

Configure the Variant B offer override

In the Variant B — Offer Override section of the modal, set:

Price USD — the price variant B players see (e.g. $2.99 vs the default $4.99)
Offer Type — bundle, limited_time, or standard. Keep the same as control if testing price only.
Template Name — optional. Leave blank to use the AI-selected template.

Change one variable at a time: if you change both price and offer type, you cannot know which caused the result.

Start the experiment and watch the dashboard

Click Create Experiment — it starts in Draft status. Click the ▶ Play button on the experiment card to set it to Running. The AI pipeline will now route players through A/B groups when the trigger fires.

Watch the detail panel for Days Live, Sample Size,ARPU Treatment vs Control, and p-value. Do not stop early — wait for p-value < 0.05 before drawing conclusions.

Mark complete and promote the winner

Click Complete to end the experiment. If the variant won (p < 0.05 and positive ARPU uplift), update your game's offer configuration to use the winning parameters. The experiment results stay visible for reference — they are never deleted.

Prerequisites — what you need before running an experiment

An experiment sits at the end of the offer pipeline. The pipeline needs item catalogue, currencies, and trigger rules in place first — otherwise the experiment variant has nothing to build an offer from.

✅

Item catalogue

At least one item with fallback_eligible=true so L4 has a product to offer

✅

Currencies with BCU rates

BCU rates let the ARPU comparison normalise across multi-currency bundles

✅

Trigger rule enabled

The event_type you pick for the experiment must be enabled in Settings → Offer Triggers

⚡

Live Ops event (optional but recommended)

Run the experiment during a live event for higher volume and amplified urgency

🔬 Full pipeline flow — item → currency → liveops → experiment → offer

1. Player triggers event

SDK fires gm_get_next_offer(player_id, "boss_raid", &result)

2. Live event check

Active Live Ops event with offer_trigger=boss_raid bypasses trigger_rules table

3. Experiment routing

FNV32a hash of (experiment_id:player_id) → bucket 0–99 → Group A or Group B

4. Group A — control offer

AI builds offer from player:features (spend_score, segment, offer_type) + L1 ONNX

5. Group B — variant override

variant_config overrides price_usd=$2.99, offer_type=bundle, template=dragon_raid_bundle

6. Item catalogue (L2/L3/L4)

If L1 (AI) returns nil → offer_templates → studio_bundles → store_items (fallback_eligible)

7. OPA compliance gate

OPA checks jurisdiction rules, price guardrails, daily offer cap — always last

8. Offer shown + telemetry

offer_shows row created with experiment_id + experiment_group columns

9. Celery: compute_experiment_results

Every 6h: sums price_cents WHERE purchased_at IS NOT NULL, computes ARPU + Mann-Whitney U

⚠️ Only change one variable at a time

The most common mistake: testing price AND bundle content simultaneously. If variant B has a lower price AND more items, you cannot know which change drove the result. Run one test per variable. Price first, then content.

Reading the experiment results

🎯

Conversion Rate (CVR)

% of players who bought the offer. Higher is better. A 2× CVR difference between variants is typically significant.

💰

ARPU Uplift

Average Revenue Per User. The holdout comparison tells you how much extra revenue the AI is generating vs. no offers at all.

📊

p-value

Statistical confidence. p < 0.05 means the result is unlikely to be random. gamemantra uses Mann-Whitney U (not t-test) for IAP data accuracy.

⚖️

FSR (Flow-to-Spend Ratio)

Are players spending currency at a healthy rate? A balanced FSR (genre-specific) means the economy is stable.

Experiment status badges explained

Running

Data is being collected. Do not draw conclusions yet.

Significant

p < 0.05 and sample size met. A winner can be declared.

Inconclusive

Not enough data or p ≥ 0.05. Run longer or increase traffic.

Completed

Winner promoted or experiment stopped manually.

Experiment ideas to try

Hypothesis	Control	Variant	Outcome
Does a 20% price cut increase revenue?	`$4.99 bundle`	`$3.99 bundle`	CVR rose 40%, net revenue up 12% — variant wins
Do 3-item bundles outperform 5-item?	`5 items (complex)`	`3 items (simple)`	CVR +18% for simpler bundle — cognitive load matters
Does urgency copy help?	`"Buy Now"`	`"Only 4 left"`	p=0.08 — inconclusive, need more data

💡 AI prompt you can use

"I have an A/B experiment running for 12 days. Control ($4.99 bundle) has CVR 3.2%, Variant ($2.99 bundle) has CVR 5.1%. p-value is 0.03. Sample: 800 control, 820 variant. Should I promote the variant? What are the revenue implications?"