A split test compares one upsell strategy with a variation or a holdout that sees no offer. Eligible shoppers are assigned consistently, and both groups are measured during the same test window.
Use one clear hypothesis. A test can identify the better complete experience, but if products, discount, copy, design, and targeting all change together it cannot tell you which change caused the result.
Start with an existing Draft or Active strategy.
A strategy already assigned to another split test cannot be selected.
Confirm the app is live and the placement is installed before launching traffic.
Decide the primary question and metric before reviewing results.
Open Apps > Order Editing > Upsell Strategies.
Open the page's More actions menu.
Select Create split test.
Choose the strategy for Control (A). Order Editing copies it for Variant B.
Choose A/B test for an editable Variant B or Holdout test for a group that sees no offer.
Set Variant B traffic from 5% to 95%. The default is 50%.
Choose the Primary Metric.
Choose how a winner should be applied.
Enter a test name and optional hypothesis.
Review the launch checklist and select Create test.
The Create Strategy menu inside the strategy list can also contain Create split test. Use More actions > View split tests to return to all tests later.
For a normal A/B test, Variant B opens in a paused setup state so you can edit and review it. A holdout does not need a second strategy configuration.
Use Ask Sidekick from the split-test area when you want help reducing an idea to one measurable comparison.
For example:
Help me design a split test for my Checkout accessories strategy. Change one variable, write a clear hypothesis, and recommend the primary metric.Review the suggestion yourself. Sidekick does not establish statistical significance or replace the winner guidance on the test page.
Variant B begins as a copy of Control. Good comparisons include:
the same products with different offer text;
the same design with a different product module;
the same product with a different discount;
the strategy against a no-offer holdout.
Control and Variant B must keep compatible placements, markets, and serving status. The editor blocks changes that would make the comparison invalid while the test is running.
Metric | Use it when |
|---|---|
Conversion Rate | The goal is the largest share of visitors accepting an offer. This is the recommended default. |
Revenue Per Visitor | The goal balances acceptance and revenue per exposed shopper. |
AOV Increase | The goal is the average amount an accepted upsell adds to an order. |
Overall Score | The decision should use the supported composite of conversion, revenue, and order-value impact. |
Choose before launch rather than selecting whichever result looks strongest afterward.
Open Advanced settings when you need to change:
Minimum samples per variant: default 100, supported range 50 to 10,000;
Confidence threshold: 90%, 95%, or 99%, with 95% as the default.
Higher thresholds normally take longer. The create page uses recent strategy traffic to estimate the likely duration.
Setting | Result |
|---|---|
Manual | The test keeps running until you choose a winner. |
Auto-apply at significance | Order Editing applies the recommended winner after the configured evidence threshold is reached. |
Auto-apply current leader on a date | The leading group is applied at 9:00 AM in the shop's selected time zone on that date. |
A scheduled winner can be applied before statistical significance. Use a fixed date only when ending on time matters more than waiting for conclusive evidence.
Review Control and Variant B.
Preview each group.
Confirm both use the intended status, placement, and markets.
Select Launch Split Test.
Confirm the test status is Running.
The test page can report views, accepted offers, acceptance rate, revenue, probability to win, sample maturity, device breakdown, and holdout incrementality, depending on available data.
Use its actions to pause or resume assignment, save internal notes, apply a winner, and review completed-test history. Real customer traffic is required; do not create artificial orders simply to reach the sample minimum.
Confirm both groups have useful samples.
Review the primary metric and probability or confidence guidance.
Check other metrics for important trade-offs.
Select Apply Winner.
Choose Control or Variant B and record the learning.
Confirm the winning strategy's status, placement, targeting, products, offer, and design.
Applying a winner completes the test. If automation applied it, review the test history to confirm whether the trigger was significance or the scheduled date.
No strategy is available: create a strategy or finish the other split test using it.
Launch is blocked: complete the checklist, install the placement, turn the app live, and align the two strategy statuses.
No winner guidance: wait for enough real traffic or review whether the minimum sample is realistic.
A scheduled date cannot be saved: choose a future date in the shop time zone shown.
Results look one-sided: confirm both groups remained eligible and active during the same test window.