--- title: "Running an A/B Test for an Upsell Strategy" description: "Compare a control with a strategy variant or holdout, choose a primary metric, and apply a winner using measured results." url: https://docs.orderediting.com/en/articles/15323714-running-an-ab-test-for-an-upsell-strategy collection: "Strategies and Configuration" updated: 2026-11-05 language: en --- # Running an A/B Test for an Upsell Strategy Compare a control with a strategy variant or holdout, choose a primary metric, and apply a winner using measured results. # What a split test does A split test compares one upsell strategy with a variation or a holdout that sees no offer. Eligible shoppers are assigned consistently, and both groups are measured during the same test window. Use one clear hypothesis. A test can identify the better complete experience, but if products, discount, copy, design, and targeting all change together it cannot tell you which change caused the result. # Before you begin - Start with an existing Draft or Active strategy. - A strategy already assigned to another split test cannot be selected. - Confirm the app is live and the placement is installed before launching traffic. - Decide the primary question and metric before reviewing results. # Create the test 1. Open **Apps > Order Editing > Upsells** and open the live strategy you want to test. 2. Select **Create A/B test** in the title bar. Before a test, the strategy's **Summary** card also shows an **A/B test** row with the same link. 3. Confirm the strategy for **Control (A)**. Order Editing copies it for Variant B. 4. You can also start from **More actions > Split tests > Create split test**, or tick a strategy in the list and select **Create split test**. 5. Choose **A/B test** for an editable Variant B or **Holdout test** for a group that sees no offer. 6. Set **Variant B traffic** from 5% to 95%. The default is 50%. 7. Choose the **Primary Metric**. 8. Choose how a winner should be applied. 9. Enter a test name and optional hypothesis. 10. Review the launch checklist and select **Create test**. During a test, the strategy's title bar shows **View A/B test**. Use **More actions > Split tests** to see all tests later. **Create A/B test** stays off on a draft strategy until it's live. For a normal A/B test, Variant B opens in a paused setup state so you can edit and review it. A holdout does not need a second strategy configuration. # Ask Sidekick to shape the hypothesis Use **Ask Sidekick** from the split-test area when you want help reducing an idea to one measurable comparison. For example: ``` Help me design a split test for my Checkout accessories strategy. Change one variable, write a clear hypothesis, and recommend the primary metric. ``` Review the suggestion yourself. Sidekick does not establish statistical significance or replace the winner guidance on the test page. # Configure Variant B Variant B begins as a copy of Control. Good comparisons include: - the same products with different offer text; - the same design with a different product module; - the same product with a different discount; - the strategy against a no-offer holdout. Control and Variant B must keep compatible placements, markets, and serving status. The editor blocks changes that would make the comparison invalid while the test is running. # Choose the primary metric

Metric

Use it when

Conversion Rate

The goal is the largest share of visitors accepting an offer. This is the recommended default.

Revenue Per Visitor

The goal balances acceptance and revenue per exposed shopper.

AOV Increase

The goal is the average amount an accepted upsell adds to an order.

Overall Score

The decision should use the supported composite of conversion, revenue, and order-value impact.

Choose before launch rather than selecting whichever result looks strongest afterward. # Metrics for content-only checkout tests For a content-only checkout comparison, the primary metric choices describe the whole checkout: **Checkout completion**, **Revenue Per Session** and **Average Order Value**. **Overall Score (Composite)** combines the supported checkout outcomes. These are different from accepted-offer conversion, revenue per visitor and upsell order-value increase. Use the labels on your test and compare both groups over the same test window. # Configure evidence settings Open **Advanced settings** when you need to change: - **Minimum samples per variant**: default 100, supported range 50 to 10,000; - **Confidence threshold**: 90%, 95%, or 99%, with 95% as the default. Higher thresholds normally take longer. The create page uses recent strategy traffic to estimate the likely duration. # Choose winner automation

Setting

Result

Manual

The test keeps running until you choose a winner.

Auto-apply at significance

Order Editing applies the recommended winner after the configured evidence threshold is reached.

Auto-apply current leader on a date

The leading group is applied at 9:00 AM in the shop's selected time zone on that date.

A scheduled winner can be applied before statistical significance. Use a fixed date only when ending on time matters more than waiting for conclusive evidence. # Launch and monitor 1. Review Control and Variant B. 2. Preview each group. 3. Confirm both use the intended status, placement, and markets. 4. Select **Launch Split Test**. 5. Confirm the test status is Running. The test page can report views, accepted offers, acceptance rate, revenue, probability to win, sample maturity, device breakdown, and holdout incrementality, depending on available data. Use its actions to pause or resume assignment, save internal notes, apply a winner, and review completed-test history. Real customer traffic is required; do not create artificial orders simply to reach the sample minimum. # Change the traffic split after launch The traffic allocation is chosen during setup. You cannot rebalance a launched test by pausing it. If you need a different allocation, stop the current comparison using the test's completion or cancellation action and create a new test with the intended split. Keep the earlier test's results as history; do not combine the two tests as though they shared one unchanged allocation. # Apply a winner 1. Confirm both groups have useful samples. 2. Review the primary metric and probability or confidence guidance. 3. Check other metrics for important trade-offs. 4. Select **Apply Winner**. 5. Choose Control or Variant B and record the learning. 6. Confirm the winning strategy's status, placement, targeting, products, offer, and design. Applying a winner completes the test. If automation applied it, review the test history to confirm whether the trigger was significance or the scheduled date. # Troubleshooting - **No strategy is available:** create a strategy or finish the other split test using it. - **Launch is blocked:** complete the checklist, install the placement, turn the app live, and align the two strategy statuses. - **No winner guidance:** wait for enough real traffic or review whether the minimum sample is realistic. - **A scheduled date cannot be saved:** choose a future date in the shop time zone shown. - **Results look one-sided:** confirm both groups remained eligible and active during the same test window. ## Related - [How to Customize Your One-Click Upsell Page](https://docs.orderediting.com/en/articles/15323698-how-to-customize-your-one-click-upsell-page.md) - [Upsell Customer Targeting](https://docs.orderediting.com/en/articles/14466316-upsell-customer-targeting.md) - [Understanding Upsell Strategy Matching](https://docs.orderediting.com/en/articles/14391612-understanding-upsell-strategy-matching.md) - [Creating an Upsell Strategy](https://docs.orderediting.com/en/articles/14466314-creating-an-upsell-strategy.md) - [Configuring Upsell Settings](https://docs.orderediting.com/en/articles/14466307-configuring-upsell-settings.md)