Sample ratio mismatch statistics for checkout A/B tests By Take Rate, Take Rate. Sources checked 2026-11-11. Compare published experiment integrity findings and reproduce a 50/50 sample-ratio check at several sizes before trusting a checkout result. ## Quick answer Microsoft Research reported a sample ratio mismatch in about 6% of its analyzed experiments. A mismatch means the observed counts depart from the configured assignment ratio. In a worked 50/50 example, the same 51/49 split is much stronger evidence of imbalance at 100,000 observations than at 1,000. Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing (https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/) · Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (https://www.microsoft.com/en-us/research/publication/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonomy-and-rules-of-thumb-for-practitioners/) ## Key findings and sources ### How often Microsoft reported a sample ratio mismatch Microsoft Research reported sample ratio mismatch in about 6% of experiments in its 2019 analysis. The finding describes Microsoft’s platform and studied period, with no ecommerce-wide prevalence estimate. Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing (https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/) · Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (https://www.microsoft.com/en-us/research/publication/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonomy-and-rules-of-thumb-for-practitioners/) ### The Microsoft threshold for an allocation check Microsoft Research’s ExP description specifies a chi-square allocation check with a p-value threshold below 0.0005. That threshold is a platform policy, distinct from an outcome significance threshold. Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing (https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/) ### Why sample size changes the interpretation of a 51/49 split For independent 50/50 assignments, a worked split of 510 versus 490 gives chi-square 0.4, while 51,000 versus 49,000 gives chi-square 40. Both have a 51/49 ratio. These are calculated examples using the allocation test Microsoft Research describes. Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing (https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/) ### Outcome loss can reverse a modeled conclusion In Gupchup et al.’s 2019 decoder example, observed freeze duration differed by −0.03%, while two modeled telemetry-loss scenarios produced −2.47% and +2.43%. The example used about 30 million observations per variant and more than 10% video telemetry loss. Gupchup et al., Trustworthy Experimentation Under Telemetry Loss (https://arxiv.org/pdf/1903.12470) ### Where the diagnostic taxonomy came from Fabijan et al.’s 2019 sample-ratio mismatch taxonomy drew on controlled-experiment work at four software companies across more than 25 products. The taxonomy describes multiple possible causes across the experiment lifecycle. Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (https://www.microsoft.com/en-us/research/publication/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonomy-and-rules-of-thumb-for-practitioners/) ### Opposite allocation errors can cancel in the total In Take Rate’s illustrative 50/50 test, weekly counts of 5,200 versus 4,800 and then 4,800 versus 5,200 yield a pooled chi-square of zero and a statistic of 16 in each week. The figures are worked examples using the allocation check described by Microsoft Research. Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing (https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/) ### An 80/20 experiment needs its own allocation expectation In a worked allocation check, 8,000 assignments at 80/20 imply expected counts of 6,400 and 1,600. Observed counts of 6,320 and 1,680 yield a Pearson statistic of 5.0 using those expected counts. Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing (https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/) ### The assigned share and buyer share had different denominators In Hillstrom’s 64,000-customer test, the men’s merchandise email arm contained 33.29% of assignments and 46.19% of observed buyers. The 12.90-point difference compares a pre-outcome assignment share with a post-assignment outcome share. blog.minethatdata.com (https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html) Historical randomized email experiment with a two-week outcome window. Divide 21,307 by 64,000 for assignments, and 267 by the 578 observed buyers for buyer share. ### The equal-split assignment statistic was approximately 0.203 The three published Hillstrom arm sizes were 21,307, 21,387 and 21,306. The Pearson allocation statistic was 0.203 against equal expected counts of 64,000 divided by three. blog.minethatdata.com (https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html) One randomized retail email test released in 2008. Sum squared deviations from equal expected counts divided by the expected count. ## Compare the results Figure description: How two loss assumptions change the same scorecard. The vertical reference mark shows zero. How two loss assumptions change the same scorecard. ## Test the assignment ratio before looking at the lift Microsoft Research reported that about 6% of experiments in its analysis exhibited a sample ratio mismatch. The number describes one organization’s experiments over the period the paper examined. It is useful because it shows why a ratio check belongs in the process even when assignments were randomized by a platform. The ratio check asks whether the counts in the groups match the allocation that was configured for the analysis. It does not say whether conversion rose. Start with assignment at the level that was actually randomized, such as customer or browser identity, and check the expected mix for the rollout window. A split expected to be 90/10 should not be tested as 50/50. Keep assignment, eligibility, exposure and completed orders in separate counts. If all eligible shoppers are assigned 50/50 but a downstream filter creates an imbalance, the timing of the divergence helps narrow the problem. An expected imbalance in voluntary exposure does not necessarily invalidate an all-assigned comparison. It does affect how a post-exposure subset can be interpreted. ## Why an identical 51/49 split can pass or fail For the worked examples, use a two-arm goodness-of-fit statistic. Subtract the expected count from the observed count, square the difference, divide by the expected count, then add the two arms. With 510 versus 490, this gives 0.4. At 5,100 versus 4,900 it gives 4.0. At 51,000 versus 49,000 it gives 40.0. Using one degree of freedom, the approximate p-values are 0.527, 0.0455 and 0.000000000254. Only the last example crosses Microsoft’s published ExP threshold of 0.0005. That is a historical policy used by one platform. The worked examples use independent assignments and a fixed split. Their purpose is to show why a universal percentage tolerance can miss a data problem. Keep the expected counts and denominator in the evidence you hand to the person troubleshooting the experiment. Check whether the imbalance starts at assignment, after a version change, or only after a trigger. Never rescale one group or remove unmatched shoppers solely to make the ratio look right. ## Balanced counts do not prove every outcome was captured The Gupchup et al. paper reports a historical decoder example with around 30 million observations per variant and more than 10% video telemetry loss. The observed scorecard’s freeze-duration difference was −0.03%. Under two stated loss assumptions the modeled values were −2.47% and +2.43%. The sign change is a sensitivity result, not an observed checkout effect. A checkout analogue would be an event that is less likely to arrive after one treatment. The assignment counts might still be balanced. If a new checkout route logs an order only after the thank-you page loads, some completed orders could be absent from the client event. Reconcile a sample against the backend paid-order record and check that both arms have equal opportunity to mature. If the outcome changes when you switch from browser events to validated paid orders, resolve the source difference before calling the lift. The sample ratio check catches one kind of failure. Outcome reconciliation and follow-up completeness catch others. ## Use the check to decide what work is next Look at the first point of disagreement. A problem at assignment calls for checking eligibility, randomization and ramp rules. A balanced assignment with an imbalance only in a triggered analysis calls for checking whether the trigger was available symmetrically. A balanced sample with missing orders calls for a measurement audit. If no trustworthy comparison remains, rerun after fixing the measurement. If the cause is isolated to a known period or platform, record the exclusion and why it was decided before using the surviving sample. Report the full planned split, observed counts, the event that created the sample, and the endpoint and horizon used to compare purchases. ## An overall 50/50 ratio can hide opposite imbalances Imagine a test configured for 50/50 assignment in both weeks. The first week records 5,200 shoppers in A and 4,800 in B. The second records 4,800 in A and 5,200 in B. In the aggregate, the test has 10,000 in each arm and a sample-ratio statistic of zero. The statistic is 16 within each week, corresponding to a one-degree-of-freedom p-value of about 0.000063. The numbers are illustrative. They show why a total ratio can conceal a problem that changes direction after a rollout, logging update or eligibility edit. A different planned split would change the expected counts and could make the same split entirely intentional. Keep the planned allocation and first observed assignment counts by rollout period. Compare agreed segments when diagnosing a problem, and account for the number of checks if you plan to use every slice as a formal decision. A useful outcome is knowing when the problem started. Once you find that boundary, trace the assignment and event records before choosing an analysis window. If assignment remains balanced but paid orders differ between the event log and the backend, investigate the outcome pipeline separately. ## Use the planned split when the holdout is smaller A small holdout is a sensible design when the business wants most eligible customers to receive a treatment. The allocation check must use the split that was specified before launch. If the system changes the planned percentage during the test, reconcile the periods against the expectations that applied at the time. In an 80/20 test with 8,000 eligible assignments, expect 6,400 treatment and 1,600 holdout assignments. If the readout contains 6,320 and 1,680, the Pearson statistic is 5.0. The smaller arm contributes four of those five points because 80 extra assignments are a larger proportional change in that arm. This is a worked case. Whether it triggers an alert depends on the prespecified diagnostic threshold. Check the counts where customers enter the randomization and again where outcomes join the scorecard. If the intended split exists upstream and drifts after the join, examine deduplication, missing events and eligibility filters. If the split is wrong upstream, inspect the assignment rule and launch timeline before reading the conversion result. ## Filtering to buyers can create the appearance of a broken split Hillstrom’s historical three-arm experiment assigned 21,307 of 64,000 customers to the men’s merchandise email, or 33.29% of everyone assigned. That arm contained 267 of the 578 recorded buyers, or 46.19%. The 12.90-point difference appears when you move from all assignments to customers who purchased. Purchases happen after the intervention and can respond to it. An SRM check on purchases would confuse a real difference in the outcome with an assignment defect. An allocation check on 21,307, 21,387 and 21,306 assignments against an equal three-way split gives a Pearson statistic of about 0.203. That diagnostic examines the correct unit. A separate outcome reconciliation asks whether purchases recorded elsewhere were attached to those same assignments. ## Reconcile the scorecard join before diagnosing the randomizer Keep a table keyed by assigned customer with one record per assignment. Add outcome records to that table without dropping people whose purchase count is zero. Then compare how many assigned records survived in each arm. If the denominator differs from the assignment log, check for eligibility applied after assignment, joined rows missing a customer key, duplicate rows, or product flags that are only present after someone buys. In a channel test, distinguish outcome events missing from the data system from assigned customers with no observed event. One can be a true zero and the other can be unknown. If the two arms use different ways of observing the event, build a small reconciliation against the same order system for both before using the scorecard. The assignment ratio is only one check on that pipeline. ## Study data Table: Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing Measure | Value | Context Microsoft experiments exhibiting sample ratio mismatch | About 6% | Empirical flag rate on Microsoft’s platform Microsoft test threshold for the allocation check | 0.0005 p-value | Platform threshold, not a universal threshold Table: Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments Measure | Value | Context Companies that informed the diagnostic taxonomy | 4 companies | Scope of methods and case reports Software products that informed the diagnostic taxonomy | Over 25 products | Scope of methods and case reports Table: Gupchup et al., Trustworthy Experimentation Under Telemetry Loss Measure | Value | Context Observations per arm used for the video example | About 30 million | Approximate observations per variant Video telemetry lost in the decoder experiment | More than 10% | Observed data loss, not the share of customers lost Video freeze duration, observed scorecard | -0.03% | Observed relative difference Video freeze duration, loss scenario A | -2.47% | Simulated relative difference under a stated loss assumption Video freeze duration, loss scenario B | 2.43% | Simulated relative difference under a stated loss assumption Video duration, observed scorecard | 0.02% | Observed relative difference Video duration, loss scenario A | 0.11% | Simulated relative difference under a stated loss assumption Video duration, loss scenario B | 0.09% | Simulated relative difference under a stated loss assumption Mean video bitrate, observed scorecard | -0.04% | Observed relative difference Mean video bitrate, loss scenario A | -0.54% | Simulated relative difference under a stated loss assumption Mean video bitrate, loss scenario B | -0.68% | Simulated relative difference under a stated loss assumption Loss-treatment interaction in the first worked sensitivity scenario | -5% | Explicit sensitivity input, not an observed loss rate Loss-treatment interaction in the second worked sensitivity scenario | 5% | Explicit sensitivity input, not an observed loss rate ## What changes when you compare the figures Table: Calculations from the published values above. Comparison | Result | Calculation Video freeze duration sensitivity range | 4.9 percentage points | = 2.43 − (-2.47) Distance from observed freeze result to the second loss scenario | 2.46 percentage points | = 2.43 − (-0.03) Hypothetical 51/49 check at 1,000 observations | 0.4 chi-square | = (510 - 500) * (510 - 500) / 500 + (490 - 500) * (490 - 500) / 500 Hypothetical 51/49 check at 10,000 observations | 4 chi-square | = (5100 - 5000) * (5100 - 5000) / 5000 + (4900 - 5000) * (4900 - 5000) / 5000 Hypothetical 51/49 check at 100,000 observations | 40 chi-square | = (51000 - 50000) * (51000 - 50000) / 50000 + (49000 - 50000) * (49000 - 50000) / 50000 Hypothetical combined allocation statistic across two balanced totals | 0 chi-square | = 0 + 0 Hypothetical first period allocation statistic for 5,200 versus 4,800 | 16 chi-square | = (5200 - 5000) * (5200 - 5000) / 5000 + (4800 - 5000) * (4800 - 5000) / 5000 Hypothetical second period allocation statistic after reversing the groups | 16 chi-square | = (4800 - 5000) * (4800 - 5000) / 5000 + (5200 - 5000) * (5200 - 5000) / 5000 Worked example, expected treatment count in an 80/20 test | 6,400 assignments | = 8000 * 0.8 Worked example, expected holdout count in an 80/20 test | 1,600 assignments | = 8000 * 0.2 Worked example, Pearson statistic for an 80/20 assignment | 5 chi-square statistic | = (6320 - 6400)**2 / 6400 + (1680 - 1600)**2 / 1600 Men’s merchandise email share of assignments | 33.292% | = 21307 / 64000 * 100 Men’s merchandise email share of buyers | 46.194% | = 267 / (267 + 189 + 122) * 100 Assignment share compared with buyer share | 12.902 percentage points | = 267 / 578 * 100 - 21307 / 64000 * 100 Three-arm assignment Pearson statistic | 0.203 chi-square statistic | = ((21307 - 64000 / 3)**2 + (21387 - 64000 / 3)**2 + (21306 - 64000 / 3)**2) / (64000 / 3) ## Methodology and sources The observed sample-ratio figure and testing policy come from Microsoft Research. The case taxonomy scope comes from the author publication abstract. The signed telemetry values are transcribed from Table 1 of Gupchup et al., with the observed scorecard and the two sensitivity scenarios kept distinct. The separate worked cases use expected counts of half the total, the two-cell Pearson statistic and the one-degree-of-freedom chi-square survival function, erfc of the square root of half the statistic. No worked case is a merchant result. [Download the source data (CSV)](decision-table.csv) [Download the calculations (CSV)](calculations.csv) [Download the citation notes (CSV)](citation-notes.csv) [Download the worked allocation checks](sample-ratio-worked-cases.csv) [Download the evidence behind the new findings (CSV)](finding-evidence.csv) [Download the analysis script for these comparisons](holdout_sensitivity.py) ## Related reading Incremental advertising return statistics and what a holdout can tell you (https://takeratehq.com/guides/incremental-advertising-return-statistics/) A/B test peeking statistics for checkout experiments (https://takeratehq.com/guides/ab-test-peeking-statistics/) Email holdout statistics for visits, purchases and revenue (https://takeratehq.com/guides/email-holdout-purchase-and-revenue-statistics/) When refunds turn an iMessage recovery win into a loss (https://takeratehq.com/guides/refund-adjusted-recovery-revenue/) Can you trust the lift if your iMessage holdout got the extra text? (https://takeratehq.com/guides/imessage-recovery-holdout-exposure/) Browse checkout recovery research (https://takeratehq.com/research/) ## Sources and definitions Microsoft Research, Diagnosing Sample Ratio Mismatch in A/B Testing (https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/) Fabijan et al., Diagnosing Sample Ratio Mismatch in Online Controlled Experiments (https://www.microsoft.com/en-us/research/publication/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonomy-and-rules-of-thumb-for-practitioners/) Gupchup et al., Trustworthy Experimentation Under Telemetry Loss (https://arxiv.org/pdf/1903.12470) The source data includes the population, period, denominator and limitations for every figure. Take Rate · takeratehq.com