Summarize these selected article findings and their source context. This brief is not the full article. Preserve units, populations, dates, assumptions and limitations. Distinguish illustrative calculations, vendor descriptions and measured results. Do not invent omitted information or claim to have read the complete article. Sample ratio mismatch statistics for checkout A/B tests Selected article findings, not the complete article. Microsoft Research reported a sample ratio mismatch in about 6% of its analyzed experiments. A mismatch means the observed counts depart from the configured assignment ratio. In a worked 50/50 example, the same 51/49 split is much stronger evidence of imbalance at 100,000 observations than at 1,000. How often Microsoft reported a sample ratio mismatch Microsoft Research reported sample ratio mismatch in about 6% of experiments in its 2019 analysis. The finding describes Microsoft’s platform and studied period, with no ecommerce-wide prevalence estimate. https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/ https://www.microsoft.com/en-us/research/publication/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonomy-and-rules-of-thumb-for-practitioners/ The Microsoft threshold for an allocation check Microsoft Research’s ExP description specifies a chi-square allocation check with a p-value threshold below 0.0005. That threshold is a platform policy, distinct from an outcome significance threshold. https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/ Why sample size changes the interpretation of a 51/49 split For independent 50/50 assignments, a worked split of 510 versus 490 gives chi-square 0.4, while 51,000 versus 49,000 gives chi-square 40. Both have a 51/49 ratio. These are calculated examples using the allocation test Microsoft Research describes. https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/ Outcome loss can reverse a modeled conclusion In Gupchup et al.’s 2019 decoder example, observed freeze duration differed by −0.03%, while two modeled telemetry-loss scenarios produced −2.47% and +2.43%. The example used about 30 million observations per variant and more than 10% video telemetry loss. https://arxiv.org/pdf/1903.12470 Where the diagnostic taxonomy came from Fabijan et al.’s 2019 sample-ratio mismatch taxonomy drew on controlled-experiment work at four software companies across more than 25 products. The taxonomy describes multiple possible causes across the experiment lifecycle. https://www.microsoft.com/en-us/research/publication/diagnosing-sample-ratio-mismatch-in-online-controlled-experiments-a-taxonomy-and-rules-of-thumb-for-practitioners/ Opposite allocation errors can cancel in the total In Take Rate’s illustrative 50/50 test, weekly counts of 5,200 versus 4,800 and then 4,800 versus 5,200 yield a pooled chi-square of zero and a statistic of 16 in each week. The figures are worked examples using the allocation check described by Microsoft Research. https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/ An 80/20 experiment needs its own allocation expectation In a worked allocation check, 8,000 assignments at 80/20 imply expected counts of 6,400 and 1,600. Observed counts of 6,320 and 1,680 yield a Pearson statistic of 5.0 using those expected counts. https://www.microsoft.com/en-us/research/articles/diagnosing-sample-ratio-mismatch-in-a-b-testing/ The assigned share and buyer share had different denominators In Hillstrom’s 64,000-customer test, the men’s merchandise email arm contained 33.29% of assignments and 46.19% of observed buyers. The 12.90-point difference compares a pre-outcome assignment share with a post-assignment outcome share. https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html The equal-split assignment statistic was approximately 0.203 The three published Hillstrom arm sizes were 21,307, 21,387 and 21,306. The Pearson allocation statistic was 0.203 against equal expected counts of 64,000 divided by three. https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html