Ecommerce A/B Testing Statistics on Effect Size and Where Experiments Happen https://takeratehq.com/guides/ecommerce-a-b-testing-statistics-on-effect-size-and-where-experiments-happen/ ## Quick answer In a meta-analysis of 2,732 A/B tests from 252 US ecommerce firms, the median absolute conversion-rate movement was 0.1 percentage points and the mean was 0.7 points. The largest 20% of movements accounted for 81% of the total absolute movement. The analysis discards direction, so these figures describe how strongly variants differed, not the share of tests that produced a winning change (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). A testing program can produce many small movements and a few large ones. A merchant planning a recovery experiment needs to understand the distribution before treating an average as the expected result of the next test. This analysis separates unsigned movement from wins, examines the published intervention categories and identifies what a record must preserve for later decisions. ## Key Takeaways Mean movement was seven times the median The ecommerce A/B-testing study reports a median absolute conversion-rate movement of 0.1 percentage points and a mean of 0.7 points, a 7-to-1 ratio (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). The top 20% accounted for 81% of absolute movement The largest 20% of absolute experiment movements contributed 81% of aggregate absolute movement in the study (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). The three type counts cover 2,225 experiments Published counts of 1,542 design, 415 promotion and 268 shipping experiments total 2,225, leaving 507 outside those counts within the 2,732-test sample (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). Design represented 69.30% of the classified subset Design experiments account for 69.30% of the 2,225 tests in the paper’s three published type categories (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). ## Can a few large changes dominate an experiment program? A team can run several careful experiments and see very little change in conversion. That experience does not establish that experimentation has failed. It may reflect a distribution in which many variants differ only slightly and a smaller group produces much larger movements. Figure description: Ecommerce A/B testing statistics planning sensitivity: at an assumed 2% baseline, detecting a 0.1 percentage point change needs approximately 307,328 visitors per arm versus 6,272 for 0.7 points under stated fixed-analysis assumptions. Figure 1: The mean movement is a much more optimistic traffic-planning input than the median, even before the lost effect direction is considered. Source: Miller and Hosanagar, effect-size inputs (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), source checked Nov 2026. Original analysis of the cited source. Source periods and calculation limits appear in the text. Alex Miller and Kartik Hosanagar examined 2,732 tests from 252 US ecommerce firms using one testing platform. The paper reports a median absolute conversion-rate movement of 0.1 percentage points and a mean of 0.7 points. The mean is seven times the median, which is a useful warning against treating an average experimental effect as the likely outcome of the next test (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). The word absolute is essential. The researchers studied the size of differences after discarding their direction. A large movement could favor either condition. These figures cannot tell a merchant how many tested ideas should be shipped. ## What outcome and sample did the study measure? The inclusion rules selected two-condition experiments that named purchase conversion as their primary outcome. Each arm needed at least 30 observations, with enough converted and non-converted observations to meet the platform’s reporting threshold. The observation period ran from January 2014 to February 2018 (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). The firms varied substantially in size, and many contributed only a few experiments. The mean experiment duration was 42.4 days. That is a description of this dataset, rather than a universal runtime for a checkout test. Traffic, expected effects and the chosen analysis determine what another merchant needs (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). The author’s current project record confirms the 2,732-test, 252-firm sample. A later subsection in the manuscript uses a different test count when discussing p-values. The comparisons here rely on the repeatedly documented inclusion sample and the main descriptive analysis (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). ## Does a large absolute effect mean an experiment won? The largest 20% of absolute movements accounted for 81% of the total. If that total is normalized to 100 units, the top group contributes 81 and the remaining 80% contributes 19. Per experiment, the average movement in the top group is about 17.05 times that of the rest (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). This is a descriptive comparison made after the results are known. It does not identify the large-movement ideas in advance. It also does not mean that 20% of tests create 81% of positive revenue, because the analysis has removed the signs and measures conversion movement rather than revenue (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). A merchant can use the pattern to set realistic expectations. Small movements are common in this source, while a few differences can be much larger. The planning task is to generate and test defensible ideas at a sensible cost, then evaluate the direction, uncertainty and commercial consequences of each result. ## Which kinds of experiments did firms choose? The paper classifies 1,542 experiments as design, 415 as promotion and 268 as shipping. Those counts total 2,225. They leave 507 experiments outside the three published category counts, so a chart using these categories needs to state its subset (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). Within that classified subset, design accounts for 69.30%. The coding rules rely on words in the experiment descriptions and use a stated priority when categories overlap. The category assigned to a test is therefore a description of its apparent intervention, not a verified explanation for its outcome (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). This matters when comparing a testing backlog with the paper. A checkout button experiment and a product-image experiment may both be called design while addressing very different buying problems. Keep the actual shopper question alongside the category label. ## Which part of the buying decision can an intervention change? The paper also examines where an intervention appears in the conversion funnel. Its modeling suggests that intervention type and location interact. A price message encountered while comparing products can do a different job from a shipping offer shown near the final payment step. For checkout recovery, start with the reason the person stopped. A test aimed at an unanswered product question should change the relevant answer or evidence. A test of a shipping offer should preserve the other parts of the interaction closely enough to explain the result. Changing the message, incentive and destination together makes the mechanism harder to identify. The credible alternative is to prioritize by expected contribution and evidence quality rather than by a generic list of high-converting elements. The study does not establish that every design idea is weak or that every late-stage shipping offer is strong. Its categories summarize historical tests across different firms. ## What should the experiment record preserve? Record the primary outcome, the intended shopper problem, the unit randomized, the variants and the conditions that would justify a decision. After the test, retain the signed effect and its uncertainty. An unsigned effect can describe how much the variants differed, but the signed result determines which experience performed better on the measured outcome. Keep order value, margin, incentives and later returns visible where they matter. An increase in purchase conversion may still require a commercial trade-off. The same is true when a recovery interaction changes who buys or which products they choose. A useful program learns from small and unfavorable results as well as large positive ones. The evidence here supports realistic expectations about a skewed distribution. The merchant still needs a reliable decision rule for its own experiments. ## Which reported measures support these findings? The figures below preserve the source’s outcome labels and units. Read comparisons within their stated experiment, population and period. Reported measure | Value and unit | Source Experiments in inclusion sample | 2732 experiments | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Unique firms | 252 firms | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Median absolute conversion movement | 0.1 percentage points | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Mean absolute conversion movement | 0.7 percentage points | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Experiment share contributing most movement | 20 percent of experiments | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Share of aggregate movement | 81 percent | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Mean experiment duration | 42.4 days | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Design-classified experiments | 1542 experiments | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Promotion-classified experiments | 415 experiments | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) Shipping-classified experiments | 268 experiments | (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026) ### How are the additional calculations reproduced? These are arithmetic comparisons of the published inputs. They retain the same sample and limitations as the inputs and do not create an additional experiment. Calculation | Expression | Result and unit Mean-to-median absolute movement ratio | 0.7 / 0.1 | 7.0 ratio Experiments in the three published type categories | 1542 + 415 + 268 | 2225 experiments Experiments outside those category counts | 2732 - 2225 | 507 experiments Design share within the three-category subset | 1542 / 2225 * 100 | 69.3034 % Relative average movement concentration | (81 / 20) / (19 / 80) | 17.0526 ratio ## How much does using the mean change a traffic plan? The two worked effect assumptions produce a 49-fold difference in approximate sample size under a fixed planning model. That follows from the square of the effect in a conventional two-arm planning approximation, not from another finding in the ecommerce sample. For a worked plan, assume a 2% baseline purchase rate, equal independent arms, a two-sided 5% significance threshold, 80% power and one final analysis. Using the baseline binomial variance, the approximate per-arm requirement is 2 × (1.96 + 0.84)² × p × (1 − p) / delta². For an absolute change of 0.1 percentage points, that gives 307,328 eligible visitors per arm. For 0.7 points, it gives 6,272. These assumptions deliberately compare the paper's rounded median and mean movement; they do not predict a positive effect of either size. (Miller and Hosanagar, effect inputs (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), source checked Nov 2026). Assumed baseline | Detect 0.1 point | Detect 0.7 points 2% | 307,328 per arm | 6,272 per arm 3% | 456,288 per arm | 9,312 per arm 5% | 744,800 per arm | 15,200 per arm At 10,000 eligible visitors per day in total, the 2% example implies about 61.5 days versus 1.3 days of enrollment. Those are arithmetic enrollment times, not stopping dates. Full business cycles, delayed purchases, assignment persistence and the planned analysis still matter. A tempting one-day calculation does not make a one-day experiment trustworthy. ### What should the team change when traffic cannot resolve the decision? First define the smallest change that would alter the commercial decision, using margin and implementation cost. Choosing a larger minimum effect only because it makes the sample size convenient leaves smaller valuable effects unresolved. If the worthwhile threshold is below what the available traffic can estimate, the honest result is a feasibility constraint. The team can use a higher-frequency diagnostic outcome to understand the mechanism, pool more eligible traffic where assignment remains valid, or choose a different decision with a larger expected practical effect. A diagnostic outcome still needs a separate link to purchase value. A competing approach is to screen many ideas cheaply, accepting that most results will be inconclusive and reserving confirmation for promising candidates. The paper discusses a highly skewed distribution as motivation for broad exploration. That can be useful as a search policy, but screening and confirmation have different jobs. Selecting the largest noisy estimate and shipping it as a verified winner overstates what the screen learned. ### Where does this approximation fail? The calculations assume independent visitors and use the baseline variance for both conditions. Repeated shoppers, clustered assignment, multiple variants, sequential monitoring, unequal allocation or a material change in outcome variance require a suitable design. The calculation also does not remove uncertainty in the source's rounded 0.1 and 0.7 inputs. Treat the 49-fold ratio as sensitivity to an effect assumption, not a precise forecast of the next test's duration. Download the planning assumptions and arithmetic (https://takeratehq.com/guides/ecommerce-a-b-testing-statistics-on-effect-size-and-where-experiments-happen/data/test-planning-sensitivity.csv) or run the calculation (https://takeratehq.com/guides/ecommerce-a-b-testing-statistics-on-effect-size-and-where-experiments-happen/analysis.py). Keep the signed treatment effect, its interval and the practical decision threshold in the final experiment record. The source's absolute movements can inform expectations about heterogeneity; they cannot supply those missing decision fields. ## Methodology and Sources We use the sample repeatedly stated in the paper’s inclusion section and the author’s current project record, 2,732 tests from 252 firms. A conflicting count in a later p-value subsection is excluded. Effect-size calculations preserve the unsigned definition. Type counts come from the paper’s mutually exclusive keyword classification, and subset percentages use only the 2,225 experiments in those three published categories (Miller and Hosanagar, source checked (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf), Nov 2026). Miller and Hosanagar: Author manuscript; inclusion period January 2014 to February 2018. Citation months identify the cited publication or version. Where a citation explicitly says “source checked”, November 2026 is the verification month, and the original study period remains stated in the text. Source observations, calculations and the proposed merchant applications have distinct meanings. A later p-hacking subsection refers to 2,782 experiments, conflicting with the repeated 2,732 inclusion count. Author’s current project page repeats 2,732. Final study must acknowledge/version-pin the discrepancy and avoid the conflicting subset. The source analyzes unsigned conversion-rate differences. A movement can favor either experimental condition. The categories are mutually exclusive under the stated priority rule. A title classification does not prove why a treatment worked. The counts cover a classified subset of the full sample. ## Glossary Absolute effect: The size of a difference without its positive or negative sign. Median: The middle value in an ordered distribution. Mean: The sum of values divided by their count. Intervention: The change tested against a comparison condition. Conversion rate: The share of eligible observations meeting the purchase definition. ## Sources Miller and Hosanagar: An Empirical Meta-analysis of E-commerce A/B Testing Strategies (https://alexmiller.phd/files/pdf/meta-analysis-ab-testing-ecommerce.pdf) Check the source figures The data file retains source labels, values, populations and limitations for this analysis.