Take Rate Take Rate PERFORMANCE RECOVERY

Incremental advertising return statistics and what a holdout can tell you

In 25 randomized ad experiments, the median ROI interval exceeded 100 percentage points. Learn how to read an uncertain checkout recovery result.

The scale and uncertainty of the experiments

25 experiments
Randomized field experiments
More than 100 percentage points
Median ROI confidence interval width
The 25 experiments included 19 retail and six financial-services campaigns. Their published median ROI interval exceeded 100 percentage points. The retail subset's median standard error is a separate figure.
About 17 min read
Updated:
Statistics
Learn about our Editorial policyLearn about our Research methodology

Sources checked 11 November 2026.

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Full article, downloads and prompt

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Download articleDownload AI source digest

Copy the complete article or download it for later.

Contents

Quick answer

Across 14 advertising experiments with a checkout outcome, Gordon and colleagues found seven where observational lift estimates were consistently more than a factor of three off the randomized result. Lewis and Rao separately found a median ROI confidence interval over 100 percentage points wide in 25 advertising experiments. A checkout holdout needs both a fair comparison and enough precision to tell whether the program pays for itself.

Gordon et al., A Comparison of Approaches to Advertising Measurement · Lewis and Rao, The Unfavorable Economics of Measuring the Returns to Advertising

Key findings and sources

How often the observational estimate missed the checkout experiment

Gordon et al. compared observational estimates with randomized results in 15 US Facebook ad experiments. In seven of the 14 studies with a checkout conversion outcome, observational point estimates were consistently off by more than a factor of three. This describes the methods and campaigns they evaluated.

Gordon et al., A Comparison of Approaches to Advertising Measurement

One campaign with a close and a distant observational estimate

In Gordon et al.’s Study 4, the randomized checkout lift estimate was 73%, compared with 316% from an exposed-versus-unexposed comparison and 74% from the richest reported stratified regression. These are lift estimates for the same campaign, not checkout conversion rates.

Gordon et al., A Comparison of Approaches to Advertising Measurement

Why a near-zero experiment can disagree sharply with modeled lift

In Gordon et al.’s Study 9, randomized checkout lift was estimated at 2.4%, while the closest displayed observational estimate was 1,306%. The 2.4% result was reported as different from zero at the 10% level, and the finding is campaign-specific.

Gordon et al., A Comparison of Approaches to Advertising Measurement

How wide advertising ROI uncertainty can be

Lewis and Rao’s 2015 analysis of 25 randomized advertising experiments reported a median ROI interval over 100 percentage points wide. The median ROI standard error was 26.1 percentage points in retail experiments and 115 percentage points in brokerage experiments.

Lewis and Rao, The Unfavorable Economics of Measuring the Returns to Advertising

Advertising experiment size depends on the decision

In Lewis and Rao’s median-campaign calculation, distinguishing 50% ROI from break-even required about nine times the experiment size discussed at baseline. That multiple depends on the authors’ study economics and uncertainty, and is not a sample size rule for checkout recovery.

Lewis and Rao, The Unfavorable Economics of Measuring the Returns to Advertising

A narrower ROI decision can require far more data

In Lewis and Rao’s 2015 study, their idealized calculation put the median campaign at roughly nine times its original size to distinguish 50% ROI from break-even and 62 times for a 10% ROI difference. The comparison assumes additional independent person-weeks and is specific to those advertising economics.

Lewis and Rao, The Unfavorable Economics of Measuring the Returns to Advertising

Why large experiments can remain noisy

Lewis and Rao reported that each of their 25 advertising field experiments exceeded 500,000 unique users and most exceeded one million. Individual-level sales standard deviation was typically about ten times the mean over the evaluation window.

Lewis and Rao, The Unfavorable Economics of Measuring the Returns to Advertising

Break-even can move while the measured order gap stays fixed

In Take Rate’s worked example, $400 of treatment cost across 2,000 assigned shoppers breaks even at a 0.50-point order-rate gap with $40 contribution per additional order. At $24 contribution it takes approximately 0.83 points. These are illustrative calculations, separate from Lewis and Rao’s published experiments.

Lewis and Rao, The Unfavorable Economics of Measuring the Returns to Advertising

Four times the sample halves standard error under a stable variance model

Taking Lewis and Rao’s reported 26.1-point median retail ROI standard error as an illustrative starting value, four times the independent observations would reduce it to 13.05 points if variance per observation stayed constant.

Lewis and Rao, The Unfavorable Economics of Measuring the Returns to Advertising

Stratified and randomized estimates disagreed on sign in three checkout studies

In Gordon et al. Table 6, three of 14 checkout comparisons (21.43%) had opposite signs for the stratified observational and randomized lift point estimates. The affected rows were Studies 8, 10 and 15.

Gordon et al., A Comparison of Approaches to Advertising Measurement

Checkout comparisons in the authors’ 2019 experimental advertising paper. Count study rows with different signs and divide by the 14 checkout rows.

Study 8 retained a 35.9-point gap after stratification

In Gordon et al. Study 8, estimated checkout lift was +179% for exposed versus unexposed users, +33% after stratification and −2.9% in the randomized estimate. The latter two point estimates differed by 35.9 points.

Gordon et al., A Comparison of Approaches to Advertising Measurement

Study 8 checkout estimates from three methods in the same paper. Subtract the randomized estimate from the stratified estimate.

Compare the results

Four lift estimates for one checkout outcome. Exact values are also in the study data.
Four lift estimates for one checkout outcome.

A larger dataset does not make two groups comparable

Gordon and colleagues studied 15 US Facebook advertising experiments comprising 500 million user-experiment observations and 1.6 billion ad impressions. They compared observational estimates with results from randomized designs. In seven of the 14 studies with a checkout outcome, the observational point estimates were consistently more than a factor of three off the randomized result. This is evidence about those studied methods and campaigns.

In Study 4, an exposed-versus-unexposed comparison suggested 316% checkout lift. Matching on age and gender brought it to 222%, while the experiment estimated 73%. The richest reported stratified regression was much closer at 74%. That is an important counterexample. Richer adjustment can work in a specific campaign. The same model choice cannot be assumed to work next time.

Study 9 shows the other side of that result. The randomized estimate was 2.4%, while the nearest observational estimate the authors reported was 1,306%. Only 6.6% were exposed in that study. The authors proposed this as one reason targeting could concentrate on unusually likely responders. They also describe Study 15, where almost every observational method underestimated lift. The correction is a more defensible comparison group, not a fixed discount applied to every attributed revenue claim.

Start with the decision threshold in orders or contribution

A merchant testing checkout recovery can set the threshold before reading the outcome. Suppose 2,000 eligible shoppers enter each arm, the control has 80 valid orders and the treatment has 100. That is 4% versus 5%, a gap of one percentage point. At a fixed $40 contribution per extra order and a $400 cost for treatment at that scale, the point estimate is $400 in additional contribution after cost.

The break-even threshold under those assumptions is 10 additional valid orders, or 0.50 percentage points over 2,000 treated shoppers. With independent customer-level assignments and a simple normal approximation, a 95% interval for the observed difference is approximately −0.29 to +2.29 percentage points. The point estimate clears the threshold while the interval spans it. The experiment leaves both profitable and unprofitable outcomes plausible.

If refunds, discounts and product mix matter, define contribution after those deductions before making the decision. The same extra-order count can have different economics depending on which products sold and which customers would have paid full price. Track contribution directly if the margin assumption is material. The worked numbers are hypothetical, and the arithmetic makes the decision rule visible.

The same point estimate can lead to a different decision

Keep the hypothetical control and treatment rates at 4% and 5%. With 10,000 independent shoppers per group, the approximate interval becomes +0.43 to +1.57 percentage points. With 20,000 per group, it becomes +0.59 to +1.41. If the cost and per-order contribution scale so the break-even gap remains 0.50 points, only the largest of these examples clears that threshold at its lower interval bound.

Lewis and Rao found a median ROI standard error of 26.1 percentage points for the retail subset of their advertising experiments, versus 115 for brokerage experiments. Their calculations illustrate why the economics and variance matter to precision. A smaller store can ask whether a longer valid window or a less noisy primary outcome would change the decision, and estimate the practical cost of waiting.

Use an analysis that matches the assignment. The simple interval in this example assumes independent shopper-level observations and one fixed comparison. Randomizing by region, repeatedly testing a running result, or placing repeat visits from one shopper in both arms changes that assumption. A causal result and a useful business decision both need the right unit.

Keep attribution, eligibility and program impact separate

Use the reporting system to understand which touchpoints it credited. Use the randomized group assignment for the rollout comparison. A buyer who already purchased before a scheduled reminder can remain in the group they were assigned to even though you correctly suppress the message. Removing that buyer because the send did not happen can recreate the selection problem the experiment was meant to solve.

Document the eligibility window, assignment unit, suppression policy, order validity rule and outcome cutoff. Check that both groups had equal opportunity for later orders and refunds to arrive. Report the assigned counts, order or contribution outcome, estimated gap, uncertainty interval and break-even threshold together. A statistically detectable improvement can still be too small to pay for the program.

Choose the smallest difference that would change the decision

Lewis and Rao observed that every trial included more than 500,000 unique users and most exceeded one million. Individual-level sales had a standard deviation typically about ten times their mean over the campaign evaluation window. This helps explain why large randomized samples could still leave wide uncertainty around returns.

Under their idealized calculation, adding independent person-weeks to the median campaign would take about nine times its size to distinguish 50% ROI from break-even and about 62 times its size to detect a 10% ROI difference at the intended power. The smaller decision threshold calls for much greater precision. Neither multiple transfers directly to a checkout program.

For an ecommerce test, write down the smallest contribution difference that would change the rollout. Set an analysis window and use historical order and refund variability to estimate whether the design has a realistic chance of distinguishing it. If the answer would be the same for effects on either side of the interval, a more precise result may add little to the decision. If the interval spans a material loss and a material gain, keep the uncertainty in the decision rather than presenting the positive midpoint alone.

A return-adjusted margin can change the rollout decision

Return to the worked example with 2,000 treated shoppers, 20 extra valid orders at the point estimate and $400 of cost. At $40 per extra order, break-even is 0.50 percentage points. If expected contribution per additional order falls to $24 after refunds, discount effects or product mix, break-even rises to about 0.83 points. Twenty additional orders at the lower margin would leave $80 after cost.

The randomized order counts have not changed. What changed is the commercial threshold. The same approximate interval around a one-point lift still spans the lower and higher thresholds. This is why the campaign can appear more or less attractive depending on a cost assumption even though the causal estimate is identical.

Where possible, define the paid-order and refund maturity rules before the test. Track contribution for both groups on the assignment denominator. If the cost and margin are uncertain, report a sensitivity range and show which range of effects would make the decision change. The arithmetic example is hypothetical; the Lewis and Rao evidence explains why enough precision to answer the economic question is difficult to obtain.

Translate uncertainty into the sample you would need

Before increasing spend to run an experiment, ask whether the larger test could actually settle the decision. Lewis and Rao report a median retail ROI standard error of 26.1 percentage points in their field experiments. This is a useful scale for thinking about noisy measurement, and a new program can have a very different variance.

In a worked example with independent observations and a stable outcome variance, quadrupling the sample halves the standard error. Starting at 26.1 points gives 13.05 points. A 20-point decision threshold moves from 0.77 to 1.53 standard errors away. The larger study is more precise. It can still leave a decision unresolved.

Start with recent outcomes for the population you can randomize. Use them to estimate how many purchases and how much contribution you normally see. Preserve customer or market clustering when estimating uncertainty. If the required sample is unrealistic, test a narrower eligibility policy or commit to a decision range before launch. Avoid raising budget solely because the first estimate looked promising.

A large adjustment can still reverse the decision

In Gordon and colleagues’ Table 6, the stratified observational estimate and randomized point estimate had opposite signs in three of 14 checkout studies. In two of them the observational adjustment suggested a loss while the experiment pointed up. In the third it suggested a gain while the experiment pointed down. The direction matters when a team is choosing whether to increase budget, cancel a program or run a more informative test.

Study 8 makes the distinction tangible. Exposed versus unexposed customers differed by +179% in the unadjusted comparison. Stratification brought the estimate down to +33%, closer to the randomized point estimate of −2.9%. The remaining distance was still 35.9 percentage points and the sign still differed. A method can reduce a large discrepancy and leave the business decision unresolved.

Treat the gap from break-even as the decision quantity

The same paper reports that six of 14 checkout experiments fell short of statistical significance at the 5% level. That is a separate property from whether the observational estimate points in the right direction. A randomized result may identify a credible comparison while leaving a broad range of economically plausible effects.

For a campaign with a fixed contribution target, write the break-even increment and plausible losses before looking at the results. Keep an observational estimate as a diagnostic for audience mix and delivery. Use the assigned groups to estimate the causal effect, then show the interval relative to the break-even target. If it spans both loss and a worthwhile gain, compare the value of learning with the cost of leaving the program on. A different channel or audience can need its own test even when the current campaign is conclusive.

A fair comparison can still leave the budget decision open

Your checkout recovery holdout ends and the treatment group spent more than the control group. The result looks promising. Before you expand the program, you need to know how much uncertainty is around that difference. A modest positive estimate can be consistent with a result below break-even and with one that would justify expanding.

Random assignment gives the comparison its force. Some shoppers would have finished their order without a reminder, and the holdout gives you a way to account for them. Precision is a separate issue. If purchase values vary widely or relatively few eligible shoppers order during the test, one period's result may leave a wide range of outcomes plausible.

What the 25 advertising experiments found

Randall Lewis and Justin Rao analyzed 25 randomized field experiments, including 19 for retailers and six for financial service firms. The campaigns represented about $2.8 million in digital advertising spend together. Across the experiments, the median confidence interval for return on investment was wider than 100 percentage points. The size of the campaigns is what makes the observation worth keeping in mind.

The paper reports a median ROI standard error of 26.1 percentage points for the retail experiments. A conventional normal approximation turns that standard error into an interval width of about 102 percentage points. That is a calculation from the study's reported standard error. A particular experiment can have a much narrower or wider interval, and a checkout recovery test has its own audience, costs and purchase patterns.

Put the break-even point next to the result

Choose an outcome before the pilot. For example, you might use incremental contribution per eligible shopper over a fixed period, with discounts, refunds and the relevant program costs treated consistently across the groups. Write down what would make a change worth operating. If your chosen measure already deducts the program cost, zero is break-even. You may still need a higher threshold to justify rollout work.

When the result comes in, put the threshold beside the estimate and its interval. An interval entirely above your threshold supports a different decision from one that stretches across it. An interval entirely below the threshold can be informative even if the program increased gross sales. The decision should follow the economics you wrote down for that test.

Check which shoppers the result represents

Say the program sends checkout reminders only to people who are eligible to receive them. Make that the starting audience. Then keep people in the groups they were assigned to, including anyone in treatment who never received a message. If you compare only shoppers whose message was delivered with everyone in the control, you risk comparing different populations. Delivery status can be useful for troubleshooting without becoming the basis for the lift claim.

Set the outcome window and assignment rules before seeing the sales. Check that you can trace refunds and store costs on the same basis for both groups. Where a few large orders can dominate the result, record the spread in order values as well as the total. Those details make the range of plausible outcomes easier to interpret and help you work out what a longer or larger test would need.

Write the conclusion somebody else could act on

Record who was eligible, what counted as an order, how long you observed outcomes, which costs you included and where the estimate sat relative to break-even. Keep the interval with the point estimate when you pass the result along. A teammate should be able to tell whether your conclusion concerns a particular channel, offer or checkout flow.

If the interval crosses the decision threshold, use the pilot to identify what would make another test informative. An unreliable link between orders and eligible shoppers is a measurement problem. A small eligible cohort or widely varying order values is a precision problem. Each suggests different work before the next decision. The original experiments are useful because they separate uncertainty from the fairness of the comparison.

Study data

The Unfavorable Economics of Measuring the Returns to Advertising
Measure Value Context
Randomized field experiments 25 experiments Paper reports 25 field experiments.
Retail field experiments 19 experiments Nineteen experiments were for retailers.
Financial-services field experiments 6 experiments Six experiments were for financial-service firms.
Combined advertising expenditure About $2,800,000 Published collective expenditure is approximately $2.8 million.
Median ROI confidence interval width More than 100 percentage points Published median ROI interval width exceeds 100 percentage points.
Median ROI standard error in retail experiments 26.1 percentage points of ROI Retail-experiment median ROI standard error is 26.1 percentage points.
Gordon et al., A Comparison of Approaches to Advertising Measurement
Measure Value Context
Randomized US advertising experiments in the Gordon study 15 experiments Scope of the method comparison
User-experiment observations in the Gordon study 500 million Observations can repeat a user across experiments
Ad impressions in the Gordon study 1.6 billion Impressions are exposure events
Smallest Gordon campaign sample 2 million users Published range endpoint
Largest Gordon campaign sample 140 million users Published range endpoint
Checkout studies consistently off by more than a factor of three 7 of 14 studies Seven of the 14 checkout outcomes had observational point estimates consistently more than threefold off
Study 4, randomized checkout lift 73% Randomized experiment benchmark in that campaign
Study 4, exposed versus unexposed checkout lift 316% Observational comparison includes differences in who was exposed
Study 4, age-and-gender matched checkout lift 222% Observational match on age and gender only
Study 4, richest stratified regression checkout lift 74% Observational estimate reported next to the 73% experimental value
Study 9, randomized checkout lift 2.4% Reported as statistically different from zero at the 10% level
Study 9, closest observational checkout lift 1,306% Closest of the displayed observational estimates in that study
Study 9, share exposed to advertising 6.6% Authors propose low exposure as one possible explanation for the discrepancy
Study 15, stratified estimate -13% Rounded lift point estimate
Study 15, randomized estimate 2.4% Rounded lift point estimate
Study 10, stratified estimate -15% Rounded lift point estimate
Study 10, randomized estimate 2% Rounded lift point estimate
Study 8, stratified estimate 33% Rounded lift point estimate
Study 8, randomized estimate -2.9% Rounded lift point estimate
Lewis and Rao, The Unfavorable Economics of Measuring the Returns to Advertising
Measure Value Context
Median ROI standard error in financial-services studies 115 percentage points of ROI Median standard error for the financial-services subset
Sample-size multiplier to distinguish 50% ROI from break-even in their median scenario 9 times Scenario-specific sample multiple
Scale the authors say informative ad experiments can require More than 10 million person-weeks Potential experiment scale
Median campaign size multiplier to distinguish a 10% ROI difference 62 times Illustrative power calculation reported by the authors
Smallest unique-user scale exceeded by every trial More than 500000 users Each trial exceeded 500,000 unique users
Unique-user scale exceeded by most trials More than 1000000 users Most trials exceeded one million unique users
Typical individual-level sales standard deviation relative to mean 10 times Authors’ description of purchase-value variation

What changes when you compare the figures

Calculations from the published values above.
Comparison Result Calculation
Illustrative normal-approximation 95% interval width at that retail median standard error 102.312 percentage points = 2 × 1.96 × standard error
Study 4 exposed comparison minus randomized estimate 243 percentage points = 316 − 73
Study 4 matched comparison minus randomized estimate 149 percentage points = 222 − 73
Study 4 richest stratified estimate minus randomized estimate 1 percentage points = 74 − 73
Study 9 closest observational estimate minus randomized estimate 1,303.6 percentage points = 1,306 − 2.4
Share of checkout studies with consistent more-than-threefold discrepancies 50% = 7 ÷ 14 × 100
Financial-services to retail median ROI standard error ratio 4.41 times = 115 ÷ 26.1
Median 10% ROI discrimination multiplier divided by the 50% comparison multiplier 6.89 times = 62 / 9
Hypothetical break-even gap at $40 contribution per order 0.5 percentage points = 400 / 40 / 2000 * 100
Hypothetical break-even gap at $24 contribution per order 0.83 percentage points = 400 / 24 / 2000 * 100
Hypothetical net contribution at 20 additional orders and $24 each $80.00 = 20 * 24 - 400
Illustration, standard error at four times the sample 13.05 percentage points = 26.1 / 2
Illustration, standardized effect for a 20-point ROI difference 0.77 standard errors = 20 / 26.1
Illustration, standardized effect after quadrupling the sample 1.53 standard errors = 20 / 13.05
Share of checkout studies with a sign reversal 21.429% = 3 / 14 * 100
Study 8, remaining difference after stratification 35.9 percentage points = 33 - (-2.9)
Study 8, original exposed-to-randomized difference 181.9 percentage points = 179 - (-2.9)
Share of checkout studies short of 5% significance 42.857% = 6 / 14 * 100

Methodology and sources

Study figures come from the final 2015 Quarterly Journal of Economics article. The example interval width multiplies the published retail median standard error by twice 1.96, using a normal approximation. It describes the order of magnitude of the original results and gives no estimate for a specific store or recovery channel.

We transcribed Gordon et al.’s sample scope, Section 7.2 cases and Figure 10 comparisons, and Lewis and Rao’s 2015 results for advertising ROI precision. Observational and experimental lift estimates share a campaign outcome in the selected examples. Results from the two papers do not share an outcome or base and are kept separate. The worked checkout examples use independent two-group binomial estimates: standard error is the square root of the sum of p × (1 − p) divided by group size for each arm. The approximate 95% bounds add and subtract 1.96 standard errors from the difference in rates. The CSV reports hypothetical counts and a 0.50 percentage-point threshold to show how the decision changes with precision.

Download the source data (CSV)

Download the calculations (CSV)

Download the citation notes (CSV)

Download the worked holdout example

Download the evidence behind the new findings (CSV)

Sample ratio mismatch statistics for checkout A/B tests

A/B test peeking statistics for checkout experiments

Email holdout statistics for visits, purchases and revenue

Browse checkout recovery research

Sources and definitions

The Unfavorable Economics of Measuring the Returns to Advertising

Gordon et al., A Comparison of Approaches to Advertising Measurement

The source data includes the population, period, denominator and limitations for every figure.

About the publisher

Take Rate Take Rate PERFORMANCE RECOVERY

Research and practical guides for DTC merchants.