Summarize these selected article findings and their source context. This brief is not the full article. Preserve units, populations, dates, assumptions and limitations. Distinguish illustrative calculations, vendor descriptions and measured results. Do not invent omitted information or claim to have read the complete article. Incremental advertising return statistics and what a holdout can tell you Selected article findings, not the complete article. Across 14 advertising experiments with a checkout outcome, Gordon and colleagues found seven where observational lift estimates were consistently more than a factor of three off the randomized result. Lewis and Rao separately found a median ROI confidence interval over 100 percentage points wide in 25 advertising experiments. A checkout holdout needs both a fair comparison and enough precision to tell whether the program pays for itself. How often the observational estimate missed the checkout experiment Gordon et al. compared observational estimates with randomized results in 15 US Facebook ad experiments. In seven of the 14 studies with a checkout conversion outcome, observational point estimates were consistently off by more than a factor of three. This describes the methods and campaigns they evaluated. https://www.kellogg.northwestern.edu/faculty/gordon_b/files/fb_comparison.pdf One campaign with a close and a distant observational estimate In Gordon et al.’s Study 4, the randomized checkout lift estimate was 73%, compared with 316% from an exposed-versus-unexposed comparison and 74% from the richest reported stratified regression. These are lift estimates for the same campaign, not checkout conversion rates. https://www.kellogg.northwestern.edu/faculty/gordon_b/files/fb_comparison.pdf Why a near-zero experiment can disagree sharply with modeled lift In Gordon et al.’s Study 9, randomized checkout lift was estimated at 2.4%, while the closest displayed observational estimate was 1,306%. The 2.4% result was reported as different from zero at the 10% level, and the finding is campaign-specific. https://www.kellogg.northwestern.edu/faculty/gordon_b/files/fb_comparison.pdf How wide advertising ROI uncertainty can be Lewis and Rao’s 2015 analysis of 25 randomized advertising experiments reported a median ROI interval over 100 percentage points wide. The median ROI standard error was 26.1 percentage points in retail experiments and 115 percentage points in brokerage experiments. https://gwern.net/doc/economics/advertising/2015-lewis.pdf Advertising experiment size depends on the decision In Lewis and Rao’s median-campaign calculation, distinguishing 50% ROI from break-even required about nine times the experiment size discussed at baseline. That multiple depends on the authors’ study economics and uncertainty, and is not a sample size rule for checkout recovery. https://gwern.net/doc/economics/advertising/2015-lewis.pdf A narrower ROI decision can require far more data In Lewis and Rao’s 2015 study, their idealized calculation put the median campaign at roughly nine times its original size to distinguish 50% ROI from break-even and 62 times for a 10% ROI difference. The comparison assumes additional independent person-weeks and is specific to those advertising economics. https://gwern.net/doc/economics/advertising/2015-lewis.pdf Why large experiments can remain noisy Lewis and Rao reported that each of their 25 advertising field experiments exceeded 500,000 unique users and most exceeded one million. Individual-level sales standard deviation was typically about ten times the mean over the evaluation window. https://gwern.net/doc/economics/advertising/2015-lewis.pdf Break-even can move while the measured order gap stays fixed In Take Rate’s worked example, $400 of treatment cost across 2,000 assigned shoppers breaks even at a 0.50-point order-rate gap with $40 contribution per additional order. At $24 contribution it takes approximately 0.83 points. These are illustrative calculations, separate from Lewis and Rao’s published experiments. https://gwern.net/doc/economics/advertising/2015-lewis.pdf Four times the sample halves standard error under a stable variance model Taking Lewis and Rao’s reported 26.1-point median retail ROI standard error as an illustrative starting value, four times the independent observations would reduce it to 13.05 points if variance per observation stayed constant. https://gwern.net/doc/economics/advertising/2015-lewis.pdf Stratified and randomized estimates disagreed on sign in three checkout studies In Gordon et al. Table 6, three of 14 checkout comparisons (21.43%) had opposite signs for the stratified observational and randomized lift point estimates. The affected rows were Studies 8, 10 and 15. https://www.kellogg.northwestern.edu/faculty/gordon_b/files/fb_comparison.pdf Study 8 retained a 35.9-point gap after stratification In Gordon et al. Study 8, estimated checkout lift was +179% for exposed versus unexposed users, +33% after stratification and −2.9% in the randomized estimate. The latter two point estimates differed by 35.9 points. https://www.kellogg.northwestern.edu/faculty/gordon_b/files/fb_comparison.pdf