Take Rate Take Rate PERFORMANCE RECOVERY

Email holdout statistics for visits, purchases and revenue

A reanalysis of 64,000 randomized retail customers shows how the same email test looks at three stages of the funnel.

Additional buyers compared with no email

6.81 buyers per 1,000 assigned
Men’s merchandise email
3.11 buyers per 1,000 assigned
Women’s merchandise email
Original reconstruction of the Hillstrom retail email experiment. Two weeks of outcomes, counted for everyone assigned.
About 7 min read
Updated:
Statistics
Learn about our Editorial policyLearn about our Research methodology

Sources checked 11 November 2026.

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Full article, downloads and prompt

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Download articleDownload AI source digest

Copy the complete article or download it for later.

Contents

Quick answer

In a public randomized retail email test, the men’s merchandise email produced 76.59 additional visitors and 6.81 additional buyers per 1,000 customers assigned, compared with no email. Recorded spend rose by about $0.77 per assigned customer over two weeks. Evaluate the result you need for rollout, using everyone assigned to the campaign or holdout.

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge · hillstorm1.s3.us-east-2.amazonaws.com

Key findings and sources

One email added 76.59 visitors and 6.81 buyers per 1,000

In Take Rate’s reanalysis of Kevin Hillstrom’s 64,000-customer randomized email dataset, the men’s merchandise email added 76.59 visitors and 6.81 buyers per 1,000 assigned versus no email over two weeks.

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge · hillstorm1.s3.us-east-2.amazonaws.com

The other email had a smaller purchase contrast

In the same randomized dataset, the women’s merchandise email added 45.23 visitors and 3.11 buyers per 1,000 assigned versus no email over two weeks. All rates use the full randomized arm.

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge · hillstorm1.s3.us-east-2.amazonaws.com

Observed spend increased by different amounts by campaign

In the Hillstrom dataset, two-week spend per assigned customer was $0.65 for no email, $1.08 for the women’s merchandise email and $1.42 for the men’s merchandise email. These are recorded spend figures before a margin adjustment.

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge · hillstorm1.s3.us-east-2.amazonaws.com

Purchases also occurred in the holdout

In the Hillstrom no-email arm, 122 of 21,306 assigned customers purchased within two weeks. This 0.57% baseline is part of the randomized comparison.

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge · hillstorm1.s3.us-east-2.amazonaws.com

Email purchase lift was similar across two pre-assignment recency bands

In the historical Hillstrom experiment, the men’s merchandise email added 7.33 buyers per 1,000 assigned among one-to-three-month buyers and 7.29 among seven-to-12-month buyers, each against its own no-email arm.

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge

Two-week outcomes after randomized assignment, split using prior purchase recency. Subtract the control purchase proportion from the treatment purchase proportion within each recency band.

The recency difference was smaller than its sampling uncertainty

The observed difference between the two recency-band purchase lifts was 0.04 buyers per 1,000. An unadjusted approximate 95% interval was -4.27 to 4.35 buyers per 1,000.

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge

Two disjoint post-hoc recency bands from Hillstrom’s historic randomized retail experiment. Use the sum of the independent binomial variances across the four cells and a 1.96 standard-error interval.

Compare the results

Additional buyers compared with no email. Exact values are also in the study data.
Additional buyers compared with no email.

Start with the outcome that decides the rollout

A report can look strong on traffic and modest on paid orders at the same time. These counts tell you where the effect appears in the funnel. Each uses the same eligible customer group, so you can compare the two without narrowing the analysis to people who clicked or visited.

The original challenge randomly assigned customers to a men’s merchandise email, a women’s merchandise email or no email. Those are campaign labels. They do not identify the customer’s gender. The no-email customers could still visit and buy, which is exactly the background activity the test needs to measure.

Put the visit and purchase differences on one scale

The men’s merchandise email added 76.59 visitors and 6.81 buyers per 1,000 assigned. The women’s merchandise email added 45.23 visitors and 3.11 buyers per 1,000 assigned. The campaigns differed in their observed effects. These figures also show why the visit count is a poor stand-in for the number of additional orders.

Dividing incremental purchases by incremental visits does not recover a conversion rate for the people persuaded to visit. The data do not identify who would have visited anyway, and the email can change what a customer does after arriving. Keep each contrast on the assignment denominator.

Translate spend into a decision the merchant can use

Average recorded spend was about $0.65 with no email, $1.08 for the women’s merchandise email and $1.42 for the men’s merchandise email. That is revenue before any adjustment for contribution margin, fulfillment cost or later returns. A merchant can define an outcome that deducts those costs and compare the groups on that measure.

For a recovery program, predefine who could be contacted and the period during which a completed order counts. Record every assigned customer, including people who buy before the message reaches them. Read visits alongside purchases so the team can see whether a change helped more people return, helped them complete the order, or both.

Both recency groups benefited in the point estimate

For customers who had last purchased one to three months earlier, the men’s merchandise email added 7.33 buyers per 1,000 assigned compared with no email. For customers last purchasing seven to 12 months earlier, it added 7.29. These are pre-assignment groups. The outcome still includes every assigned customer inside the group, including people who never visited the site.

The point estimates differ by 0.04 buyers per 1,000. A simple large-sample comparison of those two unadjusted lift estimates gives an approximate 95% interval of -4.27 to 4.35 buyers per 1,000 for the difference. The interval includes zero and materially different policies in either direction. It offers little reason to choose a recency policy just because one rounded point estimate is higher.

A different natural purchase rate can still support the same message policy

A team may see more purchases among recent buyers after a message. That alone is a reason to look at the group’s natural purchase rate. In this data, recent buyers had a higher no-email purchase rate than buyers whose last purchase was seven to 12 months earlier. Each campaign effect therefore needs its own group’s no-email comparison.

Preselect a recency band where treatment rules genuinely differ, such as a replenishment window or a recent complaint. Verify enough assigned buyers and recorded orders in each band before asking which policy won. Evaluate cost and customer experience within that band. If uncertainty spans both choices, keep a stable policy and run a separate confirmation where the actual decision is whether, or when, to contact those customers.

Study data

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge
Measure Value Context
No email, assigned customers 21,306 customers Original aggregation, assignment denominator
No email, site visitors 2,262 customers Original aggregation, assignment denominator
No email, buyers 122 customers Original aggregation, assignment denominator
No email, recorded spend $13,908.33 Original aggregation, assignment denominator
No email, visit rate 10.61673% Original aggregation, assignment denominator
No email, purchase rate 0.57261% Original aggregation, assignment denominator
No email, spend per assigned customer 0.65279 USD per customer Original aggregation, assignment denominator
Men’s merchandise email, assigned customers 21,307 customers Original aggregation, assignment denominator
Men’s merchandise email, site visitors 3,894 customers Original aggregation, assignment denominator
Men’s merchandise email, buyers 267 customers Original aggregation, assignment denominator
Men’s merchandise email, recorded spend $30,311.69 Original aggregation, assignment denominator
Men’s merchandise email, visit rate 18.27568% Original aggregation, assignment denominator
Men’s merchandise email, purchase rate 1.25311% Original aggregation, assignment denominator
Men’s merchandise email, spend per assigned customer 1.42262 USD per customer Original aggregation, assignment denominator
Men’s merchandise email, additional buyers per 1,000 6.81 buyers per 1,000 assigned Two-week randomized assignment difference
Women’s merchandise email, assigned customers 21,387 customers Original aggregation, assignment denominator
Women’s merchandise email, site visitors 3,238 customers Original aggregation, assignment denominator
Women’s merchandise email, buyers 189 customers Original aggregation, assignment denominator
Women’s merchandise email, recorded spend $23,038.11 Original aggregation, assignment denominator
Women’s merchandise email, visit rate 15.14004% Original aggregation, assignment denominator
Women’s merchandise email, purchase rate 0.88371% Original aggregation, assignment denominator
Women’s merchandise email, spend per assigned customer 1.0772 USD per customer Original aggregation, assignment denominator
Women’s merchandise email, additional buyers per 1,000 3.11 buyers per 1,000 assigned Two-week randomized assignment difference

What changes when you compare the figures

Calculations from the published values above.
Comparison Result Calculation
Men’s merchandise email, additional visitors per 1,000 76.59 visitors per 1,000 assigned = (3894 / 21307 - 2262 / 21306) * 1000
Men’s merchandise email, additional buyers per 1,000 6.81 buyers per 1,000 assigned = (267 / 21307 - 122 / 21306) * 1000
Men’s merchandise email, additional spend per assigned customer 0.77 USD per customer = (30311.69 / 21307 - 13908.33 / 21306) * 1
Women’s merchandise email, additional visitors per 1,000 45.23 visitors per 1,000 assigned = (3238 / 21387 - 2262 / 21306) * 1000
Women’s merchandise email, additional buyers per 1,000 3.11 buyers per 1,000 assigned = (189 / 21387 - 122 / 21306) * 1000
Women’s merchandise email, additional spend per assigned customer 0.42 USD per customer = (23038.11 / 21387 - 13908.33 / 21306) * 1
Women’s merchandise email, dollars per 1,000 assignments 424.41 USD per 1,000 assigned = (23038.11 / 21387 - 13908.33 / 21306) * 1000
One-to-three-month buyer lift per 1,000 7.326 buyers per 1,000 assigned = (122 / 7469 - 67 / 7438) * 1000
Seven-to-12-month buyer lift per 1,000 7.289 buyers per 1,000 assigned = (96 / 9165 - 29 / 9102) * 1000
Difference between purchase lifts in two recency bands 0.038 buyers per 1,000 assigned = (122 / 7469 - 67 / 7438 - 96 / 9165 + 29 / 9102) * 1000
Approximate contrast standard error 2.198 buyers per 1,000 assigned = ((122/7469)*(1-122/7469)/7469 + (67/7438)*(1-67/7438)/7438 + (96/9165)*(1-96/9165)/9165 + (29/9102)*(1-29/9102)/9102)**0.5 * 1000

Methodology and sources

Group the 64,000 public rows by randomized email assignment. Sum visit, conversion and spend. Divide every outcome by all customers assigned to that arm. Subtract the no-email rate for a treatment contrast. The packaged source archive was verified against its published MD5 and the reproducible script records a SHA-256. The dataset measures visits rather than email clicks. Figures describe existing customers at one historical retailer.

Download the source data (CSV)

Download the calculations (CSV)

Download the citation notes (CSV)

Download the reproducible analysis script

Download the evidence behind the new findings (CSV)

Download the analysis script for these comparisons

Incremental advertising return statistics and what a holdout can tell you

Sample ratio mismatch statistics for checkout A/B tests

A/B test peeking statistics for checkout experiments

Browse checkout recovery research

Sources and definitions

Kevin Hillstrom, MineThatData Email Analytics and Data Mining Challenge

The source data includes the population, period, denominator and limitations for every figure.

About the publisher

Take Rate Take Rate PERFORMANCE RECOVERY

Research and practical guides for DTC merchants.