Take Rate Take Rate PERFORMANCE RECOVERY

Email holdout statistics by prior purchase channel

A 64,000-customer randomized retail replay compares email purchase lift among past phone, web and multichannel buyers.

Observed email lift by prior purchase channel

5.46 buyers per 1,000 assigned
Phone
7.2 buyers per 1,000 assigned
Web
Two-week buyer lift against each channel’s own no-email control.
About 6 min read
Updated:
Statistics
Learn about our Editorial policyLearn about our Research methodology

Sources checked 11 November 2026.

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Full article, downloads and prompt

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Download articleDownload AI source digest

Copy the complete article or download it for later.

Contents

Quick answer

In the Hillstrom trial, Mens email added an estimated 5.46 buyers per 1,000 prior phone buyers, 7.2 per 1,000 prior web buyers and 10.17 per 1,000 multichannel buyers versus each group’s no-email holdout over two weeks. Channel describes earlier purchase behavior.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

Key findings and sources

The largest Mens email lift estimate came from multichannel customers

Take Rate’s Hillstrom replay found 10.17 additional buyers per 1,000 assigned multichannel customers in the Mens email arm versus their no-email holdout over two weeks.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

64,000 customers in the Hillstrom retail randomized email trial, grouped by prior purchase channel Subtract 18/2,606 from 44/2,577 and multiply by 1,000.

Prior phone and web buyers had different estimates

In Hillstrom’s retail email test, Mens email added an estimated 5.46 buyers per 1,000 past phone buyers and 7.2 per 1,000 past web buyers versus each cohort’s no-email holdout.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

64,000 customers in the Hillstrom retail randomized email trial, grouped by prior purchase channel Compare conversion per assigned customer within each historical Channel group.

Womens email also showed its largest estimate in multichannel

In Hillstrom’s two-week retail trial, Womens email added an estimated 7.05 buyers per 1,000 multichannel assignments compared with the channel-matched holdout.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

64,000 customers in the Hillstrom retail randomized email trial, grouped by prior purchase channel Subtract 18/2,606 from 36/2,579 and multiply by 1,000.

Keep the three historical channel denominators explicit

The Hillstrom no-email arm recorded 50 buyers in 9,327 prior phone assignments, 54 in 9,373 web assignments and 18 in 2,606 multichannel assignments.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

64,000 customers in the Hillstrom retail randomized email trial, grouped by prior purchase channel Count rows and sum conversion in the no-email arm by the prior Channel field.

Compare the results

Observed email lift by prior purchase channel

Use prior channel as a diagnostic. Treat the smaller multichannel result as a hypothesis for a new holdout.

Phone
Mens email 5.46 buyers per 1,000
Womens email 1.73 buyers per 1,000
Web
Mens email 7.2 buyers per 1,000
Womens email 3.43 buyers per 1,000
Multichannel
Mens email 10.17 buyers per 1,000
Womens email 7.05 buyers per 1,000
Two-week buyer lift against each channel’s own no-email control. Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge.

Use the baseline that belongs to the customer cohort

People who already buy across several channels may have a different natural purchase rate from those who buy online only. A direct comparison of treatment buyers can confuse those starting differences with the message effect.

The Mens email arm recorded 44 purchases among 2,577 multichannel customers, versus 18 among 2,606 no-email customers. The absolute denominators help readers see the smaller number of purchases behind the largest point estimate. The phone and web cohorts each have more than three times as many assignments per arm.

Separate the outreach from the earlier channel history

Channel records where someone had purchased over the preceding year. The experiment assigned email content, while channel came from earlier customer behavior. Moving a shopper to another sales channel would be a different intervention.

The two emails promoted different assortments. Report both against the same cohort control so product preference can remain a plausible explanation. If the business wants to learn whether the channel is actionable, set a new eligibility rule and randomize the message within each eligible group.

Plan a follow-up around actual reachable volume

Record the eligibility label before assigning a shopper. Keep the no-message control in each reporting group, and count everyone assigned, including people who never click. Compare two-week buyers and contribution per customer before choosing a rollout.

A high estimate from a small cohort can justify a targeted test. Check the number of observations needed to estimate a commercially useful difference, then repeat the test with another offer or period before making the segment permanent.

Study data

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge
Measure Value Context
Phone holdout assigned customers 9,327 customers Original CSV group sum
Phone holdout buyers 50 buyers Original CSV group sum
Phone holdout spend $6,010.81 Original CSV group sum
Phone Mens email assigned customers 9,240 customers Original CSV group sum
Phone Mens email buyers 100 buyers Original CSV group sum
Phone Mens email spend $11,166.79 Original CSV group sum
Phone Womens email assigned customers 9,454 customers Original CSV group sum
Phone Womens email buyers 67 buyers Original CSV group sum
Phone Womens email spend $8,290.46 Original CSV group sum
Phone Mens email incremental buyers 5.46 buyers per 1,000 assigned Original randomized contrast point estimate
Phone Womens email incremental buyers 1.73 buyers per 1,000 assigned Original randomized contrast point estimate
Web holdout assigned customers 9,373 customers Original CSV group sum
Web holdout buyers 54 buyers Original CSV group sum
Web holdout spend $6,292.90 Original CSV group sum
Web Mens email assigned customers 9,490 customers Original CSV group sum
Web Mens email buyers 123 buyers Original CSV group sum
Web Mens email spend $14,442.21 Original CSV group sum
Web Womens email assigned customers 9,354 customers Original CSV group sum
Web Womens email buyers 86 buyers Original CSV group sum
Web Womens email spend $10,174.37 Original CSV group sum
Web Mens email incremental buyers 7.2 buyers per 1,000 assigned Original randomized contrast point estimate
Web Womens email incremental buyers 3.43 buyers per 1,000 assigned Original randomized contrast point estimate
Multichannel holdout assigned customers 2,606 customers Original CSV group sum
Multichannel holdout buyers 18 buyers Original CSV group sum
Multichannel holdout spend $1,604.62 Original CSV group sum
Multichannel Mens email assigned customers 2,577 customers Original CSV group sum
Multichannel Mens email buyers 44 buyers Original CSV group sum
Multichannel Mens email spend $4,702.69 Original CSV group sum
Multichannel Womens email assigned customers 2,579 customers Original CSV group sum
Multichannel Womens email buyers 36 buyers Original CSV group sum
Multichannel Womens email spend $4,573.28 Original CSV group sum
Multichannel Mens email incremental buyers 10.17 buyers per 1,000 assigned Original randomized contrast point estimate
Multichannel Womens email incremental buyers 7.05 buyers per 1,000 assigned Original randomized contrast point estimate

What changes when you compare the figures

Calculations from the published values above.
Comparison Result Calculation
Phone Mens email incremental buyers 5.46 buyers per 1,000 assigned = (100 / 9240 - 50 / 9327) * 1000
Phone Mens email spend lift 0.56 USD per assigned customer = (11166.79 / 9240 - 6010.81 / 9327) * 1
Phone Womens email incremental buyers 1.73 buyers per 1,000 assigned = (67 / 9454 - 50 / 9327) * 1000
Phone Womens email spend lift 0.23 USD per assigned customer = (8290.46 / 9454 - 6010.81 / 9327) * 1
Web Mens email incremental buyers 7.2 buyers per 1,000 assigned = (123 / 9490 - 54 / 9373) * 1000
Web Mens email spend lift 0.85 USD per assigned customer = (14442.21 / 9490 - 6292.9 / 9373) * 1
Web Womens email incremental buyers 3.43 buyers per 1,000 assigned = (86 / 9354 - 54 / 9373) * 1000
Web Womens email spend lift 0.42 USD per assigned customer = (10174.37 / 9354 - 6292.9 / 9373) * 1
Multichannel Mens email incremental buyers 10.17 buyers per 1,000 assigned = (44 / 2577 - 18 / 2606) * 1000
Multichannel Mens email spend lift 1.21 USD per assigned customer = (4702.69 / 2577 - 1604.62 / 2606) * 1
Multichannel Womens email incremental buyers 7.05 buyers per 1,000 assigned = (36 / 2579 - 18 / 2606) * 1000
Multichannel Womens email spend lift 1.16 USD per assigned customer = (4573.28 / 2579 - 1604.62 / 2606) * 1

Methodology and sources

Use all public Hillstrom rows. Group by prior-year purchase channel and randomized email arm. Divide buyers by all assigned rows and subtract the no-email purchase rate in the same channel. Multiply by 1,000. Calculate spend per assigned customer separately. Include zero outcomes and avoid conditioning on visits. Compare subgroup point estimates descriptively.

Download the source data (CSV)

Download the calculations (CSV)

Download the citation notes (CSV)

Download the evidence behind the new findings (CSV)

Download the reproducible analysis script

Download the source-labeled chart (SVG)

Revenue lift statistics for ecommerce experiments with high-spend orders

Incremental advertising return statistics and what a holdout can tell you

Sample ratio mismatch statistics for checkout A/B tests

A/B test peeking statistics for checkout experiments

Browse checkout recovery research

Sources and definitions

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

The source data includes the population, period, denominator and limitations for every figure.

About the publisher

Take Rate Take Rate PERFORMANCE RECOVERY

Research and practical guides for DTC merchants.