Take Rate Take Rate PERFORMANCE RECOVERY

Revenue lift statistics for ecommerce experiments with high-spend orders

A randomized retail dataset shows how uncapped email revenue lift compares with two alternative customer-level spend caps.

Measured revenue lift changes when each outcome is capped

0.77 USD per assigned customer
Mens email
0.42 USD per assigned customer
Womens email
USD per assigned customer, over two weeks. Both arms use the same customer-level cap.
About 6 min read
Updated:
Statistics
Learn about our Editorial policyLearn about our Research methodology

Sources checked 11 November 2026.

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Full article, downloads and prompt

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Download articleDownload AI source digest

Copy the complete article or download it for later.

Contents

Quick answer

In Hillstrom’s retail test, Mens email lifted raw two-week spend by 0.77 USD per assignment versus no email. The estimated lift was 0.42 USD when each customer’s recorded spend was capped at $100 before averaging.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

Key findings and sources

Mens email showed lower lift after a $100 outcome cap

Take Rate’s Hillstrom replay found 0.77 USD more raw spend per Mens email assignment versus no email and 0.42 USD more after capping each customer’s outcome at $100.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract assignment-level averages on raw spend and min(spend,100).

The $250 cap retained more of the estimated lift

In Hillstrom’s two-week trial, the revenue difference at a $250 cap was 0.66 USD per Mens email assignment and 0.39 USD per Womens email assignment versus no email.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract the corresponding holdout mean using min(spend,250).

Spend above $100 was infrequent among assignments

In the Hillstrom retail test, 94 of 21,307 Mens email assignments, 92 of 21,387 Womens email assignments and 48 of 21,306 no-email assignments exceeded $100 in two-week spend.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Count individual outcomes strictly above $100 by randomized arm.

Womens email lift remained positive on both definitions

The Hillstrom Womens email arm added 0.42 USD per assignment in raw spend and 0.23 USD after a $100 per-customer cap, both relative to no email.

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract the corresponding no-email mean after applying the same outcome definition to both groups.

Compare the results

Measured revenue lift changes when each outcome is capped

Keep raw spend as the business outcome and use the cap check to explain sensitivity to larger purchases.

Mens email
Uncapped spend 0.77 USD
$100 cap per customer 0.42 USD
Womens email
Uncapped spend 0.42 USD
$100 cap per customer 0.23 USD
USD per assigned customer, over two weeks. Both arms use the same customer-level cap. Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge.

Check what sits behind the assignment-level average

Revenue per assignment keeps all shoppers and the full value of every order in the analysis. It also means a small set of high-spend customers can materially move the reported effect.

The Mens email arm had 94 of 21,307 customers with spend above $100. The no-email arm had 48 of 21,306. Capping each person’s spend at $100 reduced total recorded spend from $30,311.69 to $17,870.62 in the Mens arm and from $13,908.33 to $8,854.76 in the holdout. Those are observed features of the distribution.

Change the definition symmetrically and label the change

A spend cap limits how much any one person can contribute to the mean. Apply the same cap to treatment and holdout, keep all zeroes and preserve assignment counts. Capped spend asks how much the cohort spent up to the cutoff. Uncapped spend retains the real recorded revenue.

A smaller capped lift can reflect a real effect on big orders. Before treating any purchase as suspect, verify its payment, margin and return status using the actual order record. If the orders are valid, their value belongs in the raw business result.

Report the diagnostic next to the financial decision

For a checkout recovery test, predefine the primary outcome. Choose diagnostic caps using historical orders before reviewing trial outcomes. Report raw and capped differences with uncertainty intervals and the number of observations affected.

If the financial result depends on a few orders, review whether that purchase pattern can repeat and whether the margins justify the program. Keeping raw and capped views side by side gives a more useful decision record than removing purchases after reading the test result.

Study data

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge
Measure Value Context
No email assigned customers 21,306 customers Original count or sum of public spend records
No email uncapped spend $13,908.33 Original count or sum of public spend records
No email spend capped at $100 $8,854.76 Original count or sum of public spend records
No email spend capped at $250 $12,439.08 Original count or sum of public spend records
No email customers above $100 48 customers Original count or sum of public spend records
No email customers above $250 9 customers Original count or sum of public spend records
No email maximum recorded spend $499.00 Original count or sum of public spend records
Mens email assigned customers 21,307 customers Original count or sum of public spend records
Mens email uncapped spend $30,311.69 Original count or sum of public spend records
Mens email spend capped at $100 $17,870.62 Original count or sum of public spend records
Mens email spend capped at $250 $26,606.79 Original count or sum of public spend records
Mens email customers above $100 94 customers Original count or sum of public spend records
Mens email customers above $250 29 customers Original count or sum of public spend records
Mens email maximum recorded spend $499.00 Original count or sum of public spend records
Womens email assigned customers 21,387 customers Original count or sum of public spend records
Womens email uncapped spend $23,038.11 Original count or sum of public spend records
Womens email spend capped at $100 $13,776.46 Original count or sum of public spend records
Womens email spend capped at $250 $20,746.70 Original count or sum of public spend records
Womens email customers above $100 92 customers Original count or sum of public spend records
Womens email customers above $250 21 customers Original count or sum of public spend records
Womens email maximum recorded spend $499.00 Original count or sum of public spend records
Uncapped Mens email revenue lift 0.77 USD per assigned customer Original arm difference of assignment-level means
Uncapped Womens email revenue lift 0.42 USD per assigned customer Original arm difference of assignment-level means
Cap $100 Mens email revenue lift 0.42 USD per assigned customer Original arm difference of assignment-level means
Cap $100 Womens email revenue lift 0.23 USD per assigned customer Original arm difference of assignment-level means
Cap $250 Mens email revenue lift 0.66 USD per assigned customer Original arm difference of assignment-level means
Cap $250 Womens email revenue lift 0.39 USD per assigned customer Original arm difference of assignment-level means

What changes when you compare the figures

Calculations from the published values above.
Comparison Result Calculation
Uncapped Mens email revenue lift 0.77 USD per assigned customer = (30311.69 / 21307 - 13908.33 / 21306) * 1
Uncapped Womens email revenue lift 0.42 USD per assigned customer = (23038.11 / 21387 - 13908.33 / 21306) * 1
Cap $100 Mens email revenue lift 0.42 USD per assigned customer = (17870.62 / 21307 - 8854.76 / 21306) * 1
Cap $100 Womens email revenue lift 0.23 USD per assigned customer = (13776.46 / 21387 - 8854.76 / 21306) * 1
Cap $250 Mens email revenue lift 0.66 USD per assigned customer = (26606.79 / 21307 - 12439.08 / 21306) * 1
Cap $250 Womens email revenue lift 0.39 USD per assigned customer = (20746.7 / 21387 - 12439.08 / 21306) * 1
Mens email share of customers above $100 0.44% = 94 / 21307 * 100
Mens email share of revenue above $100 cap 41.04% = 12441.07 / 30311.69 * 100
Womens email share of customers above $100 0.43% = 92 / 21387 * 100
Womens email share of revenue above $100 cap 40.2% = 9261.65 / 23038.11 * 100

Methodology and sources

Use all 64,000 randomized customers, retaining the zero-spend rows. Compute average recorded two-week spend for each arm, then recalculate it using min(spend,100) and min(spend,250) per customer. Subtract the no-email mean under the matching definition. The thresholds are illustrative sensitivity checks and keep every person in the sample.

Download the source data (CSV)

Download the calculations (CSV)

Download the citation notes (CSV)

Download the evidence behind the new findings (CSV)

Download the reproducible analysis script

Download the source-labeled chart (SVG)

Email holdout statistics by prior purchase channel

Incremental advertising return statistics and what a holdout can tell you

Sample ratio mismatch statistics for checkout A/B tests

A/B test peeking statistics for checkout experiments

Browse checkout recovery research

Sources and definitions

Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge

The source data includes the population, period, denominator and limitations for every figure.

About the publisher

Take Rate Take Rate PERFORMANCE RECOVERY

Research and practical guides for DTC merchants.