Quick answer
In Hillstrom’s retail test, Mens email lifted raw two-week spend by 0.77 USD per assignment versus no email. The estimated lift was 0.42 USD when each customer’s recorded spend was capped at $100 before averaging.
Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge
Key findings and sources
Mens email showed lower lift after a $100 outcome cap
Take Rate’s Hillstrom replay found 0.77 USD more raw spend per Mens email assignment versus no email and 0.42 USD more after capping each customer’s outcome at $100.
Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge
64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract assignment-level averages on raw spend and min(spend,100).
The $250 cap retained more of the estimated lift
In Hillstrom’s two-week trial, the revenue difference at a $250 cap was 0.66 USD per Mens email assignment and 0.39 USD per Womens email assignment versus no email.
Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge
64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract the corresponding holdout mean using min(spend,250).
Spend above $100 was infrequent among assignments
In the Hillstrom retail test, 94 of 21,307 Mens email assignments, 92 of 21,387 Womens email assignments and 48 of 21,306 no-email assignments exceeded $100 in two-week spend.
Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge
64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Count individual outcomes strictly above $100 by randomized arm.
Womens email lift remained positive on both definitions
The Hillstrom Womens email arm added 0.42 USD per assignment in raw spend and 0.23 USD after a $100 per-customer cap, both relative to no email.
Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge
64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract the corresponding no-email mean after applying the same outcome definition to both groups.
Compare the results
Measured revenue lift changes when each outcome is capped
Keep raw spend as the business outcome and use the cap check to explain sensitivity to larger purchases.
Check what sits behind the assignment-level average
Revenue per assignment keeps all shoppers and the full value of every order in the analysis. It also means a small set of high-spend customers can materially move the reported effect.
The Mens email arm had 94 of 21,307 customers with spend above $100. The no-email arm had 48 of 21,306. Capping each person’s spend at $100 reduced total recorded spend from $30,311.69 to $17,870.62 in the Mens arm and from $13,908.33 to $8,854.76 in the holdout. Those are observed features of the distribution.
Change the definition symmetrically and label the change
A spend cap limits how much any one person can contribute to the mean. Apply the same cap to treatment and holdout, keep all zeroes and preserve assignment counts. Capped spend asks how much the cohort spent up to the cutoff. Uncapped spend retains the real recorded revenue.
A smaller capped lift can reflect a real effect on big orders. Before treating any purchase as suspect, verify its payment, margin and return status using the actual order record. If the orders are valid, their value belongs in the raw business result.
Report the diagnostic next to the financial decision
For a checkout recovery test, predefine the primary outcome. Choose diagnostic caps using historical orders before reviewing trial outcomes. Report raw and capped differences with uncertainty intervals and the number of observations affected.
If the financial result depends on a few orders, review whether that purchase pattern can repeat and whether the margins justify the program. Keeping raw and capped views side by side gives a more useful decision record than removing purchases after reading the test result.
Study data
| Measure | Value | Context |
|---|---|---|
| No email assigned customers | 21,306 customers | Original count or sum of public spend records |
| No email uncapped spend | $13,908.33 | Original count or sum of public spend records |
| No email spend capped at $100 | $8,854.76 | Original count or sum of public spend records |
| No email spend capped at $250 | $12,439.08 | Original count or sum of public spend records |
| No email customers above $100 | 48 customers | Original count or sum of public spend records |
| No email customers above $250 | 9 customers | Original count or sum of public spend records |
| No email maximum recorded spend | $499.00 | Original count or sum of public spend records |
| Mens email assigned customers | 21,307 customers | Original count or sum of public spend records |
| Mens email uncapped spend | $30,311.69 | Original count or sum of public spend records |
| Mens email spend capped at $100 | $17,870.62 | Original count or sum of public spend records |
| Mens email spend capped at $250 | $26,606.79 | Original count or sum of public spend records |
| Mens email customers above $100 | 94 customers | Original count or sum of public spend records |
| Mens email customers above $250 | 29 customers | Original count or sum of public spend records |
| Mens email maximum recorded spend | $499.00 | Original count or sum of public spend records |
| Womens email assigned customers | 21,387 customers | Original count or sum of public spend records |
| Womens email uncapped spend | $23,038.11 | Original count or sum of public spend records |
| Womens email spend capped at $100 | $13,776.46 | Original count or sum of public spend records |
| Womens email spend capped at $250 | $20,746.70 | Original count or sum of public spend records |
| Womens email customers above $100 | 92 customers | Original count or sum of public spend records |
| Womens email customers above $250 | 21 customers | Original count or sum of public spend records |
| Womens email maximum recorded spend | $499.00 | Original count or sum of public spend records |
| Uncapped Mens email revenue lift | 0.77 USD per assigned customer | Original arm difference of assignment-level means |
| Uncapped Womens email revenue lift | 0.42 USD per assigned customer | Original arm difference of assignment-level means |
| Cap $100 Mens email revenue lift | 0.42 USD per assigned customer | Original arm difference of assignment-level means |
| Cap $100 Womens email revenue lift | 0.23 USD per assigned customer | Original arm difference of assignment-level means |
| Cap $250 Mens email revenue lift | 0.66 USD per assigned customer | Original arm difference of assignment-level means |
| Cap $250 Womens email revenue lift | 0.39 USD per assigned customer | Original arm difference of assignment-level means |
What changes when you compare the figures
| Comparison | Result | Calculation |
|---|---|---|
| Uncapped Mens email revenue lift | 0.77 USD per assigned customer | = (30311.69 / 21307 - 13908.33 / 21306) * 1 |
| Uncapped Womens email revenue lift | 0.42 USD per assigned customer | = (23038.11 / 21387 - 13908.33 / 21306) * 1 |
| Cap $100 Mens email revenue lift | 0.42 USD per assigned customer | = (17870.62 / 21307 - 8854.76 / 21306) * 1 |
| Cap $100 Womens email revenue lift | 0.23 USD per assigned customer | = (13776.46 / 21387 - 8854.76 / 21306) * 1 |
| Cap $250 Mens email revenue lift | 0.66 USD per assigned customer | = (26606.79 / 21307 - 12439.08 / 21306) * 1 |
| Cap $250 Womens email revenue lift | 0.39 USD per assigned customer | = (20746.7 / 21387 - 12439.08 / 21306) * 1 |
| Mens email share of customers above $100 | 0.44% | = 94 / 21307 * 100 |
| Mens email share of revenue above $100 cap | 41.04% | = 12441.07 / 30311.69 * 100 |
| Womens email share of customers above $100 | 0.43% | = 92 / 21387 * 100 |
| Womens email share of revenue above $100 cap | 40.2% | = 9261.65 / 23038.11 * 100 |
Methodology and sources
Use all 64,000 randomized customers, retaining the zero-spend rows. Compute average recorded two-week spend for each arm, then recalculate it using min(spend,100) and min(spend,250) per customer. Subtract the no-email mean under the matching definition. The thresholds are illustrative sensitivity checks and keep every person in the sample.
Download the source data (CSV)
Download the calculations (CSV)
Download the citation notes (CSV)
Download the evidence behind the new findings (CSV)
Related reading
Email holdout statistics by prior purchase channel
Incremental advertising return statistics and what a holdout can tell you
Sample ratio mismatch statistics for checkout A/B tests
Sources and definitions
Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge
The source data includes the population, period, denominator and limitations for every figure.