Revenue lift statistics for ecommerce experiments with high-spend orders By Take Rate, Take Rate. Sources checked 2026-11-11. A randomized retail dataset shows how uncapped email revenue lift compares with two alternative customer-level spend caps. ## Quick answer In Hillstrom’s retail test, Mens email lifted raw two-week spend by 0.77 USD per assignment versus no email. The estimated lift was 0.42 USD when each customer’s recorded spend was capped at $100 before averaging. Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge (https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html) ## Key findings and sources ### Mens email showed lower lift after a $100 outcome cap Take Rate’s Hillstrom replay found 0.77 USD more raw spend per Mens email assignment versus no email and 0.42 USD more after capping each customer’s outcome at $100. Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge (https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html) 64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract assignment-level averages on raw spend and min(spend,100). ### The $250 cap retained more of the estimated lift In Hillstrom’s two-week trial, the revenue difference at a $250 cap was 0.66 USD per Mens email assignment and 0.39 USD per Womens email assignment versus no email. Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge (https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html) 64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract the corresponding holdout mean using min(spend,250). ### Spend above $100 was infrequent among assignments In the Hillstrom retail test, 94 of 21,307 Mens email assignments, 92 of 21,387 Womens email assignments and 48 of 21,306 no-email assignments exceeded $100 in two-week spend. Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge (https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html) 64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Count individual outcomes strictly above $100 by randomized arm. ### Womens email lift remained positive on both definitions The Hillstrom Womens email arm added 0.42 USD per assignment in raw spend and 0.23 USD after a $100 per-customer cap, both relative to no email. Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge (https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html) 64,000 existing customers in Hillstrom’s three-arm randomized retail email trial Subtract the corresponding no-email mean after applying the same outcome definition to both groups. ## Compare the results Figure description: For both emails, the revenue difference is smaller under a $100 cap on each recorded customer outcome. USD per assigned customer, over two weeks. Both arms use the same customer-level cap. ## Check what sits behind the assignment-level average Revenue per assignment keeps all shoppers and the full value of every order in the analysis. It also means a small set of high-spend customers can materially move the reported effect. The Mens email arm had 94 of 21,307 customers with spend above $100. The no-email arm had 48 of 21,306. Capping each person’s spend at $100 reduced total recorded spend from $30,311.69 to $17,870.62 in the Mens arm and from $13,908.33 to $8,854.76 in the holdout. Those are observed features of the distribution. ## Change the definition symmetrically and label the change A spend cap limits how much any one person can contribute to the mean. Apply the same cap to treatment and holdout, keep all zeroes and preserve assignment counts. Capped spend asks how much the cohort spent up to the cutoff. Uncapped spend retains the real recorded revenue. A smaller capped lift can reflect a real effect on big orders. Before treating any purchase as suspect, verify its payment, margin and return status using the actual order record. If the orders are valid, their value belongs in the raw business result. ## Report the diagnostic next to the financial decision For a checkout recovery test, predefine the primary outcome. Choose diagnostic caps using historical orders before reviewing trial outcomes. Report raw and capped differences with uncertainty intervals and the number of observations affected. If the financial result depends on a few orders, review whether that purchase pattern can repeat and whether the margins justify the program. Keeping raw and capped views side by side gives a more useful decision record than removing purchases after reading the test result. ## Study data Table: Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge Measure | Value | Context No email assigned customers | 21,306 customers | Original count or sum of public spend records No email uncapped spend | $13,908.33 | Original count or sum of public spend records No email spend capped at $100 | $8,854.76 | Original count or sum of public spend records No email spend capped at $250 | $12,439.08 | Original count or sum of public spend records No email customers above $100 | 48 customers | Original count or sum of public spend records No email customers above $250 | 9 customers | Original count or sum of public spend records No email maximum recorded spend | $499.00 | Original count or sum of public spend records Mens email assigned customers | 21,307 customers | Original count or sum of public spend records Mens email uncapped spend | $30,311.69 | Original count or sum of public spend records Mens email spend capped at $100 | $17,870.62 | Original count or sum of public spend records Mens email spend capped at $250 | $26,606.79 | Original count or sum of public spend records Mens email customers above $100 | 94 customers | Original count or sum of public spend records Mens email customers above $250 | 29 customers | Original count or sum of public spend records Mens email maximum recorded spend | $499.00 | Original count or sum of public spend records Womens email assigned customers | 21,387 customers | Original count or sum of public spend records Womens email uncapped spend | $23,038.11 | Original count or sum of public spend records Womens email spend capped at $100 | $13,776.46 | Original count or sum of public spend records Womens email spend capped at $250 | $20,746.70 | Original count or sum of public spend records Womens email customers above $100 | 92 customers | Original count or sum of public spend records Womens email customers above $250 | 21 customers | Original count or sum of public spend records Womens email maximum recorded spend | $499.00 | Original count or sum of public spend records Uncapped Mens email revenue lift | 0.77 USD per assigned customer | Original arm difference of assignment-level means Uncapped Womens email revenue lift | 0.42 USD per assigned customer | Original arm difference of assignment-level means Cap $100 Mens email revenue lift | 0.42 USD per assigned customer | Original arm difference of assignment-level means Cap $100 Womens email revenue lift | 0.23 USD per assigned customer | Original arm difference of assignment-level means Cap $250 Mens email revenue lift | 0.66 USD per assigned customer | Original arm difference of assignment-level means Cap $250 Womens email revenue lift | 0.39 USD per assigned customer | Original arm difference of assignment-level means ## What changes when you compare the figures Table: Calculations from the published values above. Comparison | Result | Calculation Uncapped Mens email revenue lift | 0.77 USD per assigned customer | = (30311.69 / 21307 - 13908.33 / 21306) * 1 Uncapped Womens email revenue lift | 0.42 USD per assigned customer | = (23038.11 / 21387 - 13908.33 / 21306) * 1 Cap $100 Mens email revenue lift | 0.42 USD per assigned customer | = (17870.62 / 21307 - 8854.76 / 21306) * 1 Cap $100 Womens email revenue lift | 0.23 USD per assigned customer | = (13776.46 / 21387 - 8854.76 / 21306) * 1 Cap $250 Mens email revenue lift | 0.66 USD per assigned customer | = (26606.79 / 21307 - 12439.08 / 21306) * 1 Cap $250 Womens email revenue lift | 0.39 USD per assigned customer | = (20746.7 / 21387 - 12439.08 / 21306) * 1 Mens email share of customers above $100 | 0.44% | = 94 / 21307 * 100 Mens email share of revenue above $100 cap | 41.04% | = 12441.07 / 30311.69 * 100 Womens email share of customers above $100 | 0.43% | = 92 / 21387 * 100 Womens email share of revenue above $100 cap | 40.2% | = 9261.65 / 23038.11 * 100 ## Methodology and sources Use all 64,000 randomized customers, retaining the zero-spend rows. Compute average recorded two-week spend for each arm, then recalculate it using min(spend,100) and min(spend,250) per customer. Subtract the no-email mean under the matching definition. The thresholds are illustrative sensitivity checks and keep every person in the sample. [Download the source data (CSV)](decision-table.csv) [Download the calculations (CSV)](calculations.csv) [Download the citation notes (CSV)](citation-notes.csv) [Download the evidence behind the new findings (CSV)](finding-evidence.csv) [Download the reproducible analysis script](replay_hillstrom_link_assets.py) [Download the source-labeled chart (SVG)](evidence-chart.svg) ## Related reading Email holdout statistics by prior purchase channel (https://takeratehq.com/guides/email-holdout-lift-by-purchase-channel-statistics/) Incremental advertising return statistics and what a holdout can tell you (https://takeratehq.com/guides/incremental-advertising-return-statistics/) Sample ratio mismatch statistics for checkout A/B tests (https://takeratehq.com/guides/sample-ratio-mismatch-statistics/) A/B test peeking statistics for checkout experiments (https://takeratehq.com/guides/ab-test-peeking-statistics/) When refunds turn an iMessage recovery win into a loss (https://takeratehq.com/guides/refund-adjusted-recovery-revenue/) Can you trust the lift if your iMessage holdout got the extra text? (https://takeratehq.com/guides/imessage-recovery-holdout-exposure/) Browse checkout recovery research (https://takeratehq.com/research/) ## Sources and definitions Kevin Hillstrom, MineThatData E-Mail Analytics and Data Mining Challenge (https://blog.minethatdata.com/2008/03/minethatdata-e-mail-analytics-and-data.html) The source data includes the population, period, denominator and limitations for every figure. Take Rate · takeratehq.com