Checkout holdout statistics for repeat order counts and customer variance By Take Rate, Take Rate. Sources checked 2026-11-11. A retail ledger replay shows how treating purchase events as evenly dispersed can make a repeat-order test look more precise than it is. ## Quick answer In a public UK retail cohort, the variance of 90-day repeat purchase days was 1.92 times the mean. For a worked holdout of 774 versus 775 customers, using observed customer variance gave a standard error of 5.96 purchase days per 100 customers, compared with 4.30 under a count model that equates variance and mean. Daqing Chen, Online Retail, UCI Machine Learning Repository (https://archive.ics.uci.edu/dataset/352/online%2Bretail) ## Key findings and sources ### Repeat purchase day variance was 1.92 times the mean Take Rate’s analysis of 1,549 UCI Online Retail customers found a 90-day repeat-purchase-day variance of 1.373 against a mean of 0.715, a variance-to-mean ratio of 1.92. Daqing Chen, Online Retail, UCI Machine Learning Repository (https://archive.ics.uci.edu/dataset/352/online%2Bretail) 1,549 identified customers at one historical UK online retailer, 90 days after their first observed purchase day Calculate the mean and sample variance over each customer’s later-day count, including zeros. ### The same worked holdout had standard errors of 5.96 and 4.30 Using the observed UCI customer variance, a worked comparison with 774 and 775 customers gave 5.96 purchase days per 100 customers as the standard error. Equating variance with the mean gave 4.30. Daqing Chen, Online Retail, UCI Machine Learning Repository (https://archive.ics.uci.edu/dataset/352/online%2Bretail) 1,549 identified customers at one historical UK online retailer, 90 days after their first observed purchase day Use identical arm sizes, substituting the observed customer variance or the Poisson mean for the variance term. ### Repeat purchase activity concentrated in customers with multiple days In the UCI first observed cohort, people with two or more later purchase days contributed 700 of 1,107 later days, or 63.23%. There were 892 customers with no later day. Daqing Chen, Online Retail, UCI Machine Learning Repository (https://archive.ics.uci.edu/dataset/352/online%2Bretail) 1,549 identified customers at one historical UK online retailer, 90 days after their first observed purchase day Count distinct later dates per customer and subtract 407 days contributed by customers with exactly one. ### The empirical standard error was 38.62% above the count-model estimate In Take Rate’s worked holdout using the UCI cohort, the standard error based on observed customer variance was 38.62% higher than the estimate using the equal-mean-and-variance count assumption. Daqing Chen, Online Retail, UCI Machine Learning Repository (https://archive.ics.uci.edu/dataset/352/online%2Bretail) 1,549 identified customers at one historical UK online retailer, 90 days after their first observed purchase day Divide standard errors on the unrounded scale and subtract one. ## Compare the results Figure description: Two precision assumptions for the same holdout. Exact values are also in the study data. Two precision assumptions for the same holdout. ## The customers doing the most buying also contribute to the noise The 1,549 customers had 1,107 later purchase days over 90 days. More than half had none. The 407 who had exactly one contributed 407 days, while customers with at least two contributed 700 days, or 63.23% of the total. Those counts are why the spread between customers matters for the precision of a holdout. The average was 0.715 later days per customer and the sample variance was 1.373. A Poisson count model that uses the mean as its variance is a convenient starting assumption. Here, the observed variance was 1.92 times that starting value. An experiment can look conclusive too soon if the planning assumption smooths out real differences between customers. ## Translate the variance into the decision you are trying to make For arms of 774 and 775, the estimated standard error was 5.96 later purchase days per 100 assigned customers using the observed variance. The equal-variance count model gave 4.30 on the same scale. That is a 38.62% difference in standard error before any message has been sent. Neither figure is a measured lift. An example of a result that needs care is a lift of five additional days per 100 assigned shoppers. It is near one standard error on the scale seen here. The merchant should pick a meaningful minimum effect, then use historical customer-level variance from the actual eligible group to plan how many assignments and how much time are needed. ## Keep the assignment and counting units aligned If a shopper starts two carts, preserve one assignment before messaging and retain their full order history for the outcome window. Avoid counting each cart or purchase as a separate randomized person. Someone who never buys still belongs in the analysis with zero. If the program randomizes stores or households, the analysis needs to handle clustering at that unit. Compare paid orders, contribution and returns separately when the decision warrants it. A team can check whether order-count spread is stable across acquisition cohorts before finalizing its design. If one purchase during a sales event changes the count distribution, size the test using an eligible cohort that looks like the one the campaign will reach. ## Study data Table: The full UCI Online Retail cohort Measure | Value | Context Eligible first observed customers | 1,549 customers | Original count Repeat purchase days across the cohort | 1,107 customer purchase days | Original count Mean later purchase days | 0.714655 days per customer | Original mean Variance of later purchase days | 1.373306 days squared | Original sample variance No repeat purchase days | 892 customers | Original frequency One repeat purchase day | 407 customers | Original frequency Two repeat purchase days | 156 customers | Original frequency Three repeat purchase days | 48 customers | Original frequency Four or more repeat purchase days | 46 customers | Original frequency Standard error using observed customer variance | 5.96 purchase days per 100 customers | Original standard error calculation Standard error under an equal-mean-and-variance count model | 4.3 purchase days per 100 customers | Original standard error calculation Repeat days contributed by people with at least two | 700 customer purchase days | Original count ## What changes when you compare the figures Table: Calculations from the published values above. Comparison | Result | Calculation Variance relative to the mean | 1.92 times | = 1.37330578 / 0.71465462 * 1 Share of assigned profiles with zero repeat days | 57.59% | = 892 / 1549 * 100 Share of days from customers with at least two | 63.23% | = 700 / 1107 * 100 Standard error under observed variance | 5.96 purchase days per 100 customers | = (1.3733057753 * (1 / 774 + 1 / 775))**0.5 * 100 Standard error under the equal-variance count model | 4.3 purchase days per 100 customers | = (0.7146546159 * (1 / 774 + 1 / 775))**0.5 * 100 Standard error increase over the count model | 38.62% | = ((1.3733057753 / 0.7146546159)**0.5 - 1) * 100 ## Methodology and sources Count distinct positive purchase days in the 90 days after each customer’s first observed purchase day. Keep everyone with an identified positive purchase day from January through April 2011, including the 892 with no later days. Calculate the mean and sample variance of the customer counts. For hypothetical arms of 774 and 775 customers with the same underlying variance, use the square root of variance times the sum of reciprocal arm sizes. Substitute the mean for the variance in the Poisson planning comparison. The workbook has no experiment assignments. [Download the source data (CSV)](decision-table.csv) [Download the calculations (CSV)](calculations.csv) [Download the citation notes (CSV)](citation-notes.csv) [Download the evidence behind the new findings (CSV)](finding-evidence.csv) [Download the reproducible analysis script](retail_repeat_units.py) ## Related reading Incremental advertising return statistics and what a holdout can tell you (https://takeratehq.com/guides/incremental-advertising-return-statistics/) Sample ratio mismatch statistics for checkout A/B tests (https://takeratehq.com/guides/sample-ratio-mismatch-statistics/) A/B test peeking statistics for checkout experiments (https://takeratehq.com/guides/ab-test-peeking-statistics/) Email holdout statistics for visits, purchases and revenue (https://takeratehq.com/guides/email-holdout-purchase-and-revenue-statistics/) Experiment variance statistics for pre-test and post-test retail spend (https://takeratehq.com/guides/experiment-variance-reduction-spend-statistics/) When refunds turn an iMessage recovery win into a loss (https://takeratehq.com/guides/refund-adjusted-recovery-revenue/) Can you trust the lift if your iMessage holdout got the extra text? (https://takeratehq.com/guides/imessage-recovery-holdout-exposure/) Browse checkout recovery research (https://takeratehq.com/research/) ## Sources and definitions Daqing Chen, Online Retail, UCI Machine Learning Repository (https://archive.ics.uci.edu/dataset/352/online%2Bretail) The source data includes the population, period, denominator and limitations for every figure. Take Rate · takeratehq.com