Take Rate Take Rate PERFORMANCE RECOVERY

Checkout holdout statistics for repeat order counts and customer variance

A retail ledger replay shows how treating purchase events as evenly dispersed can make a repeat-order test look more precise than it is.

Two precision assumptions for the same holdout

5.96 purchase days per 100 customers
Observed variance
4.3 purchase days per 100 customers
Equal mean and variance
A planning comparison on identical arm sizes. Values are standard errors, expressed as purchase days per 100 customers.
About 5 min read
Updated:
Statistics
Learn about our Editorial policyLearn about our Research methodology

Sources checked 11 November 2026.

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Full article, downloads and prompt

Uses a selected article brief with sources. ChatGPT and Perplexity send the brief and start a request. Claude may need sign-in or paste; Gemini uses copy/paste. Full article below.

Download articleDownload AI source digest

Copy the complete article or download it for later.

Contents

Quick answer

In a public UK retail cohort, the variance of 90-day repeat purchase days was 1.92 times the mean. For a worked holdout of 774 versus 775 customers, using observed customer variance gave a standard error of 5.96 purchase days per 100 customers, compared with 4.30 under a count model that equates variance and mean.

Daqing Chen, Online Retail, UCI Machine Learning Repository

Key findings and sources

Repeat purchase day variance was 1.92 times the mean

Take Rate’s analysis of 1,549 UCI Online Retail customers found a 90-day repeat-purchase-day variance of 1.373 against a mean of 0.715, a variance-to-mean ratio of 1.92.

Daqing Chen, Online Retail, UCI Machine Learning Repository

1,549 identified customers at one historical UK online retailer, 90 days after their first observed purchase day Calculate the mean and sample variance over each customer’s later-day count, including zeros.

The same worked holdout had standard errors of 5.96 and 4.30

Using the observed UCI customer variance, a worked comparison with 774 and 775 customers gave 5.96 purchase days per 100 customers as the standard error. Equating variance with the mean gave 4.30.

Daqing Chen, Online Retail, UCI Machine Learning Repository

1,549 identified customers at one historical UK online retailer, 90 days after their first observed purchase day Use identical arm sizes, substituting the observed customer variance or the Poisson mean for the variance term.

Repeat purchase activity concentrated in customers with multiple days

In the UCI first observed cohort, people with two or more later purchase days contributed 700 of 1,107 later days, or 63.23%. There were 892 customers with no later day.

Daqing Chen, Online Retail, UCI Machine Learning Repository

1,549 identified customers at one historical UK online retailer, 90 days after their first observed purchase day Count distinct later dates per customer and subtract 407 days contributed by customers with exactly one.

The empirical standard error was 38.62% above the count-model estimate

In Take Rate’s worked holdout using the UCI cohort, the standard error based on observed customer variance was 38.62% higher than the estimate using the equal-mean-and-variance count assumption.

Daqing Chen, Online Retail, UCI Machine Learning Repository

1,549 identified customers at one historical UK online retailer, 90 days after their first observed purchase day Divide standard errors on the unrounded scale and subtract one.

Compare the results

Observed variance implies more uncertainty in the same holdout

The same holdout looks less precise when you account for how repeat purchases vary across customers.

5.96 days
Observed customer variance
4.3 days
Equal mean and variance
Public UK retail cohort. Worked holdout of 774 versus 775 customers, with both standard errors in purchase days per 100 customers. Daqing Chen, Online Retail, UCI Machine Learning Repository.

The customers doing the most buying also contribute to the noise

The 1,549 customers had 1,107 later purchase days over 90 days. More than half had none. The 407 who had exactly one contributed 407 days, while customers with at least two contributed 700 days, or 63.23% of the total. Those counts are why the spread between customers matters for the precision of a holdout.

The average was 0.715 later days per customer and the sample variance was 1.373. A Poisson count model that uses the mean as its variance is a convenient starting assumption. Here, the observed variance was 1.92 times that starting value. An experiment can look conclusive too soon if the planning assumption smooths out real differences between customers.

Translate the variance into the decision you are trying to make

For arms of 774 and 775, the estimated standard error was 5.96 later purchase days per 100 assigned customers using the observed variance. The equal-variance count model gave 4.30 on the same scale. That is a 38.62% difference in standard error before any message has been sent. Neither figure is a measured lift.

An example of a result that needs care is a lift of five additional days per 100 assigned shoppers. It is near one standard error on the scale seen here. The merchant should pick a meaningful minimum effect, then use historical customer-level variance from the actual eligible group to plan how many assignments and how much time are needed.

Keep the assignment and counting units aligned

If a shopper starts two carts, preserve one assignment before messaging and retain their full order history for the outcome window. Avoid counting each cart or purchase as a separate randomized person. Someone who never buys still belongs in the analysis with zero. If the program randomizes stores or households, the analysis needs to handle clustering at that unit.

Compare paid orders, contribution and returns separately when the decision warrants it. A team can check whether order-count spread is stable across acquisition cohorts before finalizing its design. If one purchase during a sales event changes the count distribution, size the test using an eligible cohort that looks like the one the campaign will reach.

Study data

The full UCI Online Retail cohort
Measure Value Context
Eligible first observed customers 1,549 customers Original count
Repeat purchase days across the cohort 1,107 customer purchase days Original count
Mean later purchase days 0.714655 days per customer Original mean
Variance of later purchase days 1.373306 days squared Original sample variance
No repeat purchase days 892 customers Original frequency
One repeat purchase day 407 customers Original frequency
Two repeat purchase days 156 customers Original frequency
Three repeat purchase days 48 customers Original frequency
Four or more repeat purchase days 46 customers Original frequency
Standard error using observed customer variance 5.96 purchase days per 100 customers Original standard error calculation
Standard error under an equal-mean-and-variance count model 4.3 purchase days per 100 customers Original standard error calculation
Repeat days contributed by people with at least two 700 customer purchase days Original count

What changes when you compare the figures

Calculations from the published values above.
Comparison Result Calculation
Variance relative to the mean 1.92 times = 1.37330578 / 0.71465462 * 1
Share of assigned profiles with zero repeat days 57.59% = 892 / 1549 * 100
Share of days from customers with at least two 63.23% = 700 / 1107 * 100
Standard error under observed variance 5.96 purchase days per 100 customers = (1.3733057753 * (1 / 774 + 1 / 775))**0.5 * 100
Standard error under the equal-variance count model 4.3 purchase days per 100 customers = (0.7146546159 * (1 / 774 + 1 / 775))**0.5 * 100
Standard error increase over the count model 38.62% = ((1.3733057753 / 0.7146546159)**0.5 - 1) * 100

Methodology and sources

Count distinct positive purchase days in the 90 days after each customer’s first observed purchase day. Keep everyone with an identified positive purchase day from January through April 2011, including the 892 with no later days. Calculate the mean and sample variance of the customer counts. For hypothetical arms of 774 and 775 customers with the same underlying variance, use the square root of variance times the sum of reciprocal arm sizes. Substitute the mean for the variance in the Poisson planning comparison. The workbook has no experiment assignments.

Download the source data (CSV)

Download the calculations (CSV)

Download the citation notes (CSV)

Download the evidence behind the new findings (CSV)

Download the reproducible analysis script

Incremental advertising return statistics and what a holdout can tell you

Sample ratio mismatch statistics for checkout A/B tests

A/B test peeking statistics for checkout experiments

Email holdout statistics for visits, purchases and revenue

Experiment variance statistics for pre-test and post-test retail spend

Browse checkout recovery research

Sources and definitions

Daqing Chen, Online Retail, UCI Machine Learning Repository

The source data includes the population, period, denominator and limitations for every figure.

About the publisher

Take Rate Take Rate PERFORMANCE RECOVERY

Research and practical guides for DTC merchants.