A/B test duration statistics for one-week and two-week results
https://takeratehq.com/guides/a-b-test-duration-statistics-for-one-week-and-two-week-results/
## Quick answer
A/B test duration statistics show that another week can change more than precision. Microsoft’s November 2024 ExP analysis found that standard errors increased in 17% of qualifying metric movements when 7-day scorecards became 14-day scorecards. The later estimate fell outside the original 95% interval in 9% of cases (Microsoft ExP (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024).
A store needs to know whether a measured gain will persist under different operating conditions. This analysis separates that question from repeated significance checking. It preserves Microsoft’s selected sample and overlapping windows, then reproduces a stable-mean probability calculation to explain why the observed changes deserve attention without turning them into ecommerce failure rates.
## Key Takeaways
A second week widened standard errors in 17% of cases. Microsoft’s selected metric sample did not always become more precise with more time (Microsoft ExP (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024).
9% of 14-day estimates left the original 95% interval. These are overlapping cumulative estimates, rather than independent experiments (Microsoft ExP (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024).
Temporal instability was not explained only by novelty. The researchers observed surprises with and without clear monotonic trends (Microsoft ExP (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024).
## Why can more data produce wider error bars?
More time can change the mix of users and the conditions under which a feature operates. If those changes affect the treatment response, a second week is not simply another independent sample from an unchanged population.
Figure description: A/B test duration statistics worked model: first-week effect plus 0.20 percentage points, second-week effect negative 0.20 points and equal-weight cumulative effect zero, illustrating that the cumulative window is not a replication.
Figure 1: A cumulative result can average away a reversal. Source: Microsoft ExP, temporal-stability question (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024. Original explanation and arithmetic from the cited primary research.
### What population did Microsoft inspect?
Microsoft’s ExP report examined experiments with both 7-day and 14-day scorecards over a month. The main analysis restricted metric movements to those significant at the 5% level after 7 days and excluded metrics that intrinsically accumulate with time. Standard errors narrowed in 83% of cases and widened in 17% (Microsoft ExP (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024).
The unit is a qualifying metric movement, not a store, shopper or independent experiment. Several metrics from one experiment can share users and causes. The report says its qualitative conclusion also held without the significance restriction, but it does not publish a full merchant-level raw dataset.
### What does the second week test?
It tests whether the measured effect remains useful under another week of operating conditions. That is a temporal transport question. A technically correct estimate for one week can fail to describe the next week when traffic composition, user learning, promotions or other product changes alter the response.
For a merchant, this means a valid first-week result and a reliable ongoing revenue forecast are related but different judgments. It does not mean that every test should run until a favorable second week appears.
## How surprising were the later estimates?
The report found that 9% of 14-day estimates fell outside the earlier 95% interval, and 4% fell outside the earlier 3-sigma interval. The 14-day estimate contains the first week’s data, so treating the two estimates as independent would give the wrong theoretical comparison (Microsoft ExP (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024).
### What can be calculated under a stable model?
If both weeks have the same number of independent observations, an unchanged mean effect and equal variance, the difference between the 14-day and 7-day mean estimates has a standard deviation equal to the original standard error divided by the square root of 2. The normal-model escape probabilities are approximately 0.56% for the original 95% interval and 0.0022% for the original 3-sigma interval. These are our reproducible model calculations, not new Microsoft measurements.
These reference probabilities describe an unselected stable model. They do not reproduce Microsoft’s screening for significant first-week results. Selection, shared users and changing conditions all affect the comparison, so the observed 9% and 4% cannot be converted into a calibrated excess-failure rate using this calculation.
### Is novelty the only explanation?
No. Microsoft examined monotonic trends using Kendall’s tau. Some surprises were consistent with learning, novelty or adoption, while others occurred in series without a clear trend. The report explicitly treats those mechanisms as a mixture (Microsoft ExP (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024).
A checkout redesign may also interact with a weekend, a promotion or the customers returning after their first exposure. A chart that rises steadily and one that jumps with traffic mix require different diagnosis, even if their first-week aggregate estimates look similar.
## What should a merchant decide before extending a test?
Set the decision window and the rule for interpreting longer observation before looking at the second-week winner. Keep assignment persistent, inspect daily effects and preserve mature order and return outcomes. The reason to extend is to resolve a decision-relevant uncertainty, not to search for significance.
### What would justify additional observation?
An effect that reverses across user cohorts, large day-to-day movements beyond expected uncertainty or an intervention that changes repeat behavior can justify a longer planned observation. A holdout after rollout can also test whether the response persists. These options are useful when valid measurement can continue without changing the intervention.
### What limits the analogy to DTC?
Microsoft’s population includes software product experiments. Checkout traffic, purchase frequency and return maturity differ. The published rates are evidence that temporal stability deserves a separate check, rather than a recommended test duration or an expected DTC failure percentage. A store must choose its window around its own purchase cycle and assignment unit.
## Can a positive two-week result conceal a failed second week?
Yes. With equal eligible exposure in both weeks and equal week weights in both randomized arms, a two-week conversion-rate difference is the average of the two separate weekly differences. A worked first-week gain of 0.20 percentage points followed by no second-week difference produces a positive cumulative difference of 0.10 points. The cumulative result remains positive even though the second week adds no gain. A second-week difference of negative 0.20 points would instead bring the two-week cumulative difference to zero.
Worked first week | Worked second week | Equal-weight cumulative difference
+0.20 points | +0.20 points | +0.20 points
+0.20 points | 0.00 points | +0.10 points
+0.20 points | −0.20 points | 0.00 points
These examples are algebra, not Microsoft experiment observations. They show why overlapping seven-day and fourteen-day estimates are not independent replications. The source reports temporal surprises in a selected metric population; the worked examples explain one way a cumulative estimate can conceal a change. (Microsoft ExP (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/), Nov 2024).
### Can the second-week effect be reconstructed?
Under the equal-weight assumptions, the separate second-week difference is 2 × cumulative difference − first-week difference. For a first-week 0.20-point result and a cumulative 0.10-point result, that gives zero in the second week. If exposures differ between weeks or arms, recover each arm's weekly event and denominator totals instead. Subtracting two rounded lift percentages or applying this identity to ratios does not supply a valid weekly estimate.
An unequal-weight example makes the distinction concrete. If the first week contributes 40% of each arm's observations and the second contributes 60%, a first-week 0.20-point effect and second-week negative 0.20-point effect combine to negative 0.04 points, rather than zero. A traffic surge can therefore change the cumulative result even with the same two period-specific effects.
### What should the diagnostic view preserve?
Show separate nonoverlapping period results beside cumulative results, with their exposure counts and the same assignment rule. Inspect new versus returning users where the question concerns learning or repeated use. Record promotions, inventory changes and relevant interface changes before assigning a mechanism. A cumulative effect fading toward zero can reflect novelty, a different traffic mix or sampling noise; its shape alone does not identify which one.
Do not independently test every daily movement and then select a favorable stopping date. The purpose of the period view is diagnosis under a planned analysis, not another chance to declare significance. A longer observation can improve precision under stable conditions while still answering a different operating-period question.
The cumulative-window arithmetic (https://takeratehq.com/guides/a-b-test-duration-statistics-for-one-week-and-two-week-results/data/weekly-cumulative-model.csv) and calculation (https://takeratehq.com/guides/a-b-test-duration-statistics-for-one-week-and-two-week-results/analysis.py) reproduce all examples. They do not recreate the source's selection process or justify a universal fourteen-day runtime. The merchant's decision still requires its own outcome maturity and relevant operating cycles.
## Methodology and Sources
We read Microsoft’s 20 November 2024 ExP report and preserved the source text on 13 November 2026. The data export transcribes its selected noncumulative-metric results, not independent shopper observations. The script uses the complementary error function erfc(z) to calculate the probability that an equal-size two-week cumulative mean falls outside a first-week interval under a stable independent mean model. Those assumptions are stated and are not asserted for the observed software experiments. Counts of qualifying metrics and experiment-level dependence are not published in a form supporting new confidence intervals, so we do not calculate them. Temporal stability, internal validity and merchant economics remain distinct.
## Glossary
Temporal transport: Applying an estimate from one time period to another.
Standard error: The estimated sampling uncertainty of a statistic.
Cumulative estimate: An estimate calculated using all observations through a given date.
Novelty effect: A response that changes as users become familiar with an intervention.
## Sources
Microsoft ExP: External Validity of Online Experiments (https://www.microsoft.com/en-us/research/articles/external-validity-of-online-experiments-can-we-predict-the-future/)
Quick answer
A/B test duration statistics show that another week can change more than precision. Microsoft’s November 2024 ExP analysis found that standard errors increased in 17% of qualifying metric movements when 7-day scorecards became 14-day scorecards. The later estimate fell outside the original 95% interval in 9% of cases (Microsoft ExP, Nov 2024).
A store needs to know whether a measured gain will persist under different operating conditions. This analysis separates that question from repeated significance checking. It preserves Microsoft’s selected sample and overlapping windows, then reproduces a stable-mean probability calculation to explain why the observed changes deserve attention without turning them into ecommerce failure rates.
A second week widened standard errors in 17% of cases. Microsoft’s selected metric sample did not always become more precise with more time (Microsoft ExP, Nov 2024).
9% of 14-day estimates left the original 95% interval. These are overlapping cumulative estimates, rather than independent experiments (Microsoft ExP, Nov 2024).
Temporal instability was not explained only by novelty. The researchers observed surprises with and without clear monotonic trends (Microsoft ExP, Nov 2024).
More time can change the mix of users and the conditions under which a feature operates. If those changes affect the treatment response, a second week is not simply another independent sample from an unchanged population.
Figure 1: A cumulative result can average away a reversal. Source: Microsoft ExP, temporal-stability question, Nov 2024. Original explanation and arithmetic from the cited primary research.
Microsoft’s ExP report examined experiments with both 7-day and 14-day scorecards over a month. The main analysis restricted metric movements to those significant at the 5% level after 7 days and excluded metrics that intrinsically accumulate with time. Standard errors narrowed in 83% of cases and widened in 17% (Microsoft ExP, Nov 2024).
The unit is a qualifying metric movement, not a store, shopper or independent experiment. Several metrics from one experiment can share users and causes. The report says its qualitative conclusion also held without the significance restriction, but it does not publish a full merchant-level raw dataset.
It tests whether the measured effect remains useful under another week of operating conditions. That is a temporal transport question. A technically correct estimate for one week can fail to describe the next week when traffic composition, user learning, promotions or other product changes alter the response.
For a merchant, this means a valid first-week result and a reliable ongoing revenue forecast are related but different judgments. It does not mean that every test should run until a favorable second week appears.
The report found that 9% of 14-day estimates fell outside the earlier 95% interval, and 4% fell outside the earlier 3-sigma interval. The 14-day estimate contains the first week’s data, so treating the two estimates as independent would give the wrong theoretical comparison (Microsoft ExP, Nov 2024).
If both weeks have the same number of independent observations, an unchanged mean effect and equal variance, the difference between the 14-day and 7-day mean estimates has a standard deviation equal to the original standard error divided by the square root of 2. The normal-model escape probabilities are approximately 0.56% for the original 95% interval and 0.0022% for the original 3-sigma interval. These are our reproducible model calculations, not new Microsoft measurements.
These reference probabilities describe an unselected stable model. They do not reproduce Microsoft’s screening for significant first-week results. Selection, shared users and changing conditions all affect the comparison, so the observed 9% and 4% cannot be converted into a calibrated excess-failure rate using this calculation.
No. Microsoft examined monotonic trends using Kendall’s tau. Some surprises were consistent with learning, novelty or adoption, while others occurred in series without a clear trend. The report explicitly treats those mechanisms as a mixture (Microsoft ExP, Nov 2024).
A checkout redesign may also interact with a weekend, a promotion or the customers returning after their first exposure. A chart that rises steadily and one that jumps with traffic mix require different diagnosis, even if their first-week aggregate estimates look similar.
Set the decision window and the rule for interpreting longer observation before looking at the second-week winner. Keep assignment persistent, inspect daily effects and preserve mature order and return outcomes. The reason to extend is to resolve a decision-relevant uncertainty, not to search for significance.
An effect that reverses across user cohorts, large day-to-day movements beyond expected uncertainty or an intervention that changes repeat behavior can justify a longer planned observation. A holdout after rollout can also test whether the response persists. These options are useful when valid measurement can continue without changing the intervention.
Microsoft’s population includes software product experiments. Checkout traffic, purchase frequency and return maturity differ. The published rates are evidence that temporal stability deserves a separate check, rather than a recommended test duration or an expected DTC failure percentage. A store must choose its window around its own purchase cycle and assignment unit.
Yes. With equal eligible exposure in both weeks and equal week weights in both randomized arms, a two-week conversion-rate difference is the average of the two separate weekly differences. A worked first-week gain of 0.20 percentage points followed by no second-week difference produces a positive cumulative difference of 0.10 points. The cumulative result remains positive even though the second week adds no gain. A second-week difference of negative 0.20 points would instead bring the two-week cumulative difference to zero.
Worked first week
Worked second week
Equal-weight cumulative difference
+0.20 points
+0.20 points
+0.20 points
+0.20 points
0.00 points
+0.10 points
+0.20 points
−0.20 points
0.00 points
These examples are algebra, not Microsoft experiment observations. They show why overlapping seven-day and fourteen-day estimates are not independent replications. The source reports temporal surprises in a selected metric population; the worked examples explain one way a cumulative estimate can conceal a change. (Microsoft ExP, Nov 2024).
Under the equal-weight assumptions, the separate second-week difference is 2 × cumulative difference − first-week difference. For a first-week 0.20-point result and a cumulative 0.10-point result, that gives zero in the second week. If exposures differ between weeks or arms, recover each arm's weekly event and denominator totals instead. Subtracting two rounded lift percentages or applying this identity to ratios does not supply a valid weekly estimate.
An unequal-weight example makes the distinction concrete. If the first week contributes 40% of each arm's observations and the second contributes 60%, a first-week 0.20-point effect and second-week negative 0.20-point effect combine to negative 0.04 points, rather than zero. A traffic surge can therefore change the cumulative result even with the same two period-specific effects.
Show separate nonoverlapping period results beside cumulative results, with their exposure counts and the same assignment rule. Inspect new versus returning users where the question concerns learning or repeated use. Record promotions, inventory changes and relevant interface changes before assigning a mechanism. A cumulative effect fading toward zero can reflect novelty, a different traffic mix or sampling noise; its shape alone does not identify which one.
Do not independently test every daily movement and then select a favorable stopping date. The purpose of the period view is diagnosis under a planned analysis, not another chance to declare significance. A longer observation can improve precision under stable conditions while still answering a different operating-period question.
The cumulative-window arithmetic and calculation reproduce all examples. They do not recreate the source's selection process or justify a universal fourteen-day runtime. The merchant's decision still requires its own outcome maturity and relevant operating cycles.
We read Microsoft’s 20 November 2024 ExP report and preserved the source text on 13 November 2026. The data export transcribes its selected noncumulative-metric results, not independent shopper observations. The script uses the complementary error function erfc(z) to calculate the probability that an equal-size two-week cumulative mean falls outside a first-week interval under a stable independent mean model. Those assumptions are stated and are not asserted for the observed software experiments. Counts of qualifying metrics and experiment-level dependence are not published in a form supporting new confidence intervals, so we do not calculate them. Temporal stability, internal validity and merchant economics remain distinct.
Use the citation below when referring to this analysis. Source populations and calculation limits accompany each finding.
Take Rate. "A/B test duration statistics for one-week and two-week results." Take Rate, November 2026. https://takeratehq.com/guides/a-b-test-duration-statistics-for-one-week-and-two-week-results/
<a href="https://takeratehq.com/guides/a-b-test-duration-statistics-for-one-week-and-two-week-results/">A/B test duration statistics for one-week and two-week results</a> (Take Rate, November 2026)