Branded paid search incrementality statistics from the eBay experiment https://takeratehq.com/guides/branded-paid-search-incrementality-statistics-from-the-ebay-experiment/ ## Quick answer Branded paid search incrementality statistics can look very different from paid-link attribution. In eBay’s March 2012 MSN branded-search test, the adjusted ad-off coefficient implied that about 99.5% of click traffic was retained as organic links absorbed much of the change. The coefficient’s standard error was large enough for an approximate 95% interval to include no traffic change (Blake, Nosko and Tadelis (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). This analysis recalculates the click ratio and uncertainty from the May 2014 working paper. It keeps branded search separate from non-brand acquisition, and total traffic separate from attributed sales. The merchant decision is whether paying for a branded query adds a sale that would otherwise not occur, under the store’s current organic and competitive conditions. ## Key Takeaways About 99.5% of branded search traffic was retained. The adjusted MSN ad-off coefficient was −0.00529, with standard error 0.0177 (Blake, Nosko and Tadelis (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). The simple before and after comparison looked different. Its coefficient was −0.0560 before the researchers controlled for platform and seasonal changes (Blake, Nosko and Tadelis (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). Non-brand findings should stay separate. The national geographic experiment used 68 ad-off regions, 65 matched controls and 77 other regions (Blake, Nosko and Tadelis (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). ## What happened when branded ads were paused? Most of the branded search traffic remained. Shoppers could choose an organic result when the paid result disappeared, so a reduction in paid clicks did not imply the same reduction in visits to the business. Figure description: Branded paid search incrementality statistics: simple before-and-after total-click change is approximately negative 5.45%, while adjusted change is negative 0.53% with an interval spanning negative 3.92% to positive 2.98%. Figure 1: Adjustment changes both the estimate and the decision. Source: Blake, Nosko and Tadelis, Table 1 (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014. Original explanation and arithmetic from the cited primary research. ### How much traffic was retained? In the March 2012 MSN branded-search test, the adjusted coefficient on the ad-off interaction was −0.00529 with standard error 0.0177. Exponentiating the coefficient gives a retained-traffic ratio of approximately 99.47% relative to the comparison trajectory. The authors round this to 99.5% retained traffic (Blake, Nosko and Tadelis (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). Using the coefficient plus or minus 1.96 standard errors gives an approximate 95% interval for the change in total clicks of −3.92% to +2.98%. This is our arithmetic from the published log coefficient and standard error. The range includes no change. It is not a new estimate from raw click records, and it does not establish zero advertising effect. ### Why did the simple comparison look larger? Table 1's simple MSN before and after coefficient is −0.0560. That corresponds to roughly 5.45% fewer clicks, using the same exponential transformation. The adjusted interaction corresponds to roughly 0.53% fewer clicks. The comparison illustrates why an apparent fall after pausing ads can include seasonality and other platform movement (Blake, Nosko and Tadelis (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). The result is about total branded-search click traffic in this test. It is not a 99.5% customer retention rate, a 99.5% order rate or a universal estimate of paid-search waste. ## Does the result apply to non-brand acquisition? No single brand-query comparison can answer a non-brand acquisition question. A shopper searching a known brand is different from someone searching for a product category without a chosen seller. ### What did the broader experiment add? The researchers' non-brand geographic test suspended ads in 68 regions and compared their outcomes with 65 matched control regions and 77 other regions. Paid-search attributed sales fell by more than 72%, while the total-sales difference was much smaller. The paper’s discussion emphasizes positive responses among new and infrequent users and weak responses among frequent buyers (Blake, Nosko and Tadelis (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). This is useful counterevidence to a blanket instruction to stop search advertising. Brand-query displacement and acquiring shoppers unfamiliar with the assortment are different mechanisms. The appropriate local test would keep them separate at commissioning, rather than discover the segmentation after choosing a favorable result. ### How strong is the branded-search causal claim? The MSN analysis uses other search platforms as a comparison and adjusts for dates and platform trends. It is not a randomized user holdout. The later Google branded-search pause had no viable contemporaneous control group, and the paper explicitly warns that its estimated traffic decline could be reduced by a better comparison (Blake, Nosko and Tadelis (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). The non-brand geographic assignment also used historical matching rather than a purely random final test allocation. Preserve those designs when citing the paper. Calling every table a randomized controlled trial would overstate what the branded-search coefficient establishes. ## How should a DTC merchant test branded search? Define the decision as incremental paid orders or contribution, alongside the diagnostic traffic movement. Track paid brand clicks, organic brand clicks and total orders separately. If paid clicks fall while organic clicks rise, the experiment needs the combined outcome to distinguish displacement from lost demand. ### Which conditions make the eBay result hard to transfer? A small merchant may have weak organic visibility, competitors bidding on the same query or customers who do not recognize the domain. A dominant marketplace in 2012 can provide a strong mechanism example without supplying that store’s current effect. Search layouts and advertising products have also changed. ### What should the decision rule preserve? Use a valid assignment unit, a defined branded-query set and a mature sales window. Account for cross-device shopping, geographic leakage and simultaneous promotions. Examine new and existing customers using rules set before the test. An ad platform’s attributed revenue is useful operational context, but the decision requires total outcome differences and costs on the assignment denominator. An inconclusive interval should remain inconclusive. A store can continue a bounded test or make a decision using the range of plausible economic outcomes, rather than describing a statistically uncertain traffic effect as proven zero. ## Does an interval containing zero prove an ad pause is harmless? No. The adjusted branded-click interval allows approximately a 3.92% decline as well as a 2.98% increase. A merchant who needs to rule out a loss larger than 1% cannot treat that interval as evidence of equivalence within 1%. Failing to reject zero and demonstrating a suitably small effect are different decisions. The public result remains a historically informative estimate, not proof that every commercially relevant loss is absent. (Blake, Nosko and Tadelis, Table 1 (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf), May 2014). The unadjusted MSN before-and-after coefficient implies a 5.45% decline, with an approximate normal interval of 7.03% to 3.84% decline after the log transformation. Adjustment moves the point estimate by about 4.92 percentage points toward zero and substantially changes its uncertainty. That is a comparison of source specifications, not a separate estimate of the amount of traffic caused by seasonality. ### How should the merchant set the economic loss allowance? A worked decision starts with the cost saved, not the attributed revenue credited to the ads. Suppose pausing a defined brand campaign saves $10,000 during the decision window and the comparable counterfactual contribution before that campaign cost is $500,000. A contribution loss of $10,000, or 2%, would consume the entire saving, before other changes in cost. If a valid holdout estimates the contribution difference directly, its range can be compared with that threshold. The 2% is an explicit worked economic assumption. It cannot be compared mechanically with eBay's 3.92% lower traffic bound, because clicks and contribution are different outcomes and the two businesses have different populations. The point is to choose a decision threshold in the same units as the local experiment, then ask whether the evidence resolves it. ### What result would distinguish displacement from lost demand? Inspect paid-brand, organic-brand and combined eligible traffic, then total contribution on the assignment denominator. If paid traffic disappears while organic traffic replaces it and contribution remains within the acceptable loss allowance, the operational meaning differs from a pause that also loses total demand. Existing customers may be easier to displace to organic links than shoppers unfamiliar with the store, which is consistent with the paper's separate non-brand heterogeneity discussion. A local test should preserve the brand-query definition and current organic visibility. An important negative control is a query where the merchant's own site is difficult to find without an ad. Another is a market with active competitors on the brand term. A dominant marketplace's substitution mechanism is less transferable in those cases. Record the observed search conditions during the test rather than assuming the 2012 environment persists. The specification intervals and economic threshold (https://takeratehq.com/guides/branded-paid-search-incrementality-statistics-from-the-ebay-experiment/data/brand-search-decision.csv) and calculation (https://takeratehq.com/guides/branded-paid-search-incrementality-statistics-from-the-ebay-experiment/analysis.py) keep causal design, uncertainty and cost separate. A result can support a bounded local decision without establishing universal paid-search waste or a proof of zero effect. ## Methodology and Sources We read NBER Working Paper 20171, issued May 2014, and its Table 1 and experiment design sections. Data concern 2012 tests, rather than the publication year. We transform log coefficients with 100×(exp(beta)−1), and form approximate normal 95% bounds before exponentiating. We do not reestimate raw data or treat standard errors as new evidence. The branded MSN design is a cross-platform difference in differences comparison; the non-brand geographic design is described separately and includes historical matching. The figure contains the adjusted brand coefficient only. Search markets and eBay’s scale limit transfer to an individual DTC merchant. ## Glossary Incrementality: The additional outcome caused by an intervention. Attribution: A rule giving an interaction credit for an outcome. Difference in differences: Comparing outcome changes between an intervention group and a comparison group. Log coefficient: An estimate on a logarithmic outcome scale that requires a transformation for a proportional change. ## Sources Blake, Nosko and Tadelis: Consumer Heterogeneity and Paid Search Effectiveness (https://www.nber.org/system/files/working_papers/w20171/w20171.pdf)