Paid Search Incrementality Testing: A Practical Framework

You may know exactly how much revenue Google Ads claims and still not know how much revenue the ads created. That gap matters most when branded campaigns, strong organic rankings, and direct traffic all reach the same customer.

A paid search incrementality test replaces that ambiguity with a controlled absence. You pause a defined slice of advertising, measure what actually disappears and what moves elsewhere, then compare the incremental loss with the spend you avoided. The goal isn’t to prove that paid search works or doesn’t. It is to identify where it acquires demand, where it supports another channel, and where it charges you for demand you already own.

Attribution records a route; incrementality measures an effect

Platform attribution answers, “Which tracked interaction received credit?” Incrementality answers, “What would have happened without this interaction?” Only the second question tells you whether removing or reducing spend would materially change the business outcome.

Suppose a customer searches your company name, clicks an ad above your top organic result, and buys. The advertising platform can correctly record the ad click while still overstating the ad’s causal value. The unresolved question is whether that same customer would have clicked the organic listing and bought anyway.

You can’t settle that question with last-click, first-click, data-driven, or multi-touch attribution alone. Changing the credit rule redistributes recorded value among observed touches. It doesn’t create the missing counterfactual.

The prior evidence is genuinely mixed. Google’s pause experiments across more than 400 advertisers estimated that 89% of ad clicks were incremental on average, while eBay’s branded-search experiment found that almost all missing paid clicks and sales moved to organic. Google’s result is platform-supplied evidence, and neither finding is a universal rule. The difference is the point: brand strength, organic visibility, query type, competition, and account structure can produce very different answers.

For a useful diagnosis, classify paid search at the query or campaign level:

These aren’t permanent labels. A branded query can be largely cannibalized while you rank first, then become more incremental if organic visibility falls or a competitor changes the search results. Your test should therefore support a budget rule with conditions, not a timeless verdict about the channel.

Key takeaways

• Test a material but reversible slice of spend instead of switching off the entire account by default.
• Judge the test on total business outcomes, not on the revenue that disappears from the advertising platform’s report.
• Separate branded search, non-brand search, Shopping, and Performance Max because their substitution patterns can differ.
• Join paid search-term data with organic query data before the pause so you know where paid and organic already overlap.
• Allow for delayed substitution. A short test can make paid search look more incremental than it is if customers and reporting take time to move.
• Make the final decision with incremental contribution or profit, not attributed ROAS.

Design the pause around one budget decision

A broad question such as “Does paid search work?” cannot produce a clean action. Define the decision first: whether to keep branded ads in a particular market, reduce spend on terms where you already rank strongly, or retain a non-brand campaign that appears to introduce new customers.

Then write the test plan before changing the campaigns:

1. State the counterfactual. Write what you expect customers to do when the selected ads disappear. For example, they may move to organic listings, arrive directly, choose a competitor, or not visit at all. This forces you to measure the channels where substitution should appear.
2. Choose one testable slice. Isolate branded search from non-brand search, Shopping, and Performance Max. A result from brand terms should not be used to cut prospecting campaigns whose job and audience are different.
3. Select the test unit. A campaign, coherent query group, or market can be paused while a comparable unit remains active. A credible control helps distinguish the pause from seasonality, promotions, or a general change in demand. If no good control exists, be explicit that a pre-versus-post result carries more uncertainty.
4. Lock the primary outcome. Use total revenue, qualified leads, purchases, or another business result that exists outside the ad platform. Record paid-attributed revenue, organic revenue, direct revenue, organic clicks, and total query clicks as diagnostic measures rather than competing versions of success.
5. Define the economic rule. Decide in advance how you will compare the incremental outcome with avoided media cost. Where margin data is available, use contribution rather than revenue; otherwise a high-revenue, low-margin campaign can appear more valuable than it is.
6. Record known disruptions. Promotions, price changes, inventory constraints, site outages, tracking changes, SEO releases, and brand publicity can alter the same metrics as the pause. Log them during the test and exclude or qualify affected periods instead of explaining them away after seeing the result.
7. Set exposure and rollback conditions. Specify the largest acceptable business loss before launch. If the downside could be material, stage the pause or use a narrower market. Don’t invent the rollback threshold after an uncomfortable result appears.
8. Declare the observation window. Include enough time for buying cycles, channel switching, and revenue reporting to settle. One documented pause recovered 30% of paid-attributed revenue through organic and direct within six weeks, but that figure rose to 65% by week 13. That is evidence that substitution can lag, not a universal thirteen-week minimum.

Build the overlap baseline before you pause

Export Google Ads search terms with their spend and outcomes, then export matching Google Search Console queries and organic clicks for the same dates. Normalize obvious differences such as capitalization and whitespace, but preserve query intent. A brand name, a brand-plus-product query, and a generic category query shouldn’t be collapsed into one row merely because all three contain the company name.

For each matched query, record paid clicks, paid spend, paid outcomes, organic clicks, and whether a meaningful organic result is present. This gives you a map of expensive overlap. It does not prove cannibalization on its own: customers can still respond differently when both listings appear. The pause provides the causal evidence; the query join tells you where to look and how to interpret the movement.

Protect the business without protecting the assumption

A total-account blackout can create unnecessary financial exposure. Choose the largest coherent slice whose potential loss the business can tolerate, while retaining enough volume to produce a useful signal. If a small unit cannot distinguish normal variation from a real effect, acknowledge that limitation or have an analyst assess the design before increasing exposure.

Monitor competitor activity on branded results during the pause, but don’t treat a competitor impression as proof that your ad is incremental. The relevant outcome is whether the changed results cause a measurable loss in total clicks, conversions, revenue, or contribution. Brand protection can be a legitimate job for paid search; it should be named and valued as protection rather than reported as customer acquisition.

Measure substitution outside the advertising dashboard

The moment you pause ads, paid clicks and paid-attributed revenue will fall. That is an implementation check, not the test result. The result is the difference between the total outcome you observed and the total outcome you would reasonably have expected with the ads still running.

Use a comparable control market or campaign when you have one. Measure how the control changed over the same period, then apply that movement to the test unit’s baseline. This is more defensible than assuming the week before the pause would otherwise have repeated exactly. Without a control, compare against a predeclared baseline and carry the added uncertainty into the decision.

Calculate the readout in this order:

1. Estimate the paid-on counterfactual. Determine the total revenue, purchases, or qualified leads you would have expected in the test unit if ads had remained active.
2. Measure the total incremental loss. Subtract the observed total outcome during the pause from the paid-on counterfactual. This is the business effect attributable to removing the ads, subject to the design’s uncertainty.
3. Measure channel substitution. Compare organic, direct, and any other plausible substitute channels with their counterfactual levels. Use these movements to explain where demand went, not to override the total-outcome calculation.
4. Calculate recapture. Divide verified substitute-channel lift by the paid-attributed revenue that disappeared. State clearly which channels were counted and how their counterfactuals were estimated.
5. Compare incremental value with avoided cost. For a revenue-based view, divide the incremental revenue preserved by the ad spend required to preserve it. For the economic decision, apply the relevant contribution margin and subtract media cost.

Direct traffic deserves special care. A rise in direct revenue may represent people who saw no ad and typed the address, customers returning through bookmarks, or a change in how analytics classified the visit. The first two can be genuine substitution; the third is measurement reclassification. Look for timing, market specificity, and corresponding stability in total business outcomes before counting the entire increase as recaptured demand.

The same caution applies to organic traffic. More organic clicks after a pause are persuasive when they occur on the affected queries, in the affected market, during the declared window, and alongside the expected loss of paid clicks. A sitewide organic increase caused by an unrelated SEO release shouldn’t be credited to paid-search substitution.

What a delayed recapture looks like in practice

One company paused branded search in the United States, United Kingdom, Australia, and Canada, then paused most non-brand paid search by the end of the month. Its prior spend across branded search, non-brand search, Shopping, and Performance Max averaged $113,000 per month. In one branded campaign, organic already held 71% of overlapping clicks while ads were active, and only $3,945 of $36,129 in spend appeared to purchase clicks that organic could not capture. The remaining $32,184, or 89.1%, functioned as brand defense in that analysis.

The important pattern is the delay, not a benchmark you should copy. A six-week read would have made the ads appear much more incremental than the later observation did. The shift toward direct revenue also shows why a paid-versus-organic traffic comparison is too narrow: substitution can cross both channel and attribution boundaries.

Don’t treat the remaining 35% as automatically incremental. Some of it may be a real paid-search effect, but the strength of that conclusion depends on the counterfactual, controls, tracking, and outside events. Report the observed total loss, the estimated substitute lift, the avoided spend, and the uncertainty separately. A single blended percentage hides the assumptions leadership needs to judge.

Turn the result into campaign-level budget rules

An incrementality test should end with a rule someone can execute in the account. “Paid search is incremental” and “brand ads are wasteful” are both too broad.

• High incremental contribution: retain the tested campaign when the contribution it protects exceeds media cost. Treat expansion as a new hypothesis; the next dollar may not perform like the current dollar.
• Low incrementality with strong organic substitution: keep the slice paused or reduce it, then monitor organic visibility, total query clicks, revenue, and competitor pressure. Define the conditions that would trigger a retest or restart.
• Dependent organic performance: manage paid and organic as a combined search system. Investigate which queries lost total clicks or outcomes rather than assuming that an organic ranking alone guarantees replacement.
• Primarily defensive value: label the budget as brand protection. Decide whether the measured conversion or revenue loss justifies that protection instead of letting attributed ROAS disguise it as acquisition.
• Uncertain result: don’t force a binary decision. Restore only what is required by the predeclared guardrail, improve the control or measurement, and run a better-bounded test.

Keep a permanent test record containing the hypothesis, test and control units, campaign changes, baseline dates, primary outcome, rollback rule, exclusions, calculation method, and final decision. Revisit the rule when organic visibility changes, competitors become more aggressive, margins shift, tracking changes, or the campaign begins serving a materially different mix of queries.

Your next step is to choose one material but reversible slice of paid search. Write its counterfactual, export the paid-organic overlap, lock the business guardrail, and schedule the readout far enough beyond the pause to observe substitution. If the spend returns, it should return with a clear job description: acquisition, channel support, or brand defense. If it doesn’t, you can redirect the budget toward demand you weren’t already positioned to capture.

References

• Search Engine Land — The $113,000 paid search test: How much traffic did organic replace?