Incrementality Testing

The Problem with Attribution-Based Measurement

As Patrick Gilbert argues in Never Always, Never Never, the modern marketing obsession with accountability has led to measurement systems that confuse correlation with causation. Traditional attribution models track customer touchpoints and assign credit based on proximity to conversion, but they cannot answer the fundamental question: would this sale have happened without our marketing? This distinction matters enormously. A customer who clicks a branded search ad after seeing a TV commercial might have searched for the brand anyway. Attribution gives the search ad credit, but incrementality testing reveals whether the ad actually influenced the purchase decision. The difference between these measurements can be dramatic. Many branded search campaigns show excellent attributed ROAS but minimal incremental lift when properly tested. The bias toward easily measurable metrics has created what Gilbert calls "the illusion of control." Marketers optimize for attribution-friendly channels like paid search while underinvesting in harder-to-measure but potentially more effective brand-building activities. This systematic measurement bias undermines long-term growth while appearing to improve short-term efficiency.

How Incrementality Testing Works

Incrementality testing applies experimental design principles to marketing measurement. The basic methodology involves randomly dividing a target audience into test and control groups, exposing only the test group to marketing activity, then comparing outcomes between groups after a predetermined period. For digital channels, this often means using geo-holdout tests or user-level randomization. A geo-holdout might exclude certain ZIP codes or designated market areas from advertising, then compare sales performance in those areas to matched markets receiving normal advertising pressure. User-level tests randomly suppress ads for a percentage of the target audience while maintaining normal serving for the remainder. The key is ensuring groups are truly comparable except for marketing exposure. Proper randomization eliminates selection bias and confounding variables that could skew results. Statistical significance testing determines whether observed differences between groups represent genuine lift or random variation.

Types of Incrementality Tests

Channel-level tests measure the incremental impact of entire marketing channels. A brand might pause all Facebook advertising in randomly selected markets to understand Facebook's true contribution to sales. These tests often reveal that high-attribution channels generate less incremental value than expected, while channels with poor attribution tracking drive more lift than credited. Campaign-level tests evaluate specific creative executions, audience segments, or bidding strategies within a channel. Rather than measuring total channel impact, these tests optimize tactical decisions by isolating the incremental effect of individual campaign elements. Budget incrementality tests determine optimal spending levels by testing different investment amounts across matched audience segments. These reveal diminishing returns curves and help identify the point where additional spend generates insufficient incremental value. Cross-channel tests examine how different marketing activities interact and influence each other's effectiveness. These complex experiments can reveal synergies between channels that single-channel measurement misses entirely.

The more we chase accountability, the less effective we become.

Patrick Gilbert, Never Always, Never Never

Incrementality vs Attribution: Key Differences

Attribution models answer "which touchpoints preceded conversion?" while incrementality testing answers "which activities caused additional conversions?" This fundamental difference leads to dramatically different strategic conclusions. Attribution typically overvalues bottom-funnel activities like branded search and retargeting because these intercept high-intent customers close to conversion. However, incrementality testing often shows these activities capture existing demand rather than create new demand. A customer retargeted after visiting a website might have returned and purchased anyway. Conversely, attribution undervalues upper-funnel brand-building activities that influence purchase intent over longer time horizons. A display campaign might generate minimal attributed conversions but substantial incremental lift by making the brand more mentally available when customers enter purchase mode weeks later. Incremental lift testing also accounts for cannibalization effects that attribution misses. When a new campaign launches, it might generate attributed conversions while reducing performance in other channels. Attribution sees only the positive impact, while incrementality testing measures net impact across the entire marketing ecosystem.

Implementation Challenges and Solutions

The primary obstacle to incrementality testing is organizational resistance to measurement ambiguity. As Gilbert observes in Chapter 20, marketing teams have become addicted to the precision of attribution models, even when that precision is misleading. Incrementality tests require patience, statistical rigor, and comfort with confidence intervals rather than exact numbers. Technical implementation presents additional challenges. Geo-holdout tests require sufficient market coverage and geographic isolation to avoid spillover effects. User-level tests need robust randomization systems and careful audience matching. Both approaches require statistical expertise to design experiments with adequate power and proper controls. Budget allocation during testing periods creates political tensions. Marketing managers resist pulling spend from high-attribution channels, even temporarily, because short-term attributed performance will decline. This requires leadership support and clear communication about testing objectives and timelines. Data integration complexity compounds these challenges. Incrementality tests must combine advertising exposure data, sales results, and external factors like seasonality or competitive activity. Many organizations lack the technical infrastructure and analytical capabilities for comprehensive test design and result interpretation.

Strategic Applications and Business Impact

Incrementality testing fundamentally reshapes marketing strategy by revealing true performance rather than correlated outcomes. Brands consistently discover that their highest-attribution channels generate less incremental value than expected, while undervalued channels like connected TV, audio, or upper-funnel display drive substantial unmeasured lift. This insight rebalances media mix toward effectiveness over efficiency. Rather than maximizing attributed ROAS, marketers optimize for incremental return on ad spend, leading to increased investment in brand-building activities that compound over time. The strategic shift mirrors Les Binet and Peter Field's research showing that effectiveness-focused campaigns outperform accountability-focused ones. Budget optimization becomes more sophisticated when based on incremental curves rather than attribution models. Incrementality testing reveals diminishing returns within channels and optimal allocation across channels based on marginal incremental value rather than average attributed performance. Perhaps most importantly, incrementality testing builds organizational confidence in brand marketing investments. When leadership understands that a display campaign generated 300 incremental conversions despite attributing only 50, they become more willing to fund long-term brand building over short-term performance marketing.

Related Terms