Throughout my 18+ years in marketing analytics, including 14 years at Google navigating the measurement stacks of Fortune 500 firms, I worked through a fundamental shift in business intelligence. We moved from an era of data abundance to limited visibility.
Today, the Measurement Gap is the defining friction of the C-suite. In a typical Tuesday morning leadership meeting, the conflict plays out predictably:
- Your performance team reports profitable Return on Ad Spend (ROAS) on social platforms.
- Your media agency presents a separate, optimistic narrative of brand lift.
- Your Marketing Mix Model (MMM) offers a third, contradictory and conservative view.
This is the battle within attribution. It persists because our favorite tool, tracking via a pixel, has been rendered obsolete by privacy shifts like Apple’s iOS 14.5 and the erosion of third-party cookies.
When the data is broken, every department retreats to its own siloed version of the truth. To lead through this, savvy marketing teams must stop searching for better tracking and start searching for better causality.
We Need to Stop Chasing the Perfect Experiment
For more than a decade, the standard remedy for this uncertainty has long been the Randomized Controlled Trial (RCT). You may have heard them referred to as Matched Market tests or geo experiments. I’ve run dozens of these for customers when I worked for Google on their measurement team.
They work like this: take two similar cities (say, Minneapolis and St. Louis), increase spend in one, keep it flat in the other, and measure the change. Ideally, we’d have 3-5 markets on each side, but they fundamentally work the same way.
While sound and simple in design, marketing teams using the standard RCT approach need to divert internal resources to run these.
- These tests typically require 8-12 weeks of planning and execution.
- They demand a freeze on all other market activity to avoid polluting the results, which requires a level of discipline few companies can afford if targets start slipping.
- They often require shifting budgets or asking for incremental budgets.
RCTs becomes a “test and learn” ask that often triggers a skepticism from a CFO who has grown weary of experiments that fail to translate to the bottom line.
Savvy marketing teams are shifting to a more pragmatic, statistically rigorous option: the Quasi Geo-Lift Experiment.
Creating the “Digital Twin”
Underneath a quasi-experimental design is an older concept called Synthetic Control Method (SCM).
- In a traditional experiment, you compare a “test” city to a physical “control” city and hope they remain similar enough throughout the window to provide a clean read. If a local competitor launches a flash sale in your control city halfway through the month, your experiment is ruined.
- In a quasi-experiment, we don’t hope for a perfect control. Instead, we construct one mathematically and based on data you already have.
Example
Imagine you want to test a incrementality of a YouTube campaign in Chicago. Instead of comparing Chicago to just, say, Philadelphia, we use your team’s historical performance data to build a “Digital Twin” of Chicago. This twin is a weighted combination of dozens of other markets. Perhaps 30% Houston, 25% Phoenix, 20% Seattle and 25% from smaller aggregate markets.
Historically, this specific blend of cities will have tracked Chicago’s sales patterns with >99% accuracy. When your campaign launches in the Chicago, any divergence between the actual performance and the Digital Twin’s projected trajectory represents the True Incremental Lift.
By moving from physical controls to synthetic ones, you eliminate the noise of the local market. With this approach, we are no longer comparing one city to another but instead comparing a city to its own shadow.
We Can Run These Experiments Retrospectively
The most significant advantage of this approach is not just accuracy. It’s the flexibility.
Traditional geo experiments are future-looking. We must design the test, wait for the flight to run, and then analyze the data.
Quasi-experiments, however, can be run on historical data. Because the synthetic control is built on existing data that your company already owns, we can analyze campaigns that are already live or even those that concluded last quarter.
I recently worked with a retail VP who was being pressured to cut a high-budget “Upper Funnel” video strategy because the attribution ROAS looked low. By using a quasi-experimental framework, we were able to look back at the previous 6 months of spend across 15 markets. We built synthetic controls for those markets and discovered that while the direct attribution was low, the incremental lift on site traffic was substantial.
In other words, we were able to find the causal truth using data the customer already had, without a 3-month lead time or a request for new budget. This allows a leader to audit current strategies in real-time and build a library of evidence before walking into a budget planning session.
Establishing the “Rosetta Stone” of Measurement
I often refer to geo-lift testing as the Rosetta Stone of Measurement. While different platforms seem to speaking different, incompatible languages, the quasi geo-lift experiment is one tool that translates data into the universal language of business: incremental profit.
By using quasi experiments, you can establish a Calibration Multiplier.
How Calibration Works
- Run the Test: A Quasi Geo-Lift test shows that for every $1.00 spent on Platform X, you see $1.50 in truly incremental revenue.
- Compare to Platform: Simultaneously, Platform X’s internal dashboard claims it generated $4.50 in revenue (driven by non-incremental touches).
- Calculate the Multiplier: You now know that Platform X’s data has a 0.33x Calibration Multiplier ($1.50 / $4.50).
Now, your performance team can keep using their real-time dashboards for daily optimization, but they must apply the 0.33x multiplier to their numbers. This aligns the entire organization around incremental dollars. It translates “engagements,” “views” and “platform conversions” into one of the few marketing metrics Finance cares about: Cost per Incremental Conversion (CPIC).
With the IAB’s Project Edios, we can expect to see platforms align on consistent definitions for views and engagements. Yet, the attribution systems within them will still attempt to steal credit.
Protecting Your Team’s Reputation When Asking for Test Budgets
Securing an incremental budget for an unproven channel or a bold brand play is often is often met with skepticism. In these cases, the Minimum Detectable Effect (MDE) becomes your most effective risk-management tool.
Think of the MDE as the sensitivity of your test. In a quasi-experiment, and based on your actual sales data, we can estimate the spend and how much time is required to move the needle in a statistically meaningful way, all before your team spends a dime.
This changes the conversation with the CFO. Instead of asking for a test based on industry benchmarks, you can propose a rigorous research investment with clear boundaries:
“To prove a 5% lift in the Pacific Northwest with 95% confidence, we need a budget of $200,000 over six weeks. This is our MDE. If the results fall below this threshold, we have the evidence to pivot these resources to other channels immediately.”
This framing demonstrates a level of discipline that builds immense credibility with the leadership team. You are proactively articulating the expected result and a path if the campaign fails to deliver.
Be careful not to over-rely on P-values
In my experience, the most successful marketing teams view P-values not as a binary Success/Failure results, but as an indicator of surprise.
A small P-value (day, p < 0.05) indicates that the observed data is highly surprising if your marketing had zero effect. It is a signal of compatibility with your hypothesis, not absolute proof. Rather than chasing a single statistically significant checkbox, look to the Confidence Interval, which is the range of potential lift.
If a test shows an incremental lift of 10%, but the confidence interval ranges from -2% to +22%, a savvy leader knows that while the “average” result is positive, the risk of zero impact remains high. This range provides a more realistic basis for long-term forecasting and strategic pivots than a single, misleading “ROAS” number.
A Roadmap for Better Marketing Results
Transitioning to a quasi-experimental culture does not happen overnight. It requires a shift in both talent and tooling. If you are a VP looking to implement this standard for the next fiscal year, follow this three-step roadmap:
1. Audit the team’s skill set
If your current team is only reporting on “last-click” or “multi-touch attribution,” you have a talent gap that should be filled.
2. Standardize the Retrospective Audit
Before launching any new major initiatives, run a back-test on last year’s largest campaign launch. Compare the platform-reported results to a synthetic control analysis of that same period. The gap you find will serve as the initial shock to the system required to gain C-suite buy-in for a new measurement standard.
3. Build the Incremenaltiy Dashboard
Move away from dashboards that show “Total Conversions.” Create a view that shows “Estimated Incremental Conversions” by applying your Calibration Multipliers. When the CEO asks how the weekend sale went, you should be able to say, “The platform says we did 1,000 units, but our causal models tell us 400 of those were truly incremental. Our CPIC was $12.”
The Defensible Leader
While the phrase, “quasi-experimental design” can sound academic and disconnected from business, it’s a critical tool for marketing teams to defend and growth their budgets. Marketing teams who thrive are those whose decisions can withstand cross-examination by both a rigorous data science teams and a skeptical Finance team.
By leveraging the data you already have to run retrospective, quasi-experimental analyses, you can begin establishing “causal truth” immediately. You no longer need to wait for the “perfect” experiment or the “perfect” cookie replacement. By the time you need to ask for that next incremental budget, you will be holding the causal evidence to prove that your marketing is a growth engine, not just a cost center.
