Signal Depth’s Methodology

Transparent Incrementality via Open-Source Rigor

Most black box attribution tools ask for your trust while hiding their math. We believe that if a measurement methodology isn’t reproducible by your own data science team, it’s an opinion, not science.

Our framework is built exclusively on the Synthetic Control Method (SCM) and Generalized Synthetic Control (GSC), which are gold-standard frameworks used by Meta (GeoLift) and Google (Meridian). We provide the infrastructure to run these sophisticated models at scale, with the data integrity required to make high-stakes budget decisions.

1. The Pre-Test: Power Analysis & Counterfactual Design

Before a single dollar is deployed, we answer the most critical question in experimentation: Is this test even capable of detecting a win? Many geo-tests fail because they lack the statistical power to distinguish “signal” from “noise.” We utilize a rigorous Power Analysis to determine the Minimum Detectable Effect (MDE).

• Synthetic Control Construction: Rather than picking “Control Cities” (which rarely mirror “Test Cities”), we use a weighted combination of multiple non-treated regions. This creates a “Synthetic Twin” that perfectly mimics the test market’s pre-treatment behavior.
• The MDE Threshold: If your planned spend won’t generate a lift greater than the historical volatility of your data, we advise against running the test. This saves you from “False Positives” that lead to wasted spend and effort.

2. Enforcing SUTVA

A common failure in geographic testing is the violation of the Stable Unit Treatment Value Assumption (SUTVA). If you run ads in Jersey City, but those consumers work and shop in New York City, your Control group is “polluted,” and your Lift is understated.

Our Isolation Strategy

• Iterative Market Selection: We don’t just look at sales; we analyze geographic correlation and consumer mobility patterns.
• Buffer Zones: We programmatically exclude “spillover” markets from the Control pool to ensure that the treatment effect is isolated to the Test group.
• Correlated Error Checking: We test for “Parallel Trends” during the pre-test phase. If the Test and Control markets don’t move in lockstep historically, the model is rejected before it starts.

3. The “Janitor” Layer: Automated Data Integrity

We estimate data analysts spend 80% of their time cleaning data. We’ve automated much of the data checks work to ensure the model receives high-quality and consistent inputs.

4. Why Open-Source?

The industry has finally moved beyond black box modeling. By using open source libraries like GeoLift and Meridian as our engine, we offer three advantages that proprietary vendors cannot:

1. Peer-Reviewed Reliability: These libraries are maintained by hundreds of the world’s top economists and statisticians. The math has been poked, prodded, and validated by the global DS community.
2. Reproducibility: A Data Scientist can take our raw data and the open source libraries and reach the exact same conclusion.

Comparison vs. the Measurement Landscape

We don’t ask you to trust our math. We give you the infrastructure to run the world’s most advanced incrementality frameworks with the data validation required to make the results land in leadership meetings and the boardroom.

Technical FAQ

Q: How does the model account for baseline seasonality or holiday spikes?

The Synthetic Control Method (SCM) handles seasonality by modeling the relationship between regions rather than the raw volume. Since macroeconomic trends (like Black Friday or a national holiday) typically affect all regions, the “Synthetic Twin” shifts in tandem with the Test market.

• Technical Detail: We incorporate Time-Fixed Effects to soak up common shocks and can apply Seasonal Decomposition (STL) to the time series if the business has extreme non-linear volatility.

Q: What happens if a competitor runs a localized promotion in our Control group?

This is a classic Exogenous Shock. Our pipeline monitors for statistically significant deviations in individual control regions during the treatment period.

• The Guardrail: If a specific control city diverges from the synthetic pool (detected via a spike in RMSPE), our algorithm can automatically re-weight the pool or “Winsorize” the outlier. Because we use a weighted combination of cities rather than a single “Control City,” the impact of a single-market competitor sale is mathematically diluted.

Q: What is the “Minimum Data Requirement” for a valid test?

To ensure the model isn’t just “overfitting” on recent noise, we adhere to the following heuristics:

• Pre-Treatment Duration: Ideally 2× the length of the planned test (e.g., 6 weeks of history for a 3-week test).
• Granularity: Daily or weekly data at the DMA, Zip-code, or State level.
• Signal-to-Noise Ratio: We require a Minimum Detectable Effect (MDE) that is lower than your expected lift. If your organic variance is 15% and your ad spend only drives a 5% lift, we will flag the test as “Underpowered” and advise against running it.

Q: How do you validate that the “Synthetic Twin” is actually a good match?

We use In-Time Placebo Tests (Backtesting). Before the campaign launches, we run a “fake” test on your historical data where no ads were actually served.

• The Validation: If our model correctly predicts “Zero Lift” during that historical period with a high Goodness-of-Fit (R2), we gain confidence that the synthetic twin is a valid counterfactual. We provide the RMSPE (Root Mean Square Prediction Error) for the pre-treatment period as a standard KPI in every report.

Q: How do we handle noise if our DMAs overlap?

This is the SUTVA challenge. We use a Market Selection Algorithm that prioritizes geographic isolation.

• The Approach: If you are testing in Philadelphia, we programmatically exclude adjacent markets (like parts of New Jersey or Delaware) from the Control pool. This ensures that the “Treatment” doesn’t accidentally leak into the “Control,” which would otherwise lead to an understated (conservative) Lift calculation.

Q. Can this model handle multiple overlapping channels (e.g., TV and YouTube)?

Yes, but with an important caveat. If you scale both TV and YouTube in the same Test markets at the same time, the model will measure the Total Incremental Lift of the media mix.

• The Solution: To isolate a single channel, we recommend a Cell-Based Design where Media Channel A is tested in one region and Media Channel B in another, using a shared Synthetic Control pool to maintain a consistent baseline.