Retail Media Incrementality in Python: What a Synthetic Holdout Can (and Cannot) Tell Us

I recently added a synthetic incrementality example to my public Retail Media Analytics Pipeline. It gave me an opportunity to explore a distinction that is easy to lose in advertising dashboards:

Attributed conversions are not the same as incremental conversions.

A report can tell us how many conversions were associated with an ad campaign. The harder question is: how many of those outcomes happened because of the campaign, rather than happening anyway?

This tutorial walks through a fabricated treatment/control example, shows how to calculate basic lift in Python, and explains the assumptions that would be required for a real causal experiment.

Transparency: All values below are synthetic. The code assumes a randomized experiment; it does not prove random assignment happened, and it does not establish any real-world advertising impact.

Why ROAS and incrementality answer different questions

In a conventional campaign report, return on ad spend (ROAS) is often calculated as:

ROAS = attributed revenue / advertising spend

That is a useful descriptive metric, but the numerator depends on an attribution methodology. Attribution isn't automatically causal identification.

An incrementality study instead compares outcomes for people eligible to receive an intervention with outcomes for a comparable group that did not receive it.

The simplest conceptual design has two arms:

• Treatment: eligible users assigned to the advertising intervention.
• Control: comparable users assigned to a holdout.

For this comparison to support a causal interpretation, assignment and exposure must be handled correctly. Aggregated campaign reports alone cannot establish that.

A small, intentionally synthetic dataset

The example added to the repository stores aggregate cohort counts in data/raw/simulated_holdout.csv:

group,users,conversions,revenue,spend
treatment,50000,3000,240000.00,45000.00
control,50000,2600,208000.00,0.00

Both groups have 50,000 users. Treatment has 3,000 conversions; control has 2,600.

These numbers are deliberately simple, so the calculations are easy to inspect. They are not from an employer, an ad platform, or an actual experiment.

Step 1: Compare conversion rates

Conversion rate is the number of conversions divided by users in that arm.

treatment_rate = 3000 / 50000
control_rate = 2600 / 50000

absolute_lift = treatment_rate - control_rate
relative_lift = absolute_lift / control_rate

print(treatment_rate)  # 0.060 = 6.0%
print(control_rate)    # 0.052 = 5.2%
print(absolute_lift)   # 0.008 = 0.8 percentage points
print(relative_lift)   # approx. 15.38%

The distinction between percentage points and percent matters:

• Treatment conversion rate: 6.0%
• Control conversion rate: 5.2%
• Absolute difference: 0.8 percentage points
• Relative lift: approximately 15.4%

A chart titled “15.4% incremental lift” without specifying the denominator can be misleading.

Step 2: Estimate incremental conversions at treatment scale

In this demonstration, incremental conversions are the conversion-rate difference multiplied by the number of treated users:

incremental_conversions = absolute_lift * 50000
print(incremental_conversions)  # 400.0

The estimate is 400 incremental conversions, conditional on the causal assumptions of the hypothetical experiment.

Scaling to treatment size is important when treatment and control groups are unequal. Simply subtracting raw conversion counts across differently sized cohorts would conflate group size with performance.

Step 3: Estimate incremental revenue and iROAS

The code works with revenue per user rather than just subtracting totals. For equally sized groups, the result is numerically the same here:

treatment_rpu = 240000 / 50000
control_rpu = 208000 / 50000

incremental_revenue = (treatment_rpu - control_rpu) * 50000
incremental_spend = 45000 - 0

iroas = incremental_revenue / incremental_spend

print(incremental_revenue)  # 32000.0
print(iroas)                # approx. 0.71

That produces a $32,000 estimated incremental revenue difference and iROAS of approximately 0.71 in the toy dataset.

Note that attributed ROAS and iROAS can differ substantially. They measure different things.

Also, this example does not include uncertainty estimates for revenue or iROAS. A point estimate should never be presented as definitive proof of impact.

Step 4: Add an approximate uncertainty interval

The repository calculates a large-sample, unpooled Wald approximation for the difference in proportions:

import math

pt, pc = 0.06, 0.052
nt, nc = 50000, 50000

difference = pt - pc
se = math.sqrt(
    pt * (1 - pt) / nt
    + pc * (1 - pc) / nc
)
ci = (difference - 1.96 * se, difference + 1.96 * se)

For these hypothetical counts, the approximate 95% interval for the conversion-rate difference is about 0.52 to 1.08 percentage points.

That interval covers the conversion-rate difference, not incremental revenue or iROAS. The Wald method can perform poorly with small samples or rare events; production analysis may require a better interval and a design-aware variance estimator.

A numerically narrow confidence interval does not resolve poor randomization, interference, invalid conversion definitions, or mismatched observation windows.

Step 5: Make the analysis reproducible

The repository includes a standalone module, src/incrementality.py, which:

• Validates that exactly one treatment and one control row are provided.
• Rejects negative or nonfinite monetary amounts.
• Rejects conversion counts that exceed user counts.
• Calculates conversion-rate lift, treatment-scaled incremental conversions and revenue, and incremental ROAS.
• Writes results to JSON for inspection.

Run it from the repository root:

python -m src.incrementality \
  --input data/raw/simulated_holdout.csv \
  --output data/outputs/incrementality_summary.json

It also includes seven unit tests, covering calculations and invalid inputs:

python -m unittest discover -s tests -p 'test_incrementality.py' -v

The GitHub Actions CI runs for this addition passed on October 17, 2026. This is a small educational module, not a production experiment framework.

What I'd add before calling this production-ready

A real incrementality platform needs far more than a CSV and a calculation function.

Randomization evidence. Record assignment at the relevant unit (user, household, store, region, or another cluster), and detect sample-ratio mismatches.

Measurement consistency. Align conversion definitions, time windows, exclusions, identity resolution, and delayed outcomes.

Interference and spillover. Exposure in the treatment group must not silently alter outcomes in the control group, or vice versa.

Uncertainty for money metrics. Revenue often has a skewed distribution. A credible design should estimate uncertainty for incremental revenue and iROAS rather than reporting only point estimates.

Operational checks. Pre-register primary outcomes, inspect data completeness, document assignment and analysis versions, and enforce appropriate privacy controls.

An especially important caveat: this project's randomized treatment/control design is an assumption of the synthetic exercise. The CSV has no assignment logs, so the code cannot independently validate that assumption.

What I learned from building the example

Three principles stand out:

1. Separate descriptive and causal metrics. Cross-channel campaign dashboards tell us useful things, but they do not automatically measure causal lift.
2. Write assumptions next to the numbers. “400 incremental conversions” is incomplete without treatment size, experiment validity, and uncertainty.
3. Make toy models auditable. Simple synthetic examples, unit tests, and readable JSON outputs can clarify the mathematics before a more sophisticated system is attempted.

You can explore the complete incrementality walkthrough, Python implementation, and tests.

Question for the community: What tends to cause more trouble in real incrementality programs—assignment integrity, conversion measurement, or communicating uncertainty to stakeholders?

— Rupak Kulkarni | Data Engineering, PySpark, Retail Media Analytics