Regression discontinuity designs break in six common ways

Regression discontinuity (RD) is one of the most credible designs for causal inference from observational data. When a rule assigns treatment based on whether a score crosses a cutoff (a scholarship for test scores above 80, a programme for towns under a population threshold, a policy on one side of a border), people just above and just below the cutoff should be comparable. Any jump in outcomes at the cutoff can then be attributed to the treatment.

But a jump in a plot is not the same thing as a credible effect. This post goes through the mistakes that most often undermine RD analyses and the tools that help avoid them.

A quick refresher

The variable that determines treatment is the running variable (also called the forcing variable or score). RD comes in two forms:

• Sharp RD: treatment switches on completely at the cutoff.
• Fuzzy RD: crossing the cutoff only changes the probability of treatment. The effect is estimated as the jump in the outcome divided by the jump in treatment take-up, which is essentially an instrumental-variables estimate at the cutoff. It needs a non-zero first stage, an exclusion restriction, and monotonicity, and it identifies a local effect for compliers near the cutoff.

The key identifying assumption is not that the observed outcome is smooth. It is that the potential outcomes, meaning what would happen with and without treatment, change smoothly through the cutoff. Lee and Lemieux (2010) give an accessible treatment, and the design itself goes back to Thistlethwaite and Campbell (1960).

Mistake 1: fitting high-order global polynomials

A common older practice was to fit a 4th-, 5th-, or 6th-degree polynomial to the whole range of the running variable on each side and read off the gap at the cutoff. Gelman and Imbens (2019), in the Journal of Business & Economic Statistics, argue strongly against this, for three reasons:

• Global polynomials put surprisingly large and erratic weight on observations far from the cutoff.
• Estimates are sensitive to the polynomial order.
• Confidence intervals tend to be too narrow, so real coverage falls short of the nominal level.

Better: use local linear (or local quadratic) regression within a bandwidth around the cutoff, with a triangular kernel that gives more weight to points near the threshold.

Mistake 2: choosing the bandwidth by eye (or by result)

How much data on either side of the cutoff should you use? A narrow window reduces bias but increases variance. A wide window does the opposite. Trying several bandwidths and reporting the one with the cleanest result is a garden of forking paths.

Better: use a data-driven, MSE-optimal bandwidth selector and report it up front. Then show sensitivity to, say, half and double that bandwidth as a robustness check, not as a menu to choose from.

Mistake 3: conventional confidence intervals with an optimal bandwidth

Here is a subtle one. MSE-optimal bandwidths are, by construction, wide enough that some bias remains. Conventional confidence intervals ignore that bias, so they under-cover. Calonico, Cattaneo and Titiunik (2014), in Econometrica, proposed robust bias-corrected confidence intervals that account for this. They are now standard, and implemented in the rdrobust package for R, Stata, and Python:

library(rdrobust)
out <- rdrobust(y = outcome, x = score, c = 0)   # local linear, MSE-optimal bandwidth
summary(out)          # report the robust bias-corrected CI
rdplot(y = outcome, x = score, c = 0)            # binned scatter with local fits

Report the point estimate together with the robust confidence interval, not the conventional one.

Mistake 4: not checking for manipulation of the running variable

If people can precisely control their score (retaking a test until they pass, or misreporting income to stay under a threshold), those just above the cutoff are no longer comparable to those just below. A tell-tale sign is bunching: a spike in the density of the running variable on one side.

McCrary (2008) introduced the standard density test. A more recent local-polynomial version is available in the rddensity package. A histogram of the running variable with fine bins is a cheap first look that every RD paper should show.

Note the asymmetry. A clean density test does not prove the absence of sorting, but clear bunching is a serious warning.

Mistake 5: skipping covariate balance checks

If the design works, pre-treatment characteristics should not jump at the cutoff. Run the same RD estimator with baseline covariates (age, prior outcomes, demographics) as the outcome. A significant jump in a variable the treatment could not have affected suggests sorting or a second policy that shares the same cutoff.

Mistake 6: over-extrapolating the result

An RD estimate is a local effect at the cutoff. A scholarship's effect on students scoring exactly 80 says little about students scoring 50. Reporting it as "the effect of scholarships" stretches the design beyond what it identifies. Be explicit about the population the estimate applies to.

A cautionary case: the Huai River heating study

China historically provided free winter heating north of the Huai River and not south of it, which creates a geographic discontinuity. Chen, Ebenstein, Greenstone and Li (2013), in PNAS, used this border to estimate the effect of air pollution on life expectancy. The paper drew wide attention, and also methodological criticism, notably from Andrew Gelman, centring on the use of a global cubic polynomial in distance from the river and how sensitive the estimate was to that choice. A follow-up study, Ebenstein, Fan, Greenstone, He and Zhou (2017), also in PNAS, revisited the question with more data and different specifications.

Whatever your view of the substantive result, the episode is a useful teaching case. It shows why specification choices in RD (polynomial order, bandwidth, what counts as "near" the cutoff) need to be justified and stress-tested rather than picked once.

A practical RD checklist

1. Confirm the assignment rule exists independently of the outcome data. Don't go hunting for a kink first and call it a policy cutoff afterwards.
2. Plot the raw data with rdplot or a binned scatter.
3. Check the density of the running variable for bunching.
4. Check covariate balance at the cutoff.
5. Estimate with local linear regression, an MSE-optimal bandwidth, and robust bias-corrected CIs.
6. Show sensitivity to bandwidth and kernel; avoid high-order global polynomials.
7. For fuzzy RD, report the first stage.
8. Run placebo cutoffs where no effect should exist.
9. State clearly that the estimate is local.

I wrote a longer version of this discussion, with more on the Huai River debate, in this article on common regression discontinuity mistakes.

References

• Calonico, S., Cattaneo, M. D., & Titiunik, R. (2014). Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica, 82(6), 2295–2326.
• Chen, Y., Ebenstein, A., Greenstone, M., & Li, H. (2013). Evidence on the impact of sustained exposure to air pollution on life expectancy from China's Huai River policy. PNAS, 110(32), 12936–12941.
• Ebenstein, A., Fan, M., Greenstone, M., He, G., & Zhou, M. (2017). New evidence on the impact of sustained exposure to air pollution on life expectancy from China's Huai River Policy. PNAS, 114(39), 10384–10389.
• Gelman, A., & Imbens, G. (2019). Why high-order polynomials should not be used in regression discontinuity designs. Journal of Business & Economic Statistics, 37(3), 447–456.
• Lee, D. S., & Lemieux, T. (2010). Regression discontinuity designs in economics. Journal of Economic Literature, 48(2), 281–355.
• McCrary, J. (2008). Manipulation of the running variable in the regression discontinuity design: A density test. Journal of Econometrics, 142(2), 698–714.
• Thistlethwaite, D. L., & Campbell, D. T. (1960). Regression-discontinuity analysis: An alternative to the ex post facto test. Journal of Educational Psychology, 51(6), 309–317.