A/B Test Effects Can Change over Time
A typical illustration of website A/B testing: two groups of visitors see pages that differ only in the style of the “Learn more” button. The click rates of 52% and 72% in the image are illustrative figures, not actual experimental results. Author: Maxime Lorant. Source: Wikimedia Commons. License: CC BY-SA 4.0 (format converted).
On October 23, 2026, Andrew Gelman relayed an observation by experimentation tool developer Andrea Bronzini on his blog: gains often shrink after the winning version in an A/B test is rolled out. Bronzini argues that many users have already seen the old version before assignment to groups, so experiments largely measure reactions to “change.” Gelman agrees that this is a problem and places it within a more general situation: treatment effects can change over time and depend on the entire treatment history.
Gelman’s Assessment and Suggestions
The blog post is very short, consisting mainly of a reader’s letter and Gelman’s reply. In the letter, Bronzini says that while developing experimentation tools, he has repeatedly seen a failure mode: a version that “wins” an A/B test loses its gains after being rolled out to all users. His explanation is that common experimental setups mix “novelty responses” with “steady-state effects,” because assignment takes place after users first encounter the content being tested, while the outcome metric itself also changes over time.
Gelman’s reply makes three points. First, he believes Bronzini has identified a real problem. Second, this problem is a special case of a more general phenomenon: treatment effects can change over time and can depend on the entire treatment history users have experienced, something usual between-group comparisons overlook. Third, he conjectures (using “I guess” in the original) that the better approach ultimately is to incorporate these sources of variation into a model. The post provides no specific model, data, or case study; it only links to his talk, It’s About Time, as further reading and notes that the talk has not yet been written up as a paper.
The abstract for that talk was posted on the same blog on January 3, 2024. It advocates treating time as a factor in statistical analysis, citing settings including imbalance in causal inference, generalizing A/B test findings even when groups are balanced, sequential analysis, adjustment for pretreatment measurements, and poll aggregation. In other words, Gelman sees “gains disappearing after rollout” as just one manifestation of overlooking the time dimension.
First-Exposure Contamination and First-Exposure Randomization
The article Bronzini refers to is The First-Exposure Contamination Problem, first published on December 15, 2025, and updated on December 17. The author describes it as a “methodological essay.” It appears on his company’s website, and the article provides no indication of peer review. It distinguishes between two quantities: the steady-state treatment effect (SSTE), meaning the long-term difference after users have adapted to the new version, which is what decision-makers actually want to know; and the novelty response effect (NRE), meaning the response of users already familiar with the old version when they encounter a change. The author emphasizes that the direction and magnitude of the NRE vary by context: it may accelerate conversions, or it may delay or suppress them.
The problem lies in the sequence. Most experiments assign users to groups only when they reach a particular page or trigger a particular component, but many users have already seen the control version multiple times before then. By analogy, when a restaurant changes its menu, regular customers may initially feel curious, or they may struggle to find their usual dishes. This differs from the reaction of new customers who see the new menu from the outset (analogy added for this article). If an experiment assigns regular customers to groups only after they enter, it is actually comparing “B encountered as a change” with “A that is already familiar,” rather than first experiences of the two versions.
The author also explains, in terms of “time to conversion,” why early results are more likely to be dominated by novelty responses: returning users are already partway along the decision journey and closer to conversion, so they account for a disproportionately large share of early experimental conversions, regardless of their share of traffic. The article gives an illustrative calculation: an average sales cycle of 8 weeks, a standard deviation of about 2 weeks, a conversion rate of 2%, and 10,000 visitors per week. An experiment running for 4 weeks would have about 40,000 visitors and 800 conversions in total. Assuming an approximately normal distribution, conversions occurring within 4 weeks of the first visit account for less than about 3% of conversions; the vast majority come from users who had visited before the experiment began. A duration of 4 weeks is exactly 2 standard deviations below the mean, corresponding to a proportion of about 2.3%.
Area proportions of a normal distribution divided by standard deviation. Applying the white paper’s illustrative parameters (an average of 8 weeks from first visit to conversion, with a standard deviation of 2 weeks), −2σ on the horizontal axis corresponds to 4 weeks. The two sections to its left total about 2.2% (the figures in the image are rounded; the precise value is about 2.3%), representing the share of conversions occurring within 4 weeks of the first visit. The parameters are hypothetical, not measured data. Author: M. W. Toews. Source: Wikimedia Commons. License: CC BY 2.5 (converted to a bitmap).
On this basis, the white paper argues that disappearing gains after rollout are often misdiagnosed as “regression to the mean”: estimates during an experiment may be both precise and stable, and what changes is not sampling error but the quantity being estimated itself. The author proposes a “first-exposure randomization principle”: include only users whose first visit to the site occurs after the experiment begins. Implementation requires only a first-visit timestamp, usually stored in a cookie. For businesses with long sales cycles, combine this with cohort enrollment: define an enrollment window, continue serving the assigned versions to this cohort for longer than the typical sales cycle, and exclude users arriving after the window closes. The author also acknowledges that cookie deletion and visits across multiple devices can cause returning users to be misclassified as new users, but considers this error broadly symmetric between the two groups.
The Conflict with Conventional Power Calculations
Gelman’s post does not discuss power calculations; the following is this article’s synthesis of the two sources. Conventional power calculations begin by specifying a baseline conversion rate, a minimum detectable effect, a significance level, and a target power. They then calculate the required sample size and convert it into a number of days using daily traffic. This procedure implicitly assumes that the effect being estimated is a fixed number, that error comes only from sampling noise, and that larger samples produce more precise estimates.
If the effect changes with time and exposure history, that assumption no longer holds. The white paper explicitly states that under first-exposure contamination, increasing sample size cannot recover the correct effect, and that gains disappearing after rollout are not caused by insufficient statistical power. Sample size can reduce variance, but it cannot correct a mismatch in the quantity being estimated. It is like using a highly precise scale to weigh a box containing the wrong goods: however precise the scale, its reading is not the weight you wanted.
Returning to the 8-week example makes the conflict clearer: a power calculation might conclude that 4 weeks and 800 conversions are sufficient, yet only a few dozen of those conversions can reflect the steady-state effect. Following the white paper’s recommendation to use only a “clean” cohort would also require extending the experiment beyond the sales cycle, while substantially reducing the available sample. The design therefore requires reconsidering the tradeoffs among duration, effective sample size, and the quantity being estimated; a power formula concerned only with a fixed effect does not address this layer.
The two authors also point toward different solutions. Bronzini takes a design-based approach, seeking to identify the steady-state effect directly by restricting who is enrolled. In his conclusion, he acknowledges that longer runs or dynamic models may also approximate the steady-state effect under additional assumptions, but argues that first-exposure randomization is the minimum design condition that does not depend on assumptions about habituation. Gelman leans toward a modeling approach, incorporating time and treatment history into the model as sources of variation.
Scope of the Evidence and Open Questions
Several levels of claims need to be distinguished when reading these materials. Gelman’s agreement is general: he says Bronzini has “identified a real problem,” rather than endorsing every claim in the white paper, such as the assertion that “traditional experiments mostly measure novelty responses.” The white paper’s calculations and illustrations use parameters chosen by the author, not data from actual experiments, and it provides no public dataset. The author himself also says that the magnitude of novelty responses in a particular business needs to be quantified case by case.
Another issue concerns differences between user populations (this paragraph is this article’s analysis). Including only new users identifies the effect when “new users see B from the outset.” The white paper argues that this cohort’s exposure sequence matches that of the population after full rollout, but whether returning users’ long-term responses after adapting to B are the same as those of new users still requires an additional assumption. If the two types of users differ to begin with, generalizing findings from the clean cohort to all users may still introduce bias. This is precisely where user type and exposure history need to be modeled together. In addition, using first-visit time as a cutoff assumes that the experiment’s start really is the switchover point, with no delayed serving of the old version afterward; the author also notes this in the article.
References
Gelman, A. (2026, October 23). A big problem with conventional experimentation or A/B testing is that the treatment effect can vary over time and it can depend on the entire history of the treatment. Usual comparisons miss this point. Statistical Modeling, Causal Inference, and Social Science. https://statmodeling.stat.columbia.edu/2026/10/23/a-big-problem-with-conventional-experimentation-or-a-b-testing-is-that-the-treatment-effect-can-vary-over-time-and-it-can-depend-on-the-entire-history-of-the-treatment-usual-comparisons-miss-this-po/
Bronzini, A. (2025, December 15; updated 2025, December 17). The First-Exposure Contamination Problem and the Requirement of First-Exposure Randomization for Valid Digital Experimentation. ConfidentStory.com. https://confidentstory.com/assets/files/THE_FIRST_EXPOSURE_CONTAMINATION_PROBLEM_251217.pdf
Gelman, A. (2024, January 3). It’s about time. Statistical Modeling, Causal Inference, and Social Science. https://statmodeling.stat.columbia.edu/2024/01/03/its-about-time-my-talk-for-the-upcoming-ny-r-conference/
Research Methods · Rituals can be imitated; understanding cannot be faked
Information compiled and published by Scholay One-Stop Academic Platform. Please report any infringement for removal.
Also published on EnterScholar, a community for scholars in the age of intelligence
Originally published at Metho.


Top comments (0)