How a mouse-brain experiment reveals a challenge shared by biological learning and reinforcement learning systems.
The Problem With Yesterday's Winning Move
Picture a simple learning agent with two buttons. Button A pays out 80% of the time; button B pays out 20%. After a few hundred trials, the agent's value estimates settle where you would expect, and it presses A almost every time. By any reasonable metric, it has learned.
Then, without announcement, the payouts swap.
Nothing in the agent's input tells it the world has changed. It still sees two buttons. Its value table still says A is worth about 0.8. So it keeps pressing A, collecting mostly nothing, and its estimate of A drifts downward only as fast as its update rule allows. Depending on how that rule is written, "as fast as it allows" can mean twenty trials or five hundred.
This is the problem hiding inside many deployed decision systems: learning what works is a different task from noticing that what worked has stopped working. A recommender tuned to last year's catalogue or a scheduler calibrated to traffic that a product launch just rewrote may not use reinforcement learning at all, but each carries its training conditions forward as assumptions.
Animals face this problem constantly, and a recent study offers an unusually detailed look at one piece of how a mammalian brain responds when a reliable reward fails to appear.
What the Mouse Experiment Actually Showed
The study, "Spatially heterogeneous acetylcholine dynamics in the striatum promote behavioral flexibility", by Gideon Sarpong, Jeffery Wickens and colleagues, led from the Okinawa Institute of Science and Technology (OIST), appeared in Nature Communications on 17 December 2025. It reached a wider audience about six months later, through an institutional press release carried by ScienceDaily in June 2026.
The task. Head-fixed mice ran on a floating ball through a virtual-reality Y-maze. One arm delivered a drop of sugar water; the other delivered a black screen. Nothing visually marked the rewarded arm, so the mice had to discover it by trial and error. Once a mouse chose correctly in at least 80% of a session's trials, the researchers reversed the contingency—again without any cue. The previously rewarded arm now paid nothing.
Behaviorally, the mice did what you would predict: they kept choosing the old arm for a while (average accuracy after the reversal fell to about 47%), then gradually shifted to the new one.
What was measured. The team expressed a fluorescent acetylcholine sensor (iAChSnFR) in the dorsal striatum of 11 mice and imaged it through an implanted lens with two-photon microscopy. Acetylcholine is a neurotransmitter with many jobs, from muscle control to attention and learning. In the striatum—a basal ganglia hub for action selection and habit formation—it comes mostly from a small population of cholinergic interneurons.
Three findings stand out:
Rewards produced dips. During initial learning, a delivered reward was followed by a brief drop in acetylcholine. Ordinary errors produced only smaller dips.
Unexpected non-reward after the reversal produced rises. When a mouse chose the formerly rewarded arm and got nothing, acetylcholine increased across most of the imaged tissue—about 85% of the small (~50 × 50 µm) patches the team analyzed.
The rise tracked what the mouse did next. Signals were larger on failed trials that preceded a switch to the other arm ("lose-shift") than on trials followed by a repeat. Across animals, larger responses correlated with a higher probability of lose-shift (r = 0.67, n = 11).
The responses were also spatially uneven. Patches showing increases sat more toward the medial striatum, with a trend toward anterior positions; patches showing decreases sat more laterally.
What the intervention showed. Correlation alone cannot show that acetylcholine mattered, so in a separate group of ten mice the researchers used a chemogenetic tool to suppress striatal cholinergic interneurons in half of them during reversal. Those mice made fewer lose-shift choices, performed worse in the first reversal session, and needed more trials to reach criterion (about 115 versus 63). Win-stay behavior and running speed were not significantly affected.
What remains interpretation. The inhibition was not time-locked to specific trials, so it shows these neurons are needed for timely adaptation in this task, not exactly when their signal acts. The authors themselves note they cannot yet separate a learning signal from an affective or motivational reaction to the missing reward. Their proposal—that spatially patchy acetylcholine might let new associations form in some striatal territory while sparing old ones elsewhere—is a hypothesis, not a result. And the whole study used a single reversal in a single task, with small groups.
What the study does give us is a striking contrast: the same outcome—no reward—was followed by opposite acetylcholine responses before and after a well-learned rule broke. The authors read this as a signal of violated expectation. Whatever its precise meaning, that distinction is the crux of the computational problem.
When Expectations Fail: The Computational View
Reinforcement learning formalizes expectation as a value estimate and surprise as a prediction error. A widely used version is the temporal-difference (TD) error from TD(0) learning, as presented in Sutton and Barto's textbook:
text
δt = r_t + γ · V(s{t+1}) − V(s_t)
δt — the temporal-difference error at time t
r_t — the immediate reward received after leaving state s_t
γ — the discount factor (0 ≤ γ ≤ 1), weighting future value
V(s_t) — the agent's estimated value of the current state
V(s{t+1}) — the estimated value of the next state
(Sutton and Barto write the reward as R_{t+1}; it is the same quantity under a different indexing convention.) This is one formulation among many. Q-learning, actor-critic methods, and policy-gradient methods use related but distinct error signals.
A worked example. Suppose an agent has learned that a choice leads to reward 90% of the time, so V(s_t) = 0.9. The trial then ends, so V(s_{t+1}) = 0. Let γ = 0.9.
If the reward arrives (r = 1): δ = 1 + 0.9 × 0 − 0.9 = +0.1
If the reward is missing (r = 0): δ = 0 + 0.9 × 0 − 0.9 = −0.9
With a learning rate of 0.1, the omission nudges the estimate from 0.9 to 0.81.
Now notice something uncomfortable. If the reward really does arrive 90% of the time, a −0.9 error appears on one trial in ten, and the errors average to zero. After a reversal, the identical −0.9 appears on the identical kind of trial. A single prediction error carries no label saying "bad luck" or "the rules changed." It drives a value update; it does not diagnose its own cause. Distinguishing noise from a genuine shift requires integrating evidence across trials and holding some model of how volatile the environment is likely to be. That is change detection, a problem related to—but separate from—prediction error.
This is where a careful reader should resist a tempting shortcut. Dopamine, not acetylcholine, is the neuromodulator classically linked to TD-style errors: Schultz, Dayan, and Montague (1997) reviewed evidence that the phasic firing of primate dopamine neurons resembles a reward prediction error. Striatal acetylcholine is not a copy of that signal; Duhne and colleagues (2024) found that, in rats, cholinergic pauses did not line up in time or location with dopamine prediction-error signals. Nothing in the mouse study shows acetylcholine computing the equation above. The equation is an educational bridge—a way of stating precisely what problem the mice were solving, not a description of how they solved it.
Exploration Versus Exploitation
Even an agent that suspects something has changed still has to decide what to do about it. Reinforcement learning frames this as a tradeoff:
Exploitation — keep taking the action that currently looks best.
Exploration — try alternatives that might reveal something better.
In a changing world, too little exploration anchors an agent to an outdated policy; too much throws away behavior that was still working.
The simplest strategy, epsilon-greedy selection, exploits most of the time and picks a random action with small probability ε. It is simple and gives every action a nonzero probability of being sampled, though no finite run guarantees that every action will be selected. But exploration alone does not make an agent adaptive; what it does with the information matters as much as gathering it.
The simulation below is the opening scenario in code: arm 0 pays out with probability 0.8 and arm 1 with 0.2, and the probabilities swap at step 500. Both agents use ε = 0.1 and differ only in how they average rewards. The first uses a sample average, so every reward in its history counts equally. The second uses a constant step size (α = 0.1), so recent rewards count more—a standard approach Sutton and Barto describe for tracking nonstationary problems.
python
"""Two-armed bandit whose reward probabilities reverse partway through.
Simulated agents only. Standard library, Python 3.8+.
Arm 0 pays 1 with p=0.8 and arm 1 with p=0.2, then they swap at step 500.
Both agents use epsilon-greedy selection (epsilon = 0.1). They differ in how
they update value estimates:
sample average : step size 1/n (every past reward weighted equally)
constant step size : alpha = 0.1 (recent rewards weighted more)
"""
import random
STEPS, SWITCH, RUNS, EPSILON = 1500, 500, 1000, 0.1
P_BEFORE, P_AFTER = [0.8, 0.2], [0.2, 0.8]
def run(rng, alpha=None):
q, n = [0.0, 0.0], [0, 0]
chose_best = []
for t in range(STEPS):
probs = P_BEFORE if t < SWITCH else P_AFTER
if rng.random() < EPSILON:
a = rng.randrange(2)
else:
a = 0 if q[0] > q[1] else 1 if q[1] > q[0] else rng.randrange(2)
r = 1.0 if rng.random() < probs[a] else 0.0
n[a] += 1
step = alpha if alpha is not None else 1 / n[a]
q[a] += step * (r - q[a]) # the bandit form of a prediction error update
chose_best.append(a == probs.index(max(probs)))
return chose_best
def summarize(label, alpha):
rng = random.Random(42) # fixed seed: identical numbers on every run
totals = [0] * STEPS
for _ in range(RUNS):
for t, ok in enumerate(run(rng, alpha)):
totals[t] += ok
rate = [x / RUNS for x in totals]
before = sum(rate[SWITCH - 100:SWITCH]) / 100
after = sum(rate[SWITCH:SWITCH + 200]) / 200
late = sum(rate[-100:]) / 100
first_crossing = next(t - SWITCH for t in range(SWITCH, STEPS) if rate[t] > 0.5)
print(f"{label:<20} best-arm rate | last 100 before switch: {before:.2f}"
f" | first 200 after: {after:.2f} | final 100: {late:.2f}"
f" | first step >50%: {first_crossing}")
summarize("sample average", None)
summarize("constant alpha=0.1", 0.1)
Simulated output (seed 42, 1,000 runs per agent). Each rate is the share of runs choosing the currently better arm, averaged over the stated window. "First step >50%" is the first post-switch step at which more than half of the runs chose the new best arm. It is an illustrative threshold crossing, not a per-agent recovery time or a measure of sustained adaptation:
text
sample average best-arm rate | last 100 before switch: 0.95 | first 200 after: 0.05 | final 100: 0.94 | first step >50%: 519
constant alpha=0.1 best-arm rate | last 100 before switch: 0.95 | first 200 after: 0.83 | final 100: 0.95 | first step >50%: 21
Before the switch, the two agents perform almost identically by this metric. Afterward, the sample-average runs mostly stay on the old arm: only 5% of their choices in the first 200 post-switch steps go to the new best arm, and the majority does not cross over until step 519—even though every agent keeps exploring with the same probability throughout. Each agent's estimate for arm 0 is anchored by hundreds of earlier successes, so each new failure moves it only by 1/n. The constant-step runs cross the 50% line by step 21 and stay above it.
This is not a verdict that constant step sizes are better. They never fully converge, so in a truly stationary problem they keep reacting to noise. Neither agent detects the change; one simply forgets faster. More principled methods model the change explicitly—sliding-window and discounted UCB base their confidence estimates on discounted or recent observations, and Bayesian online change-point detection maintains a running probability over how long ago the last change occurred. Each embeds an assumption about how often the world shifts, and each pays for it when that assumption is wrong.
What Neuroscience Can—and Cannot—Teach AI Engineers
The legitimate connection between the mouse study and the bandit is a shared problem, not a shared mechanism. Both must answer the same questions. Was this failure noise or signal? How many failures justify reconsidering a learned strategy? When the strategy changes, should the old one be erased or kept in reserve?
The neuroscience offers useful prompts:
Context-dependent responses to failure. In the study, ordinary errors during learning and unexpected failures after a well-learned rule broke produced different acetylcholine signals. If the authors' reading is right, the response to a bad outcome depended on how established the expectation had been. A fixed ε has no such sensitivity; it explores at the same rate whether the agent is confident or confused.
Uncertainty as a control variable. Computational neuroscientists have long proposed that neuromodulators report different kinds of uncertainty. Yu and Dayan (2005) suggested that acetylcholine signals expected uncertainty and norepinephrine signals unexpected uncertainty, such as an unsignaled context switch. That theory concerned cortical circuits, and a reversal like the one in this study would arguably fall in its "unexpected" category—a reminder that mapping neuromodulators onto computational variables remains unsettled.
Adapting without erasing. The authors' hypothesis that patchy acetylcholine might protect old associations while new ones form echoes a problem continual-learning researchers know well: catastrophic forgetting. It is an intriguing parallel, but it is a hypothesis about mice, not evidence for any AI method.
The limits need stating just as plainly:
A single-reversal task in small groups of mice cannot establish how human decision-making works.
Artificial agents have no cholinergic interneurons; "neuromodulation" in a model is a design analogy.
Similar behavior does not imply similar internal processes.
The study proposes no AI algorithm.
Nothing in it shows that copying acetylcholine dynamics would improve agent performance.
Biology can inspire a computational hypothesis; only experiments on the computational system can validate it.
The Bigger Engineering Lesson
Return to the two-button agent. Before the switch, it looked excellent: 95% optimal choices. That number described its performance under the original reward structure, but it did not reveal how slowly the agent would adapt when that structure changed. Robustness to change shows up only when you test under change.
That is why optimizing against a fixed reward structure is not enough. Adaptive systems need a way to update expectations, an estimate of how much to trust those expectations, and a policy for responding when they fail. None of these settings is universally correct. Switching after every disappointment is just as maladaptive as never switching; the switching the mouse study examined followed failures that violated a well-established rule, not every missing reward.
The mice did not solve reinforcement learning. They did illustrate, with unusual precision, where the hard part lies.
Learning which decisions lead to reward is only half of intelligence. The other half is recognizing when yesterday's right decision has quietly become today's mistake.
References
Sarpong, G. A., Pass, R., Liyanagama, K., Chu, K.-Y., Kurima, K., Akamine, Y., Chouinard, J. A., Looger, L. L., & Wickens, J. R. (2025). Spatially heterogeneous acetylcholine dynamics in the striatum promote behavioral flexibility. Nature Communications, 16, 10877. https://doi.org/10.1038/s41467-025-66826-1
Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. http://incompleteideas.net/book/the-book-2nd.html
Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599. https://doi.org/10.1126/science.275.5306.1593
Duhne, M., Mohebi, A., Kim, K., Pelattini, L., & Berke, J. D. (2024). A mismatch between striatal cholinergic pauses and dopaminergic reward prediction errors. Proceedings of the National Academy of Sciences, 121(41), e2410828121. https://doi.org/10.1073/pnas.2410828121
Yu, A. J., & Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron, 46(4), 681–692. https://doi.org/10.1016/j.neuron.2005.04.026
Garivier, A., & Moulines, E. (2011). On upper-confidence bound policies for switching bandit problems. In Algorithmic Learning Theory (ALT 2011). Springer. https://doi.org/10.1007/978-3-642-24412-4_16
Adams, R. P., & MacKay, D. J. C. (2007). Bayesian online changepoint detection. arXiv:0710.3742. https://arxiv.org/abs/0710.3742
Okinawa Institute of Science and Technology Graduate University. (2026, June 8). Scientists discover the brain chemical that helps you break bad habits. ScienceDaily. https://www.sciencedaily.com/releases/2026/06/260606075901.htm
Top comments (1)
Strong, clear bridge between a careful neuroscience result and a real RL engineering problem.
The distinction you draw is especially useful: a single negative prediction error is ambiguous (noise vs. regime shift), so adaptation requires something beyond the basic TD update—whether that’s a faster effective learning rate, an explicit change-point model, or a context-sensitive exploration policy. The mouse data (opposite ACh responses to the same outcome before vs. after the rule is well learned, plus the chemogenetic effect on lose-shift) makes that computational distinction concrete rather than hand-wavy.
One practical implication that stands out: many production systems still rely on something closer to sample-average or slowly decaying estimators under the assumption of quasi-stationarity. Your simulation result (constant α recovering in ~20 steps vs. sample average taking hundreds) is a clean reminder that the choice of update rule is itself a statement about how volatile the world is expected to be. Pairing that with an uncertainty or volatility estimate (as in some Bayesian or meta-learning approaches) feels like a natural next design lever—exactly the kind of question the ACh findings prompt without claiming the biology supplies the algorithm.
Curious whether you’ve seen any recent work that tries to make the “was this failure expected under the current context?” distinction more explicit in deep RL or continual-learning setups. The post already does the hard work of keeping the analogy honest; that restraint is part of what makes it valuable.