DEV Community: Mitansh Gor

RL 3: Bellman Equations and Markov Decision Processes (1950s–1960s)

Mitansh Gor — Fri, 18 Sep 2026 20:47:23 +0000

Where we left off

Two quick reminders.

Thorndike (1898) watched cats escape a puzzle box and gave us the Law of Effect: do something, get a good result, do it again. Get a bad result, stop. Learning is bookkeeping over consequences.

Blog 2 turned that idea into hardware. Shannon's mouse ran a maze and remembered the route. Minsky's SNARC strengthened the wires that led to good outcomes.

Both worked. Both were missing the same thing.

Thorndike's cat knew "pulling that loop felt good." It did not know it was three moves from freedom. Shannon's mouse could replay a path, but it couldn't tell you why one path was better than another.

Nobody had a word for this yet:

Value — a number attached to a situation that says how good does the future look from right here?

That one missing number is what this whole post is about.

Side note: the biologists were chasing the same number

Worth thirty seconds, because it makes the maths feel less alien.

Rescorla and Wagner (1972) showed that animals don't learn from repetition. They learn from surprise. If a bell already predicts food perfectly, adding a light next to the bell teaches the animal almost nothing about the light. There's no surprise left to learn from.

Their rule, in plain English:

new belief = old belief + (small step) × (what actually happened − what I expected)

That last bracket — reality minus expectation — is called prediction error. Hold onto it. It comes back in Blog 4 with a much bigger job.

(Yes, 1972 is out of order here. I'm putting it early because it explains "value" better than a grid does.)

Enter Richard Bellman

In 1949, a mathematician named Richard Bellman started working at RAND, a US think tank. His job was a specific shape of problem: you make a choice, the world changes, then you have to choose again. Missile guidance. Supply planning. Anything where today's decision quietly reshapes tomorrow's options.

He cracked it. Then he needed a name for it, So he picked two words that sounded harmless and industrial: Dynamic Programming.

(The story has probably been tidied up over the years — the dates don't quite line up — but the name stuck, and it's now one of the most important terms in computer science.)

Here's the problem he was actually solving.

The problem, in the form we'll use all post

Forget missiles. Here's the setup we'll use for everything that follows.

You're on a grid. You start in one corner. The goal is somewhere else. Every step costs you a little energy. Some cells are bad news. You want to reach the goal having collected as much reward as possible.

Easy to describe. Genuinely nasty to solve systematically.

The obvious approach is to list every possible path and pick the best. That works right up until it catastrophically doesn't. Paths multiply exponentially with grid size. 10×10 is already unpleasant. 100×100 is hopeless. Bellman later gave this failure mode a name — the curse of dimensionality — and we'll come back to it at the end, because it's the villain of the whole post.

So he needed a fundamentally different kind of reasoning.

Dynamic Programming: solve it backward, and only solve it once

Bellman's core insight is called the Principle of Optimality. Here it is in his words:

"An optimal policy has the property that whatever the initial state and initial decision are, the remaining decisions must constitute an optimal policy with regard to the state resulting from the first decision."

That's a mouthful. In human:

If the best route from A to C goes through B, then the B-to-C part of it must itself be the best route from B to C.

Because if it weren't — if there were a better way from B to C — you'd just swap it in and get a better A-to-C route. Contradiction. Done.

Sounds trivial. It isn't, because of what it licenses you to do:

You only ever have to solve each sub-problem once.

Once you know the best way from B onward, every route that passes through B can reuse that answer for free. No recomputation. That's Dynamic Programming: chop a giant sequential problem into overlapping sub-problems, solve them from the end backward, and cache the answers.

The backward part is the counterintuitive bit and it's worth sitting with. Bellman didn't ask "what's my best first move?" He asked "what's my best last move?" Then second-to-last. Then third-to-last, propagating backward until every square on the board has an answer.

This is backward induction, and it's not a historical curiosity — it is still, structurally, how modern RL agents plan.

Hold onto this: the future is easier to reason about than the past, because the future has an ending you can anchor to.

Five nouns you need before the equation shows up

So: five nouns first. Equation after. All five together are called a Markov Decision Process (MDP) — the formal container the maths lives in.

1. State — s
Where you are. On the grid, "row 3, column 4." In chess, the board position. In a delivery robot, position plus battery plus whether it's holding a package. The full set of states is written S.

2. Action — a
What you can do from here. On the grid: up, down, left, right. The full set is A.

3. Transition — P(s' | s, a)
Read it as: "the probability of landing in state s', given that I was in s and did a." The apostrophe just means "next."

4. Reward — R(s, a)
A number the environment hands you for doing a in s. This is where you, the designer, encode what "good" means. Reach the goal: +10. Take a step: −1. Fall in the pit: −100.

Get this wrong and your agent will cheerfully optimise the wrong thing. Reward design is 90% of applied RL pain, and I'm not going to pretend otherwise.

5. Policy — π (pi)
Your strategy. A rule that maps state → action. "When in this square, go right." That's it. A policy isn't a plan or a path — it's a lookup table of what to do anywhere you might find yourself.

The whole field, in one sentence: an MDP describes the world; a policy describes your behaviour in it; and the rest of RL is the search for the best policy.

The one assumption holding all of this up

There's a condition the whole framework quietly depends on, and it's called the Markov Property:

The future depends only on the present state — not on the history of how you got there.

First reaction to this is usually: that's obviously false, history matters all the time. Fair. But that's not quite what it says.

It says history must already be baked into the state.

If your robot's battery level affects the next decision, then battery level belongs in the state definition. Once it's in there, you don't need to replay the last 50 timesteps — you just look at where you are now.

Two quick examples to make it concrete:

Chess is Markov. The board position tells you everything. It doesn't matter whether you reached it through a brilliant sacrifice or a series of blunders — the position is the position.
Poker is not Markov if your state is just "my cards." Whether your opponent has been bluffing all night genuinely changes what you should do. Fix it by expanding the state to include betting history. Now it's Markov again — and much bigger. (Notice the trade you just made. That's foreshadowing.)

The neat way to say it: the state is allowed to have memory. The policy isn't.

This assumption is exactly what makes RL computable. Without it, an agent would have to carry and process its entire past at every step. With it, a policy is just a lookup: current state in, action out.

One more assumption, and it's a big one. For this entire post, the agent is assumed to already know P and R. It knows the physics of the world and the reward structure before it takes a single step. That's called model-based RL, and it's what makes the maths clean here.

It is also completely unrealistic, and Blog 4 takes a sledgehammer to it.

Return and γ: why "how good" needs a definition

One more thing before the equation. If value means "how promising is the future from here," we need to say what the future means numerically.

Add up all the reward you'll collect from now until the end. That total is called the return.

Except there's a problem: if the task never ends, that sum runs to infinity, and "infinity" is a terrible thing to compare against another infinity. So Bellman introduced a discount factor, γ (gamma), a number between 0 and 1. Reward one step away is multiplied by γ. Two steps away, γ². Three steps, γ³. And so on.

G_{t} = R_{t} + γ R_{t + 1} + γ^{2} R_{t + 2} + γ^{3} R_{t + 3} + \dots

Two things fall out of this, and both matter:

Mathematically: the sum now converges. It's a geometric series. No infinities.
Behaviourally: γ is a personality dial.
- γ near 0 — a myopic agent. Only right now exists. It will grab the nearest +1 and ignore a +100 two steps away.
- γ near 1 — a patient agent. Happy to eat a −1 today for a +100 later.
- γ = 0.9 — a reasonable default that says "about ten steps of foresight."

It's compound interest, run backward. A reward ten steps away at γ = 0.9 is worth about 35% of its face value today.

Value: the number Thorndike's cat never had

Now we can define the thing this whole post is chasing.

V(s) — the value of state s — is the expected return you'll collect if you start in s and behave well from there on.

That's the number the cat didn't have. The cat knew "this felt good." A value function knows "this square is worth 7.1, that one's worth 5.4, so go left."

And that's the whole trick: once every square has a number, choosing an action stops being a search problem and becomes a comparison. Look at the neighbouring numbers. Walk toward the big one. Done.

The Bellman Equation

In his 1957 book Dynamic Programming, Bellman wrote down the recursion that ties all of this together:

V (s) = a max [R (s, a) + γ s^{'} \sum P (s^{'} ∣ s, a) V (s^{'})]

In one sentence:

The value of where you are = the best immediate reward you can grab + the discounted value of wherever you end up next.

Every symbol is one of the five nouns you already know:

Piece	Reading
`V(s)`	how good it is to be here
`max_a`	you, choosing. Try every action, keep the best
`R(s, a)`	what you get right now
`γ`	how much you care about later
`∑ P(s':s,a) · V(s')`	the world, responding.Average over every place you might land, weighted by likelihood

Two things I want to flag, because they're where people quietly lose the thread.

First: max and ∑ are doing opposite jobs. The max is the agent choosing. The ∑ is the world rolling dice. You control your action. You do not control the outcome. If the transitions were deterministic, that ∑ would collapse to a single term and disappear — it only exists because the world is uncertain.

Second: V(s) appears on both sides. That's not sloppy notation, it's the entire idea. The equation is self-referential — the value of a state is defined in terms of the values of other states.

Which raises the obvious question: if V is defined in terms of V, how do you ever compute it?

You guess, then you improve. Start with V = 0 everywhere (a wrong but harmless guess), apply the equation, get better numbers, apply again. The values stop changing when they're all consistent with each other. Mathematicians call that a fixed point — the place where applying the rule doesn't change anything anymore.

That "keeps changing until it doesn't" process is the entire computational game. And it's easier to believe once you've watched it happen, so let's watch it happen.

Let's actually put numbers on the board

Here's the smallest example that shows the real behaviour. A corridor, four squares, then the goal:

[ S1 ][ S2 ][ S3 ][ S4 ][ GOAL ]

Rules:

Actions: move left or move right. Deterministic — no ice.
Every move costs −1.
Stepping into GOAL pays +10.
γ = 0.9.
GOAL is terminal, so V(GOAL) = 0.

Work backward, exactly as Bellman said to.

S4 is one step from GOAL: V(S4) = −1 + 10 + 0.9 × 0 = 9.00
S3 is one step from S4: V(S3) = −1 + 0.9 × 9.00 = 7.10
S2: V(S2) = −1 + 0.9 × 7.10 = 5.39
S1: V(S1) = −1 + 0.9 × 5.39 = 3.85

[ 3.85 ][ 5.39 ][ 7.10 ][ 9.00 ][ GOAL ]

Look at what happened. The goal's reward leaked backward through the corridor and left a slope behind it. The agent doesn't need a map, a plan, or a search. It just walks uphill. The value function turned a planning problem into a gradient you can follow like a scent trail.

Animated_heatmap_of_the_x_gr.mp4 - Google Drive

drive.google.com

Now watch it converge from nothing

That was the answer. Here's the process — starting from V = 0 everywhere and sweeping the equation across all four states repeatedly:

Sweep	V(S1)	V(S2)	V(S3)	V(S4)
0	0.00	0.00	0.00	0.00
1	−1.00	−1.00	−1.00	9.00
2	−1.90	−1.90	7.10	9.00
3	−2.71	5.39	7.10	9.00
4	3.85	5.39	7.10	9.00
5	3.85	5.39	7.10	9.00

Three things worth noticing here, and honestly this table is the most important object in the post:

Information travels exactly one square per sweep. Sweep 1, only S4 knows the goal exists. Sweep 2, S3 finds out. It's a wavefront moving backward.
The early numbers are hilariously wrong. At sweep 3, S1 thinks it's worth −2.71 — it can only see step costs and no payoff yet. It's a pessimist with incomplete information. It gets over it.
You don't need a stopping rule, you need a stopping condition. When a full sweep changes nothing, you're done. That's the fixed point, made of actual numbers.

Howard's Policy Iteration: from knowing values to having a strategy

Bellman gave us the value of every square. But there's a subtly separate question: how do you systematically build the strategy — the policy — that goes with it?

Ronald Howard, in his 1960 monograph Dynamic Programming and Markov Processes, noticed the problem is circular:

To compute V(s), you need to know which policy π the agent is following.
To find the best π, you need to know V(s).

Classic chicken and egg. Howard's answer: stop trying to do both at once. Alternate.

Step 1 — Policy Evaluation: "how good is my current strategy?"

Freeze the policy. Don't try to improve it. Just ask: if I followed this exact strategy forever, what would each state be worth?

V^{π} (s) = R (s, π (s)) + γ s^{'} \sum P (s^{'} ∣ s, π (s)) V^{π} (s^{'})

Notice the max is gone. There's no choosing here. π(s) tells you what to do; you're just measuring the consequences. This isn't optimisation, it's scorekeeping.

Step 2 — Policy Improvement: "could I do better?"

Now take those scores and ask, at every state: given what I now know things are worth, is there an action better than the one my policy currently prescribes? If yes, switch to it.

π^{'} (s) = ar g a max [R (s, a) + γ s^{'} \sum P (s^{'} ∣ s, a) V^{π} (s^{'})]

arg max vs max, since this trips everyone up: max returns the best value. arg max returns the action that achieved it. Evaluation wants the number; improvement wants the move.

And repeat

Evaluate. Improve. Evaluate. Improve. Stop when an improvement step changes nothing.

Howard proved two things about this loop that are genuinely reassuring:

The greedy step never makes the policy worse. Every improvement is an improvement or a tie. It can't wander off toward a worse policy — the local, greedy move is globally safe here. (That's a strong guarantee, and one that quietly evaporates once we start using neural networks in later posts.)
On a finite MDP, it terminates in a finite number of iterations. There are only finitely many policies, you never repeat one, so you must stop. Usually in a handful of rounds.

And here's the part that should give you a small jolt: that two-role split — one part measuring, one part deciding — is the direct ancestor of the Actor–Critic architectures we'll get to in Blog 7. The Critic does Policy Evaluation. The Actor does Policy Improvement.

That structure wasn't invented for deep learning. It was sitting in a 1960 operations research monograph the whole time.

Value Iteration: the impatient version

Policy Iteration is a bit fussy. It insists on fully evaluating a policy — running evaluation to convergence — before it'll consider improving anything.

Value Iteration says: why wait? Fold the improvement directly into the update.

V (s) \leftarrow a max [R (s, a) + γ s^{'} \sum P (s^{'} ∣ s, a) V (s^{'})]

The max is back, and it's inside the loop. Every single update implicitly re-decides the best action. No separate improvement phase exists.

Value Iteration never stores a policy at all. It just hammers on the value function until it settles, and then you read the policy off: stand in each state, look at the neighbours, take the action pointing at the biggest number.

(That sweep table you looked at earlier? That was Value Iteration. You've already seen it run.)

Policy Iteration vs Value Iteration, side by side

Same equation, same destination, different temperament.

Policy Iteration is the coach who watches the entire game. Films it. Writes a 40-page report. Rewrites the playbook from scratch before the next match. Few iterations, each one expensive and each one a big leap.

Value Iteration is the coach who stops play after every snap. "Stop — standing three feet left would've been worth two more points." Cheap per update, but it needs a lot of them. Constantly nudging everyone's understanding of the field.

	Policy Iteration	Value Iteration
Explicit policy?	Yes, maintained separately	No, implicit in the values
`max` in the update?	Only in the improvement step	Every update
Cost per iteration	High (full evaluation)	Low (one sweep)
Iterations needed	Few	Many
Terminates when	Policy stops changing	Values stop changing
Best when	State space is small and exact	State space is large or approximate

Which should you use? Small, well-defined state space where you want exactness — Policy Iteration. Large or approximate — Value Iteration.

And worth flagging for later: almost every modern RL algorithm is spiritually a descendant of Value Iteration. They update values continuously and never wait for a full evaluation to finish. Q-learning is Value Iteration with the model ripped out. DQN is Q-learning with a neural network glued on.

The curse they named but couldn't escape

Now the bad news, and Bellman himself is the one who delivered it.

Both algorithms require sweeping over every state in S. Every single one. Every iteration.

4-square corridor: 4 states. Trivial.
10×10 grid: 100 states. Fine.
Chess: roughly 10⁴⁴ positions.
Go: more legal positions than there are atoms in the observable universe.

Bellman named this the curse of dimensionality, in the same 1957 book that gave us the equation. Add one variable to your state description and the state space doesn't grow — it multiplies. A robot tracking position, velocity, battery, and grip status doesn't have four problems. It has (positions × velocities × battery levels × grip states) problems.

And remember the poker example from earlier? Fixing the Markov violation by stuffing betting history into the state made the state space explode. That's the curse arriving in person. Making a problem Markov and making it small are usually opposing goals.

This is why RL went quiet for decades after Bellman. It wasn't that the theory was shaky. The theory was perfect and the computer was too small. Dynamic Programming solves the structure of sequential decision-making completely — it just needs a table with one row per state, and sometimes that table can't exist.

The escape route — stop storing V(s) in a table and start approximating it with a function — wouldn't properly arrive until DeepMind's DQN in 2013. But the problem was named here, in 1957. Nearly everything between then and now is an attempt to get around it.

Why 1957 mathematics is running inside your 2026 chatbot

It sounds like a stretch that equations written for missile guidance underpin modern reasoning models. It isn't.

When a model like o1 or DeepSeek-R1 works through a hard problem, it isn't just emitting the most likely next token. It's exploring reasoning paths, scoring how promising each one looks, and abandoning the ones heading toward a dead end.

Read that again with this post's vocabulary loaded:

A partial chain of reasoning is a state.
The next thing to write is an action.
"Is this line of thinking going anywhere?" is a value estimate.
Backtracking out of a bad path is backward induction — the same move Bellman made when he asked about the last step instead of the first.

The state space changed from grid squares to token sequences. The tables became neural networks. The core question — how good is this situation, really? — is unchanged since 1957.

Six things to keep

If every equation falls out of your head by Thursday, keep these:

The world is states and actions — that's an MDP.
Rewards define what "good" means — and you're the one writing them, so be careful.
Value is a score for how promising the future looks from here — the number Thorndike's cat never had.
The Bellman Equation makes future reward flow backward through states, leaving a slope you can climb.
A policy is just a rule for picking actions — a lookup table, not a plan.
Iteration fixes the circularity. Guess, measure, improve, repeat, stop when nothing changes.

If Bellman clicks, the rest of RL is refinement.

What's coming next

Everything in this post rested on one assumption I flagged and then leaned on hard: the agent already knows P and R. It knows the physics. It knows the payoffs. It's not learning about the world — it's calculating against a world it already has a complete map of.

Real agents don't get that map.

In Blog 4 — Early Heuristics and the Birth of Temporal Difference Learning (1959–1968), that assumption gets torn up. We'll meet a program that had to work out what positions were worth by playing, not by computing — and we'll finally see what happens when you take Rescorla and Wagner's "learn from surprise" idea and let it fire on every single step instead of once per trial.

Remember the prediction error term from the top of this post? It's about to get a job.

The equation was ready. The machine was next.

Next: Blog 4 — Early Heuristics and the Birth of Temporal Difference Learning (1959–1968).

Previously: Blog 1 — Biological Foundations and the Law of Effect (1898–1949) · Blog 2 — The First Machines.

RL 2: The first physical and computational RL machines(1948–1954)

Mitansh Gor — Sat, 12 Sep 2026 05:57:48 +0000

Quick recap, if you're just arriving: Blog 1 showed that every core idea in reinforcement learning was discovered by someone watching animals, not computers. Thorndike's cats taught us that learning is selection, not insight. Pavlov's dogs taught us that a signal can carry information about the future. Skinner's pigeons taught us that big behaviors are built from small rewarded steps. Hebb's neurons taught us that learning lives in the strength of a connection.

Four beautiful observations. Not one of them tells you how to build something that does this.

This is the decade where people tried to build it anyway. What followed was ten years of beautiful, clunky, brilliant hardware — machines that stumbled through mazes, adjusted their own wiring, and learned, in the most primitive sense, from their mistakes. None of the people building them knew they were inventing Reinforcement Learning. They were just trying to make something move on its own.

The Missing Link: Why "a signal predicts the future" Needed a Number

Before we jump to the machines, one 20-second detour — because it's the cleanest bridge between Blog 1 and everything below.

Pavlov noticed his dogs salivated at footsteps, before food even showed up. Nice story. But it doesn't tell you how much the dog should update its expectation each time, or why the association eventually stops getting stronger.

Decades later — 1972, well after everything in this post — psychologists Robert Rescorla and Allan Wagner finally wrote that down as a formula:

Δ V = α β (λ - V)

In plain words: how much you learn from an event equals how surprising it was. V is what you currently expect, λ is what actually happened, and (λ - V) is the surprise — the gap between prediction and reality. If a bell always predicts food, eventually there's no surprise left, and no more learning happens. That gap term is the whole engine.

Why mention a 1972 psychology paper in a post about 1948–1954 engineering? Because the people you're about to meet — Wiener, Turing, Minsky, Shannon — independently backed into the exact same idea from a completely different direction: engineering, not biology. None of them had read Pavlov's fine print. They just kept discovering, on their own, that intelligence seems to boil down to measuring a gap and closing it. Keep that gap-closing idea in your head — it's the thread connecting every section below.

The Cybernetic Spark: Wiener's Feedback Loops (1948)

In 1948, mathematician Norbert Wiener published Cybernetics: Or Control and Communication in the Animal and the Machine, and made a claim that was radical for its time: the same mathematical principle governs both living organisms and machines — negative feedback.

Forget dogs and food for a second. Think of a thermostat. It doesn't "know" temperature in any deep sense — it just tracks one number obsessively: the gap between where it is and where it's supposed to be.

D = T - S

Where T is the target state and S is the current state. If D ≠ 0, the system fires a correction to close the gap. That's it. That's the whole idea.

This is exactly the Rescorla-Wagner gap from above, just wearing an engineer's clothes instead of a psychologist's. D here is the direct ancestor of the Temporal Difference (TD) error you'll see in modern RL — the signal that tells an agent "something's off, adjust." Every gradient update in every deep RL system running today is, underneath the math, a Wiener-style error minimization.

Wiener's punchline: Intelligence isn't magic. It's error minimization on a loop. If you can define a goal and measure your distance from it, a machine can "learn" to close that distance.

So Wiener gave the field a drive — a reason to move, a number to chase to zero. But a drive isn't a mechanism. It doesn't tell you what happens inside the machine when it's wrong versus when it's right. That's exactly what Alan Turing was working out, in the very same year, from a completely different angle.

The Turing Blueprint: Pain, Pleasure, and the First Neural Network on Paper (1948)

Turing's 1948 paper Intelligent Machinery introduced the P-type (Pain-type) Unorganized Machine, and reading it today feels like finding the first design document for a neural network, forty years too early.

Turing's argument was blunt: a human infant's cortex is mostly a random, disorganized network that only becomes useful through experience. So why not build the machine equivalent? He proposed a network built from NAND gates — think of these as the smallest possible decision units, each taking two inputs and spitting out one output based on a simple rule. Each gate could sit in one of two internal "configuration states," and that state is what determined how the gate behaved.

The learning happened through two signals, and this is the part that matters:

Pain — when the machine got something wrong, the gates that had just fired got scrambled at random:

C_{next} = RandomSample (C)

Pleasure — when the machine got something right, those same configurations froze in place. No more random reshuffling:

P (Δ C ∣ Pleasure) = 0

So: wrong answer, shake things up. Right answer, lock it in. It's almost embarrassingly simple — which is exactly why it's important.

Turing also spotted the obvious problem immediately: pure random shuffling doesn't scale. If a machine has n units, each with k possible states, the number of configurations to search through is k^n — a number that explodes into uselessness almost immediately, even for tiny machines.

His fix: bring in a teacher. Instead of letting the machine flail randomly forever, a teacher hands out Pleasure signals early to nudge it toward useful sub-behaviors, and then Pain prunes away the remaining mistakes. That's not just a clever patch — it's the seed of Supervised Fine-Tuning (SFT) and Imitation Learning, ideas that wouldn't get their modern names for another fifty-plus years. Turing had quietly moved the burden of "being smart" off the programmer's shoulders and onto the environment's.

His paper sat mostly unread for decades. But look at the shape of it: a randomly wired network, shaped over time by reward and punishment, converging on useful behavior. Every RLHF pipeline running behind a modern chatbot is, structurally, still running Turing's 1948 loop.

Wiener gave us the why — close the gap. Turing gave us the what happens inside — freeze on success, scramble on failure. What nobody had shown yet was a machine that could do this outside of a chalkboard, with actual physical parts that changed themselves. That's where Marvin Minsky comes in.

Minsky's SNARC: The First Machine That Rewired Itself (1951–1954)

While Turing's ideas stayed on paper, Marvin Minsky and his Princeton colleague Dean Edmonds went to the workshop and actually built something.

In 1951, they built the SNARC — Stochastic Neural Analog Reinforcement Calculator — the first machine to physically implement Hebbian learning (remember Blog 1: neurons that fire together, wire together) combined with reinforcement, in real hardware. Minsky later wrote it up in his 1954 thesis.

The SNARC simulated a rat finding its way through a maze using 40 artificial synapses. Each "synapse" was a vacuum tube paired with a potentiometer — picture a volume knob, a simple dial that changes how much resistance flows through a circuit. That dial's position was the connection's strength, standing in for what we'd now call a "weight."

Here's the learning loop, in plain terms:

A signal representing the "rat" travels through the network, picking a path.
If that path leads to reward, the potentiometers ("volume knobs") along that path get turned to lower resistance — meaning that path becomes electrically easier to travel next time.
If the path leads nowhere, those knobs stay put or drift the other way.

Notice what's happening here: this is credit assignment, done with actual electricity. The machine had no rule anywhere that said "this specific connection caused the reward." It used the reward signal traveling backward through whichever path was active to figure out which knobs to turn. That's structurally the same job that backpropagation does in a modern neural network — just with copper and vacuum tubes instead of calculus.

The SNARC couldn't scale — 40 synapses is barely a network by today's standards, and analog parts drift and degrade over time. But it proved something that had never been physically shown before: a machine can change its own internal wiring in response to reward, with no one programming the change directly.

That single idea — connections that adjust themselves because of reward — is the founding premise of every neural network trained with RL today.

So now we've got a drive (Wiener), a mechanism (Turing), and proof that self-adjusting hardware works (Minsky). One year later, Claude Shannon showed what happens when you point that same idea at something a person can actually watch move.

Claude Shannon's Theseus: Learning You Can See (1952)

Shannon's contribution was the most tangible of the whole decade. In 1952, he built Theseus — a copper mouse that could navigate a 5×5 maze, controlled by a cabinet of 75 telephone relays sitting underneath.

Here's the trick: the mouse itself had no brain. The relays — think of a relay as a mechanical switch that can remember whether it's "on" or "off" — were the brain. Each of the 25 cells in the maze corresponded to a relay setup that remembered which directions had already failed from that spot. Hit a wall, and the relay for that direction-at-that-position flipped off, so the mouse tried something else next time. Find a working path, and those relay settings locked in for good.

The payoff: drop Theseus anywhere in a maze it had already solved, and it walked straight to the goal, no hesitation. Drop it somewhere new, and it explored from scratch — then remembered that too.

What Shannon had actually built was a physical lookup table: a direct map from state (position in the maze) to action (which way to go), stored as the mechanical position of 75 relays. This is the hardware great-grandparent of what's now called a Q-table.

But the genuinely clever part was the "anti-neurotic" circuit. Shannon noticed that a strategy based purely on "eliminate what failed" could trap the mouse in an infinite loop — try North, fail, try East, fail, loop back to North forever, repeat. His fix: a circuit that detected this looping behavior and injected a random direction change to break it.

This is, as far as anyone can tell, the first deliberate hardware solution to exploration vs. exploitation — the tension that sits at the center of nearly every RL algorithm ever written. In plain terms: should the mouse keep using the move it already knows works (exploit), or try something new in case there's a better option (explore)? Shannon's answer was a random kick when stuck. Modern RL answers with names like epsilon-greedy (occasionally pick a random action on purpose) or entropy bonuses (reward the agent slightly for staying unpredictable). Same problem, seventy years of better tools.

Theseus proved something important: "learning" could be reduced entirely to hardware state changes. No probability distributions, no gradients — just relays flipping. Intelligence as a lookup table.

The Bridge (1954–1957): From Wires to Equations

By the mid-1950s, the hardware approach was hitting a wall. Shannon's mouse could solve one specific maze. Minsky's SNARC topped out at 40 synapses. Turing's ideas mostly lived on paper. Every one of these machines was an impressive, custom-built solution to one specific problem — and none of them generalized.

Richard Bellman, among others, got impatient with that. Building a maze-solving mouse was a great parlor trick, but it wasn't a theory. What he wanted was a general rule — something that could handle any problem made of a long chain of decisions, no matter what the environment looked like underneath.

The question shifted from "can we make this specific thing work?" to "can we prove the mathematically best way to act, in general?"

Bellman's answer became the Principle of Optimality and the Bellman Equation: the idea that the value of being in a state is just the immediate reward, plus the (slightly discounted) value of wherever you end up next. He took the mechanical "vibes" of the SNARC and Theseus and translated them into a single recursive formula — one equation that compresses an agent's entire future into a present-tense number.

That equation, and the computational nightmare it revealed (Bellman's own term for it: the Curse of Dimensionality), is where Blog 3 picks up. For now, the point is simpler: after a decade of wiring things together by hand, the field realized it needed math to go any further.

The wires were beautiful. But equations scale.

The Decade in One Frame

Zooming out, here's what this era actually built — not a summary, a thesis:

Wiener (1948) gave learning a drive: minimize the gap on a feedback loop.
Turing (1948) gave learning a mechanism: reward freezes, punishment randomizes.
Minsky (1951/54) gave learning physical weight: the first adjustable synapses that existed as hardware, not theory.
Shannon (1952) gave learning memory: state-action mappings stored in the physical position of relays.
Bellman (1957) gave learning math: a recursive formula that would eventually unify everything above it — that's Blog 3.

In under a decade, "learning" went from a biological observation about cats and pigeons to a wired, probabilistic, physically real process you could point to on a workbench. The machines were small, fragile, and each one could only do exactly one thing. But the ideas inside them were load-bearing — every one of them is still holding up the RL systems built today.

RL 1: Biological foundations and the "Law of Effect" (1898– 1949)

Mitansh Gor — Wed, 02 Sep 2026 22:58:37 +0000

Before GPUs. Before neural networks. Before transistors were even a rumor.

There was a hungry cat in a wooden crate, trying to get to a piece of fish.

The main character of this era isn't an engineer. It's a loose group of psychologists — some of whom actively disliked each other's ideas — poking at animals to answer one question: why do living things do the things they do?

They never wrote a line of code. They were not trying to build AI. And yet, in the fifty years between 1898 and 1949, they collectively reverse-engineered the core loop that every modern reinforcement learning system still runs on.

This post is that story. No math yet — I promise. Just the loop, and the four people who found its pieces.

First, the thing we're building

Let me give you the shape of the answer before the story, so you know what you're collecting.

Reinforcement learning describes any system that learns by doing stuff and seeing what happens. That's it. Not by being shown correct answers — by trying, failing, and adjusting.

We'll call the learner the agent (the cat, the rat, the robot, the model). Everything outside it is the environment (the box, the room, the game). The loop runs like this:

The agent observes its state — the situation it's currently in.
It picks an action.
The environment responds with a reward — a number saying "that was good" or "that was bad."
The agent lands in a new state, and the loop repeats.

Two more, and they're the ones that do the heavy lifting later.

The agent needs a policy: a strategy for choosing actions. Not a fixed script — a rule of thumb that says in this kind of situation, do this kind of thing.

And it needs a sense of value: how good a state is, based not on the reward sitting in it right now, but on everything it tends to lead to. An empty hallway is worth nothing on its own. An empty hallway with the kitchen at the end of it is worth quite a lot.

Six words: agent, state, action, reward, policy, value. Every single one of them was discovered by someone studying animals, decades before anyone could program a computer.

Let's go get them.

1898 — Thorndike's Cats and the Law of Effect

In 1898, a graduate student named Edward Thorndike was annoyed.

The animal literature of his day was full of charming anecdotes — dogs who understood latches, cats who figured out doorknobs. Thorndike's suspicion was that people were flattering their pets. Animals weren't reasoning. They were doing something else, and nobody had bothered to measure it.

So he built the puzzle box.

The setup is almost comically simple. A hungry cat goes in a small wooden crate. The door opens only if the cat pulls a loop of string, or presses a lever, or steps on a pedal. A piece of fish sits outside, in plain view. Then Thorndike started a stopwatch.

Trial one is chaos. The cat claws the bars. It shoves its paws through every gap. It bites things, yowls, thrashes at anything loose. After a few minutes of pure flailing, it hits the lever by accident. The door swings open. Fish.

Trial two: slightly less flailing.

Trial twenty: the cat walks in, hits the lever, and strolls out.

Here's what Thorndike noticed, and it's the part that matters: there was never a lightbulb moment. No trial where the cat suddenly got it. If cats were reasoning about levers, you'd expect the escape time to fall off a cliff — one long trial, then instantly fast ones. Instead, when he plotted escape time against trial number, he got a slow, noisy, gradual curve. The useless behaviors quietly faded. The useful one got faster.

That curve is the first real evidence in this story. Learning wasn't insight. It was selection — good behaviors surviving, bad ones dying off.

Thorndike wrote it up as the Law of Effect:

If an action is followed by a satisfying state of affairs, the bond between that situation and that action is strengthened. If followed by discomfort, it is weakened.

Read that again with our vocabulary. Situation is the state. Action is the action. Satisfying is the reward. Bond strengthened is the policy update.

Thorndike, in 1898, wrote pseudocode for a policy gradient. He just used the word "bond."

🎒 In the backpack: trial-and-error learning. Rewards make actions more likely. Learning is gradual, not sudden.

But something's missing

Thorndike's cat only learns after it gets the fish. The reward does all the teaching.

But watch a real animal for five minutes and you'll notice it reacting to things that aren't rewards at all. Your dog loses its mind at the sound of a cupboard opening. Nothing has been eaten yet. The cupboard predicts food.

Thorndike had no account of that. Rewards were where his story started and ended.

The person who noticed the signal was, characteristically, not looking for it.

The 1900s–1920s — Pavlov and the Problem of Prediction

Ivan Pavlov was a Russian physiologist studying digestion. He won a Nobel Prize for it in 1904. He was, by most accounts, mildly irritated that history remembers him for the dogs.

His actual research required measuring saliva. And his dogs kept ruining the measurements by salivating early — at the sight of the lab assistant who usually brought food, at the sound of footsteps in the corridor. Before any food existed.

Most researchers would call that noise and try to control for it. Pavlov called it the experiment.

He ran it systematically. Present a neutral signal — a metronome, a buzzer, a light, a tone — then deliver food, a second or two later. Repeat.

(Quick myth-correction, since it costs one sentence: the famous bell is mostly folklore. Pavlov's lab notes lean heavily on metronomes, buzzers, and tuning forks. The bell survived because it's easier to draw.)

After enough pairings, the signal alone made the dogs salivate. Nothing about the food had changed. What changed was that the dog had learned the signal carries information about the future.

This is the second piece, and it's a big one. Thorndike's cat learned what to do. Pavlov's dog learned what's coming.

In RL terms, that's value. The value of a state is roughly: how much good stuff do I expect from here on out? A metronome that reliably precedes food is a high-value state — not because a metronome is nutritious, but because of what tends to follow it.

You do this constantly. The smell of coffee is not caffeine. Your inbox badge is not the email. A green light is not arriving anywhere. Every one of these is a state you've assigned value to, purely because of what historically came next.

And this is why RL agents can learn anything at all in sparse environments. A chess engine that only felt reward at checkmate would learn approximately nothing — the signal is 60 moves away from most of the decisions that caused it. What it actually learns is to value positions: this pawn structure tends to precede winning. That's a metronome.

🎒 In the backpack: states have value. Value comes from prediction, not from the reward itself.

But something's still missing

Pavlov's dogs were passive. They sat in a harness and predicted. The dog's salivation doesn't make the food arrive any sooner.

So now we have two halves of an agent and no glue:

	Thorndike's cat	Pavlov's dog
Learns	which actions pay off	which signals predict payoff
Can it act?	✅	❌
Can it anticipate?	❌	✅

A real agent needs both — and it needs someone to make this a controllable science rather than a collection of observations. That took a genuine control freak.

The 1930s–50s — Skinner Turns Behavior Into Engineering

B.F. Skinner's question was different from everyone before him. Not "how do animals learn?" but: "how precisely can I control what an animal learns?"

His tool was the operant conditioning chamber — the Skinner Box. A rat inside can press a bar. A pigeon can peck a disc. Food is dispensed on rules Skinner sets. Simple inputs, measurable outputs, and — critically — the animal stays in the box.

That last detail seems minor and isn't. Thorndike's cats escaped, which meant one trial, then a reset by hand. Skinner's animals could run thousands of trials in an afternoon, uninterrupted. He'd built the first high-throughput training environment. Every RL researcher who has ever run a simulator overnight is doing the same thing.

Two of his findings matter for us.

Finding 1: Unpredictable rewards are the stickiest

Skinner varied when reward arrived — every press, every fifth press, every fifth press on average. These are schedules of reinforcement.

The counterintuitive result: variable reward — where the animal can't predict which press pays — produced behavior that was hardest to extinguish. Reward a pigeon every single time, then stop, and it gives up quickly. Reward it unpredictably, then stop, and it keeps going for a remarkably long time.

Slot machines run on this. So do infinite-scroll feeds. Skinner found the exploit in the 1950s; the attention economy just productized it.

For us, the useful version is that reward structure is a design decision with enormous consequences. Which sets up an entire genre of modern RL failure — agents that learn the reward signal instead of the task, in ways nobody intended. We'll spend real time there in a later post.

Finding 2: You can build complex behavior out of crumbs

The bigger idea is shaping.

Skinner taught pigeons to play a passable game of table tennis. Not by waiting for a pigeon to spontaneously play table tennis — you'd wait forever. He rewarded incremental steps. First, reward the bird for merely facing the table. Once that's reliable, stop rewarding it and only pay for approaching the ball. Then only for contact. Then only for direction.

https://www.youtube.com/shorts/RPJyKSmH9WM?feature=share

You are moving the goalposts on purpose, in small enough increments that the animal never falls off the curve.

This is reward shaping, and it's how we train robots today. A robot arm rewarded only for "successfully assembled the part" will flail randomly forever, because the odds of stumbling into success by accident are effectively zero. So we hand it crumbs: reward for moving toward the part, then for gripping it, then for lifting. Same ladder. Different species.

And the strategy the animal ends up with — in this situation, do this — is exactly what we called a policy at the top of this post. Skinner spent thirty years training policies. He just did it with pigeons and a food dispenser instead of gradients and a GPU.

🎒 In the backpack: policies can be trained, not just observed. Reward timing and reward structure change everything.

But something's still missing

We now have a full behavioral picture: act, predict, get shaped, improve.

Every bit of it is described from the outside. Skinner was deliberate about this — he treated the mind as a black box and argued you should only ever talk about observable behavior. Which is scientifically disciplined and completely useless if you want to build one.

Because if you're going to build this thing, you need to answer a question Skinner refused to ask: where does the learning physically live?

1949 — Hebb Finds the Storage

Donald Hebb, a Canadian psychologist, published The Organization of Behavior in 1949 and proposed a mechanism so simple it fits in a sentence.

If neuron A repeatedly takes part in firing neuron B, some growth process or metabolic change takes place in both cells such that A's efficiency in firing B is increased.

That's Hebb's Rule, and the pop version is stickier: neurons that fire together, wire together.

Two neurons active at the same time? The connection between them — the synapse — physically strengthens. Next time, A triggers B more easily. Do it enough and the path becomes a highway.

Run Thorndike's cat through this. The cat sees the lever (a pattern of neurons firing) and presses it (another pattern firing). They're active together. Their connection thickens. A hundred trials later, seeing the lever is pressing the lever — there's a paved road between them.

That's a habit, described in hardware.

And this is where artificial neural networks come from. In a network, the connection strength between two artificial neurons is a number called a weight. Learning means adjusting weights so that useful inputs reliably produce useful outputs. Hebb is the reason "learning = changing connection strengths" is the default assumption in the entire field.

One important correction, because this gets flattened a lot: Hebb's rule is not backpropagation, and the difference is worth thirty seconds.

Hebb's rule is local. A synapse strengthens based only on what its own two neurons are doing right now. No supervisor, no error signal, nobody keeping score.

Backpropagation — the algorithm that actually trains modern networks — is global and error-driven. It compares the network's output to a desired output, computes how wrong it was, and pushes that error signal backward through every layer, adjusting each weight by how much it contributed to the mistake. Every weight update depends on information from far away in the network.

These are genuinely different mechanisms. In fact, the gap between them is one of the liveliest open arguments in neuroscience: brains clearly don't run backprop — there's no known biological machinery for shipping a precise error signal backward across a network — yet backprop works spectacularly well. Whether the brain approximates something like it, or does something else entirely, is unresolved.

So the honest lineage is: Hebb gave us the idea that memory lives in connection strengths. Backprop is a much later, mathematically-driven answer to how to update those strengths. Same neighborhood, different building.

🎒 In the backpack: learning is stored in connection weights. Local, physical, incremental.

Where that leaves us in 1949

Fifty-one years, four researchers, and the loop is complete:

Idea	From	What it becomes
Rewards strengthen the actions that earned them	Thorndike, 1898	the reward signal
Signals predict future reward	Pavlov, 1900s	value
Behavior can be sculpted step by step	Skinner, 1930s–50s	policies, reward shaping
Learning is stored in connection strengths	Hebb, 1949	weights

Read that table top to bottom and you have a specification. Something that acts, predicts, gets shaped by consequences, and stores what it learned in the strength of its connections.

Nobody had built it. Nobody could — the ideas were all written in the language of biology, describing wet tissue that nobody could manufacture.

But 1949 is the year that stops being true. The war had just produced digital computers. A new field called cybernetics was arguing that control and feedback worked the same way in animals and machines, and that the distinction was less fundamental than everyone assumed.

For the first time, people had a spec for learning and a machine to run it on.

One thing they all got wrong

Before we leave the psychologists, here's the crack in the foundation — worth holding onto, because it takes another twenty years to fix.

Every idea above says learning happens when things co-occur. Metronome and food together → learn. Lever and fish together → learn. Neurons active together → wire together.

But co-occurrence can't be the whole story:

Your alarm has gone off at 7am every day for a year. By the co-occurrence rule, this morning should have strengthened that association a little further, same as every other morning. It didn't. You learned it in the first week and then stopped — even though the pairing kept happening, perfectly, hundreds more times.

So what switches learning off?

Nobody in this post can answer that. The answer arrives in 1972, from a pair of psychologists still studying rats, and when it does it turns out to be the single most important equation in this entire series. We'll get there.

Next up

RL #2 — Cybernetics and Early Machines (1948–1954). The first people to actually build this thing. Alan Turing sketching a "pleasure-pain system" in 1948. Claude Shannon's Theseus, a mechanical mouse that learned a maze by trial and error and could remember it. And Marvin Minsky wiring together the SNARC — 40 vacuum tubes, motors, and a clutch, arguably the first machine that learned by reinforcement.

The cats gave us the algorithm. Now watch people try to solder it.

If you spot an error or think I've oversimplified something, drop it in the comments — I'd rather fix it than have it sit there.

Reinforcement Learning Series

Mitansh Gor — Wed, 19 Aug 2026 18:40:39 +0000

I created this series to make that steep learning curve far less daunting for you. My goal is to trace the evolution of RL chronologically—from its roots in early psychology and physical mechanical machines to digital binary systems and modern mathematical breakthroughs. By breaking down complex concepts with clear visual guides, graphics, and real-world analogies, I hope to demystify RL and give back to the community that inspired me. Let’s dive in!

Blog 1: The Psychological Seeds (1898–1949)

This entry explores the Law of Effect (1911), where Edward Thorndike established that actions followed by satisfaction are strengthened. It also covers Ivan Pavlov’s formal definition of reinforcement (1927) and Donald Hebb’s 1949 hypothesis that "neurons that fire together, wire together," laying the groundwork for neural learning.

Blog 2: Cybernetics and Early Machines (1948–1954)

This blog details the first computational investigations, including Alan Turing’s "pleasure-pain system" (1948) and Marvin Minsky’s construction of SNARCs (1954), the first analog neural-network reinforcement calculators. It also highlights Claude Shannon’s 1952 demonstration of "Theseus," a maze-running mouse that used trial and error to "remember" paths.

Blog 3: The Mathematical Framework (1950s–1960s)

This post focuses on Richard Bellman’s development of Dynamic Programming and the Bellman Equation (1957), which provided the formal tools for solving optimal control problems. It also introduces Markov Decision Processes (MDPs) and Ron Howard’s policy iteration method (1960), which are the bedrock of modern RL theory.

Blog 4: Early Heuristics and Checkers (1959–1968)

This entry discusses Arthur Samuel’s celebrated checkers program (1959), which was the first to implement temporal-difference (TD) ideas by using board evaluation differences to modify earlier positions. It also details Donald Michie’s MENACE (1961), a physical "matchbox" engine that learned to play Noughts and Crosses through reinforcement.

Blog 5: The Birth of Learning Automata (1961–1974)

This blog examines the Soviet lineage of RL, specifically Mikhail Tsetlin’s work on "learning automata" (1961), which were finite-state machines that updated action probabilities based on stochastic environmental feedback. It traces how this research was later systematised by Narendra and Thathachar in 1974.

Blog 6: The "Hedonistic" Revival (1972–1980)

This post covers Harry Klopf’s pivotal role in reviving the trial-and-error thread by proposing the "hedonistic neuron" (1972), which argued that neurons act as pleasure-seeking units. It also explores Paul Werbos’s 1974 thesis, which was the first to describe backpropagation in the context of "Adaptive Dynamic Programming".

Blog 7: The Sutton and Barto Foundation (1981–1984)

This blog highlights the formalisation of modern RL, beginning with Sutton and Barto’s 1981 model of classical conditioning. It details the invention of the Actor-Critic architecture (1983) and Sutton’s 1984 dissertation, which provided the first rigorous computational analysis of temporal credit assignment.

Blog 8: The Q-Learning Breakthrough (1988–1994)

This entry documents two watershed moments: Sutton’s 1988 formalisation of TD learning and Chris Watkins’s 1989 introduction of Q-Learning, the first model-free, off-policy algorithm. It also covers the introduction of SARSA (1994) as an on-policy alternative that prioritises training stability.

Blog 9: Superhuman Achievement in Games (1992–1999)

This post details Gerald Tesauro’s TD-Gammon (1992), which achieved grandmaster-level backgammon play using neural networks and self-play. It also covers the options framework (1999) by Sutton, Precup, and Singh, which introduced temporal abstraction to help RL agents solve problems across different timescales.

Blog 10: Theoretical Refinement (2000–2010)

This entry explains the Policy Gradient Theorem (2000), which proved that policy parameters could be optimised using only local information. It also introduces Sham Kakade’s Natural Policy Gradient (2002), a precursor to modern trust-region methods that account for the geometry of parameter space.

Blog 11: The Deep RL Revolution (2013–2015)

This post chronicles the birth of Deep RL with DeepMind’s DQN (2013/2015), which used Experience Replay and Target Networks to master Atari games directly from pixels. It also details DDPG (2015), which extended these successes to continuous action spaces like robotics.

Blog 12: Mastering Strategy and Control (2015–2017)

This entry explores Trust Region Policy Optimization (TRPO) (2015) and its simpler successor, Proximal Policy Optimization (PPO) (2017), which became the industry standard for stable training. It also covers the historic victory of AlphaGo over Lee Sedol in 2016.

Blog 13: Integrated Mastery: Rainbow and MuZero (2017–2019)

This post details the Rainbow agent (2017), which combined seven independent DQN improvements into one "super-agent". It also highlights MuZero (2019), which learned to master games like Go and Chess without even being told the rules of the environment.

Blog 14: Hard Exploration and Generalisation (2020–2022)

This blog focuses on Agent57 (2020), the first agent to surpass the human benchmark on all 57 Atari games by solving "hard exploration" problems. It also covers Gato (2022), a single transformer-based generalist agent capable of playing Atari, captioning images, and controlling robot arms.

Blog 15: The Transformer Pivot and LLM Alignment (2021–2023)

This entry describes the Decision Transformer (2021), which reframed RL as a sequence modeling problem, bypassing traditional bootstrapping. It also covers the rise of Reinforcement Learning from Human Feedback (RLHF) with InstructGPT and the subsequent development of Direct Preference Optimization (DPO).

Blog 16: Reasoning Models and Incentivized Thought (2024–2025)

This blog explores DeepSeek Math and GRPO (2024), which removed the need for a separate critic model to improve mathematical reasoning. It details the breakthrough of DeepSeek-R1 (2024/2025), which proved that pure RL can spontaneously elicit self-correction and reflection in language models.

Blog 17: Massive Depth and the Future of Scaling (2025–2026)

This final entry discusses the 2025 NeurIPS Best Paper on 1,000-layer RL networks, which broke the "shallow network" assumption to unlock 50x performance gains. It reflects on Kimi k1.5 and the role of the 2024 Turing Award in cementing RL as a foundational pillar of the AI revolution.

Personal Health Agent (PHA): Multi-Agent Health System

Mitansh Gor — Thu, 27 Nov 2025 05:33:43 +0000

VIDEO LINK

In the last few years, we saw different types of AI systems try to answer this question:
PH-LLM (2024) → focused on personalized coaching from wearable data.
PHIA (2024) → acted like an agent that analyzes your data, writes code, and explains results.
IR Prediction + IR Explainer Agent (2025) → estimated your metabolic risk and explained it like a doctor.

All of these systems were important steps. But each one had a weakness:

A single LLM tries to do everything.
It must read data, interpret medical meaning, check safety, and give advice.
This often leads to confident hallucinations — answers that sound correct but are actually wrong.
It also mixes medical reasoning with casual coaching, which creates risk. And one model cannot be an expert in every health domain.

So the big idea behind the Personal Health Agent (PHA) is simple:

Instead of one giant model doing everything, why not create multiple smaller agents — each with a role — working together like a mini health team?
This solves the main problem with earlier systems: no single AI can fully handle the complexity of real human health. But a team of specialized agents can.

Because health is multi-dimensional—where your sleep affects your stress, stress affects metabolic health, exercise influences sleep, and food and biomarkers impact everything—one AI model can’t realistically handle all of it at once. Just like a hospital relies on different specialists, a digital health system also needs different “roles”: one agent to interpret raw data, one agent to check medical safety, and one agent to translate insights into simple coaching. This is why multi-agent systems are becoming so important: they add structure, accountability, and more reliable reasoning, solving many of the issues earlier single-LLM systems struggled with.

A System of Specialized Agents

When I look at the PHA system, what stands out first is the Analyst Agent. I think of this agent like a friend who loves numbers and graphs. Its whole job is to read my raw data — sleep hours, steps, heart rate, and other signals — and turn them into clear observations. It doesn’t try to coach me or act like a doctor. It just says, “Here’s what I see happening in your data,” which makes everything feel clean and easy to understand.

Then there is the Domain Expert Agent, which to me is the “doctor brain” of the whole system. This agent takes what the Analyst found and checks if it makes medical sense. It worries about safety, accuracy, and whether the explanation aligns with real health science. I really like this part because it stops the system from giving random or misleading advice. It acts like the strict friend who always says, “Wait, is this actually correct?”

Next comes the Coach Agent, the most human-feeling part of PHA. This agent talks to me like a supportive friend who wants me to succeed. It takes all the technical pieces and translates them into simple, everyday advice I can act on. No medical jargon. No complicated charts. Just small, doable suggestions like, “Try walking after dinner” or “Aim for an earlier bedtime tonight.” It’s friendly and practical, which is exactly what people need.

What ties everything together is how these agents work as a team. Instead of one giant AI trying to do everything — and often making confident mistakes — each agent stays in its own lane. One analyzes, one verifies, one coaches. Because of this teamwork, the final answer feels clearer, safer, and much more trustworthy. It’s almost like having a mini health team inside your phone, and that makes the whole system feel both smart and human at the same time.

Technical Deep Dive

Even though all agents may start from the same base LLM (like Gemini or GPT-class models), they are trained separately, so each one becomes good at one thing instead of being average at everything.

Finetuning

Analyst Agent is finetuned from wearable logs (heart rate, steps, sleep), time-series patterns, data summaries, and statistical examples. It becomes good at reading numbers and spotting patterns.
Domain Expert needs more serious data for finetuning, like medical guidelines, curated clinical examples, safe/unsafe reasoning samples, and explanations validated by experts. It learns how to avoid hallucinations and unsafe claims
Coach Agent is the “human” one. It is finetuned on a friendly conversational dataset, lifestyle advice examples, behavior-change coaching patterns. It learns to talk in a simple, supportive way.

This division is what makes the final system feel more stable than earlier systems like PHIA or the IR Agent, where a single LLM had to juggle all roles and often made confident mistakes.

The process itself was simple but very intentional. Here’s how I imagine it:

We first collected examples for each agent’s role.
Then we cleaned and organized everything.
Then we cleaned and organized everything.
After that, we fine-tuned each agent separately.
Then we tested the agents together.
Finally, we fixed mistakes and improved the dataset.

System architecture and Data Flow

Underneath all of this, there’s a system architecture that makes their collaboration possible. I picture it like a small command center. The user’s data flows into a main controller, which decides which agent speaks first and who validates whom. The Analyst generates its interpretation, the Domain Expert checks it, and the Coach turns everything into a simple message. Each agent is isolated enough to stay focused, but interconnected enough to build on each other’s work. There’s also a safety layer running quietly in the background — something like a “medical guardrail” — to make sure the final output stays safe, consistent, and responsible before it reaches the user.

Data Pipeline Flow

My raw data comes in (wearable data, blood test results, communication history)
The system cleans and prepares the data.
The Analyst Agent gets the cleaned data first, and it gives a report.
The Expert Agent double-checks everything and makes sure the explanation is medically correct and safe.
The Coach Agent gets the approved explanation and rewrites the message in simple, friendly language
A final safety filter reviews the message and checks for risky things like any medical diagnosis, unsafe suggestions, and harmful claims.
I receive the final output.

Inference time

During inference — which is just a fancy word for “when the system actually responds to me” — the agents talk to each other almost like coworkers passing notes in a group chat. I don’t see any of this, but it’s happening behind the scenes. The Analyst looks at my raw data and says, “Here’s what I think is happening.” The Domain Expert looks at what the Analyst wrote and either approves it or corrects it. And finally, the Coach turns their combined reasoning into something that sounds human and helpful. What I receive as a single answer is actually the teamwork of multiple roles stitched together.

Tools used

To make all of this work, the system relies on a set of tools that help each agent extend its abilities. The Analyst might use data-processing tools to clean and summarize wearable data. The Domain Expert might have access to reference tools that help it compare findings against medical knowledge or validated guidelines. The Coach might use prompt templates designed for empathy, clarity, and motivation. And the main controller uses orchestration tools — something like LangGraph or a custom workflow engine — to decide who talks when and how information flows between them. These tools don’t replace the agents, but they give them structure, context, and external capabilities they wouldn’t have on their own.

Collection of data

Everything starts with a real study they ran called WEAR-ME, which included more than a thousand Fitbit users. People had to explicitly opt-in, sign digital consent, link their wearable device, and even go for a blood draw at a Quest Diagnostics center. So the data wasn’t scraped or assumed—it came from real people who agreed to share it for research. That part mattered to me because it made the whole system feel responsible, not rushed.

Once someone joined the study, their data flowed through a very controlled pipeline.So each person had a mix of: (1) wearable time-series, (2) lab biomarkers, (3) self-reported history, (4) lifestyle questionnaires.

What I liked is that the system doesn’t directly reach into a live Fitbit account or do anything “real-time.” Instead, the PHA gets access to summaries of the participant’s data, organized into tables—like a daily summary table, an activities table, and even a population-level reference table so the system could compare one user’s values against others in their age group. This gave context, not just raw numbers.

Inside the actual agent pipeline, each sub-agent only receives the part of the data that’s relevant to its job. The system never dumps the full raw dataset into every agent. Instead, the orchestrator acts like a coordinator who hands the right information to the right specialist.

Nothing is fetched unless the user asks something that genuinely needs it. And the system also avoids asking the user for things it already knows—like their sleep hours or steps—because the orchestrator checks the available data first.

PHA vs Gemini Results

PHA wasn’t just a small improvement. It was in a completely different league.

From the end-user side, I noticed that people consistently picked PHA as the best experience. Not just once or twice — almost half the time. And this is after comparing it side-by-side against a single Gemini agent and a parallel multi-agent system.

But the biggest eye-opener for me was the expert evaluation. Experts didn’t just prefer PHA — they almost abandoned the Gemini single-agent baseline. Only around 4–5% of expert rankings put Gemini in first place, while PHA was chosen as #1 about 80% of the time. That’s not a small margin.

Strengths of the system

What struck me most was how real the system felt. Not just in terms of architecture, but in how thoroughly it was tested. It wasn’t a small demo — it involved thousands of human annotations and hundreds of hours of expert evaluation. Because of that, the strengths and limitations felt very honest.

One of the biggest strengths I noticed is how well the whole system comes together. The three-agent setup — the data scientist, the domain expert, and the coach — isn’t just theoretical. In the evaluations, PHA actually outperformed both a single-agent system and a parallel multi-agent system. End-users preferred talking to PHA almost half the time, and experts loved it even more, choosing it as the best system in about 80% of cases. To me, that says the collaboration between these agents isn’t just helpful — it genuinely changes the quality of the conversation.

The data agent in particular felt like a major upgrade. It was much better at breaking down messy, vague health questions into proper statistical analyses, catching missing data, choosing the right timeframes, and generating correct code. I could see why the system became more trustworthy — it wasn’t guessing. It was reasoning over the user’s actual data. And the domain expert agent made a big difference too. It produced safer, more accurate, and more personalized medical explanations. Users rated its responses as far more trustworthy than the base model, and clinicians preferred its summaries because they were more complete and clinically meaningful.

The health coach agent also impressed me. It wasn’t just delivering motivational lines. It followed real coaching principles — active listening, SMART goals, motivational interviewing — and this made conversations feel more natural and supportive. Users were more engaged, and the conversations ended more naturally. It reminded me of talking to a real human coach who’s actually paying attention.

As a whole system, PHA feels more thoughtful. The orchestrator doesn’t just mix answers together — it understands the user’s goal, assigns the right agent, reflects on the output, and remembers important details for later turns. This gives the conversation a sense of direction and personalization that the baselines just didn’t have.

Limitations of the System

But the system isn’t perfect. One thing that stood out immediately was the cost. PHA is slower and more computationally expensive because each request triggers multiple agents and reflection steps. Where the single-agent system might respond in around 35 seconds, PHA could take over 200. It’s powerful, but not cheap.

Another limitation is that the diagnostic reasoning, while improved, is still not bulletproof. Sometimes the domain expert agent didn’t complete its reasoning chain, and its reliance on web search occasionally pulled in conflicting information. This reminded me that, even though it acts like a doctor in some ways, it’s not actually one — and the authors make that very clear.

Bias is also a real concern. The system personalizes advice based on user traits, conditions, and context… which is great, but it also means it can unintentionally repeat patterns or assumptions from datasets that might not represent everyone equally. The paper calls out this risk directly.

The coach agent, despite being strong overall, also had a weakness: it wasn’t great at tracking user progress over time. It did a good job helping set goals, but didn’t always follow up on them in later turns. For long-term coaching, that’s something they’ll need to improve.

And finally — maybe the biggest limitation — everything was tested in short-term interactions. We don’t yet know whether PHA can support behavior change over weeks or months, which is where health coaching really matters. The team also notes that this is a research framework, not a clinical tool, and major regulatory work would be needed before deploying anything like this in the real world.

So when I put it all together, here’s how I personally see it:
PHA is a huge improvement over past systems — smarter, safer, more human, and more useful — but it’s still early. It has real strengths, but also real gaps. It’s powerful as an idea, promising as a system, and clearly built with care. But it also reminds me that great AI doesn’t replace medical care — it supports it, and it still has a lot of growing up to do.

Conclusion

After reading the PHA paper, the biggest thing I realized is that this system isn’t just an AI answering health questions — it feels more like a small team working behind the scenes to help you make sense of your own data. The evaluations show that this multi-agent design really does lead to better, clearer, and more personalized guidance compared to older single-agent approaches.

At the same time, the authors are very honest about their limits. The system can still make mistakes, it can be biased depending on the data, and it should never replace real medical care. It’s powerful, but it’s also early — more like a research blueprint than a ready product.

To me, PHA represents a hopeful direction: AI that supports people in understanding their health, not by taking over, but by helping them make smarter choices with empathy, safety, and transparency in mind

PHIA: The Agentic LLM That Writes Code, Analyzes Your Data & Explains Health Insights

Mitansh Gor — Thu, 27 Nov 2025 05:33:34 +0000

VIDEO LINK

I honestly felt like I was looking at the “next version” of what health AI should be. Whenever I asked myself questions like, “Is my sleep improving?” or “Does exercising at night change my deep sleep?”, I realized how hard it is to answer these without real analysis.

That is exactly the problem the PHIA team talks about on page 1 of the paper. They explain that even a simple question like “Do I sleep better after exercising?” requires many steps: checking recent data, comparing different days, calculating metrics, and then interpreting everything in the context of what “healthy” even means. And honestly, I could relate — that’s the kind of analysis I never do on my own.
What really caught my attention is that the paper says today’s LLMs struggle with numerical reasoning, meaning they often miscalculate or oversimplify things. I’ve definitely seen models do that — giving a confident answer but completely messing up basic math. (Limitation of PH-LLM)

So PHIA tries to fix this problem by giving an LLM the ability to plan, use tools, write code, and search the web. It doesn’t just “chat.” It becomes more like a small data analyst that works with your wearable data step-by-step.

PH-LLM vs PHIA

PH-LLM was an earlier coaching system that worked only with simple, pre-aggregated 30-day summaries and relied purely on the LLM’s internal reasoning. That meant it couldn’t actually analyze detailed daily wearable data, couldn’t run calculations, and completely failed on questions that required numbers or step-by-step reasoning. PHIA addresses these limitations by granting the LLM three new capabilities: (1) it can generate authentic Python code to analyze raw wearable data, (2) it can plan and break down tasks using an agent loop, and it can search the web to incorporate fresh, verified health knowledge.

Because of this, PHIA shifts from being “just a health coach” to becoming a true data analyst + coach, delivering insights that are more accurate, personalized, and grounded in both the user’s data and real health science. PHIA feels like the bridge between “AI as a friendly coach” and “AI as a personal data scientist.” By mixing reasoning, code execution, and web search, it finally unlocks the kind of deeper, more accurate insights that wearables have always had the potential to provide.

How PHIA Works

PHIA more like a mini health data analyst that can think step-by-step, write code, check your wearable data, search the web, and then explain everything in normal language. The PHIA paper shows this clearly — PHIA literally cycles through think → act → observe just like a human analyst would.

I’ll explain this in the simple way I understood it.

Instead of guessing answers, PHIA acts more like a small data analyst that plans its steps, checks your wearable data, and only then explains what it found. The paper describes this using the ReAct loop— Thought → Action → Observation — and seeing it in action helped me understand why PHIA is so different.

PHIA starts by thinking through the question. If I ask something like “Is my resting heart rate improving?”, it doesn’t jump to a quick reply. It pauses and decides what it needs—maybe comparing two weeks of data or calculating averages. Then, PHIA takes action by writing and running Python code in a safe virtual environment tool. This lets it analyze real daily time-series data using Pandas, just like a real data scientist. The nice part is that this removes mathematical mistakes that normal LLMs often make. And if its code crashes, PHIA actually fixes the error and tries again (recurrent trials system), which the paper highlights as one of its strengths.

In their evaluation, PHIA scored 84% accuracy on objective questions, way higher than simple LLM reasoning. This happens because PHIA can recover when the code fails.

When a question requires more than numbers—like understanding recommended sleep hours—PHIA uses a built-in web search tool to fetch verified information from trusted sources. Sometimes the user needs context, like: “Is this amount of sleep normal for my age?” “What workouts improve resting heart rate?” “Is my stress score healthy?”
PHIA has a built-in search tool that pulls information from verified online sources. This mixing of data + domain knowledge is what makes PHIA feel smart and practical.

What impressed me most is how PHIA blends everything together. It doesn’t just give numbers or copy facts. It calculates, it checks, it researches, and then it explains the result in clear language. The examples in the paper, like comparing a user’s sleep with national guidelines, really show how these steps come together. In the end, PHIA feels less like a chatbot and more like someone doing careful, step-by-step reasoning to help you understand your own health patterns.

A Deep Dive Into PHIA’s Technical Architecture

PHIA is not a fine-tuned model. It’s an agent framework built around Gemini 1.0 Ultra, with two major tools: (1) Python data analysis runtime, (2) Web search tool.

These tools are orchestrated using the ReAct (Reason + Act) agent pattern, which is why PHIA can do multi-step reasoning without fine-tuning. This design gives PHIA abilities that raw LLMs don’t have: planning, correction, stepwise attention, and grounded analysis.

No Fintuning

One of the most interesting things I learned from the paper is that PHIA is not fine-tuned like PH-LLM. Instead of modifying the model’s weights, the authors built a smart scaffolding around the model and taught it how to act like an agent.

Fine-tuning a huge LLM like Gemini Ultra is expensive, slow, requires tons of supervised data, and risky (can break general reasoning ability).

So instead, PHIA keeps the base model untouched and teaches the model how it should “think” and use tools using few-shot ReAct examples.

This is a totally different philosophy: PH-LLM → change the model. PHIA → change the system around the model

Process

The authors started with thousands of wearable-related questions (objective + open-ended). They needed a tiny set of example tasks that could teach the model how to behave like an agent.
Instead of randomly picking examples, they converted every question into numerical embeddings using Sentence-T5. This turns each question into a vector representing its meaning, so similar questions sit close together in vector space.

They ran k-means clustering (k=20) on these embeddings to automatically group similar questions into 20 clusters. Each cluster represents a whole “type” of question, like sleep trends, workout comparisons, anomaly detection, correlations, and so on.
From each cluster, they selected the most central question — the one closest to the cluster centroid.This gives 20 representative queries, each standing in for a larger family of similar queries.
For each of these 20 representative queries, the team manually wrote a complete ReAct agent trajectory that included every step of reasoning:
- Thought: what the model should plan to do
- Action (Python): the exact Pandas code needed
- Observation: the real output that the code would return
- Thought again: interpreting the output
- Action (Search): when domain knowledge is needed
- Observation: the retrieved information
- Final Answer: a clear, natural-language explanation. These aren’t partial examples — they are full step-by-step walkthroughs that demonstrate how an agent should behave.
These 20 full trajectories were inserted into PHIA’s few-shot prompt, acting as demonstrations inside its system instructions. So the model is not fine-tuned — instead, it is taught by example how to: plan before acting, decide when to run code, decide when to search the web, fix code errors, combine numerical results with domain knowledge, and produce a final personalized insight.

This is called “behavior cloning via prompting.” : It will learn the “pattern of behavior” from the examples, even without changing its internal parameters.

PHIA is a perfect example of how powerful LLMs become when they are paired with the right scaffolding instead of relying only on internal reasoning.

Limitations in the PHIA

PHIA is a huge step forward, especially compared to simple chat-based coaching. But it’s still early-stage. It shows what’s possible when LLMs use tools, but it also exposes how much work is left to make an AI truly trustworthy in personal health.

PHIA hasn’t been tested on real people yet, so we still don’t know if it truly helps users change habits or understand their health better in everyday life.
It only uses wearable data, and I feel it needs more context—like nutrition, medical records, or mood logs—to give deeper and more holistic insights.
PHIA isn’t medically validated, so even though it sounds smart, its recommendations haven’t been checked by doctors or health experts for real accuracy. Personalization still feels limited, because sometimes PHIA gives generic advice even when the user’s data is available right there.
Its reasoning depends heavily on prompting and Gemini Ultra, which means its behavior might change across models or updates, making it less consistent.
Error handling is better than basic LLMs but still imperfect, since PHIA can misread data columns or fail on messy real-world data.
Its toolset is still narrow, and I’d love to see it generate visualizations, analyze other health signals, or track long-term goals instead of just answering one question at a time.

Learnings from the paper

Reading the PHIA paper honestly changed the way I think about building AI systems. Before this, I used to focus mainly on making the model smarter, but PHIA showed me that the real progress comes from giving the model the right tools and the right structure. Seeing how PHIA uses the ReAct loop—think, act, observe—made me realize how important disciplined reasoning is for avoiding hallucinations and building trust.

The use of synthetic data was another eye-opener, because it proved that we can train and evaluate personal-health agents responsibly without touching real user data. On top of that, the way the authors handled safety—650 hours of human review, strict guardrails, and cautious refusal behavior—reminded me that responsible AI isn’t optional; it’s part of the engineering.

More than anything, PHIA taught me that the future of AI isn’t “a bigger model,” but a system where models, tools, data, and reasoning frameworks all work together. As an AI engineer, this shifted how I think about agents: not as chatbots, but as full pipelines designed to solve real problems with accuracy, humility, and safety.

Key Insights I’m Taking With Me

PHIA wins because it’s structured, not because the LLM is smarter.
Reasoning and architecture beat raw model size.
PHIA shines when a question needs data + external knowledge + multi-step logic, not just basic statistics.
Its real strength is disciplined reasoning, not raw computation.
PHIA’s strict safety guardrails make it trustworthy in a way normal LLMs aren’t.

PH-LLM - A LLM that gives personalized sleep and fitness coaching using wearable data.

Mitansh Gor — Thu, 27 Nov 2025 05:32:57 +0000

VIDEO LINK

PH-LLM (Personal Health Large Language Model) is a version of Gemini Ultra that has been fine-tuned specifically for sleep and fitness coaching. It isn’t just chatting—it actually learns from up to 30 days of wearable data, understands your patterns, and gives expert-level insights. The research paper shows PH-LLM scoring 79% on sleep medicine exams and 88% on fitness exams, which is on par or better than the expert groups they tested with.

What really caught my interest is that the system can:

analyze daily-resolution sensor data
generate personalized insights
Predict how rested or tired you will feel based only on wearable data and speak to you like a sleep coach or fitness expert.

What Problem PH-LLM Tries to Solve

Wearables track you, but they don’t talk to you. They tell you what happened, but not why it happened or how to fix it. PH-LLM aims to act like a personal health coach — one that understands your data, your patterns, your goals, and the science behind all of it.

Instead of treating sensor data like random numbers on a dashboard, it tries to interpret them the way a human expert would. If your bedtime has been drifting later over the past two weeks, or your deep sleep dropped right after you increased workout intensity, PH-LLM doesn’t just point it out — it explains why it happened and how it connects to your goals. It transforms passive metrics into an actual conversation about your habits.

The system was trained specifically for this kind of reasoning. The researchers fine-tuned Gemini Ultra so it could combine textbook knowledge with real-world wearable data. Interestingly, PH-LLM doesn’t just “sound smart.” In testing, it actually outperformed sleep experts on sleep medicine exam questions and matched them in fitness knowledge. That means the model isn’t just giving generic advice — it has a near-expert understanding of the underlying science.

But what really makes PH-LLM different is that it doesn’t stop at general knowledge. It looks at a person’s actual patterns over weeks. If you always sleep well on weekdays but crash on weekends, it's noticeable. If your HRV is dropping while your workouts are getting harder, it connects the dots. In one of the paper’s examples, the model realized a user had a very regular sleep schedule but consistently slept too little, and it suggested shifting bedtime by small increments rather than giving a one-size-fits-all rule. That’s the kind of nuance real coaches provide, but most apps don’t.

High-level Working of the PH-LLM

Researchers took Gemini Ultra — a very capable general-purpose model — and taught it how to understand sleep and fitness the same way a human expert would. They basically turned a large language model into a personal health specialist.

The process happened in two major steps. First, they fine-tuned the entire Gemini Ultra model on hundreds of detailed sleep and fitness case studies. These weren’t made-up examples — each case study was based on real wearable data from real people. The case studies included up to thirty days of information such as bedtimes, wake times, restlessness, workout intensity, heart rate metrics, and more. Alongside the data, sleep physicians and athletic trainers wrote expert-level insights, possible causes, and recommendations. These human-written explanations became the “teacher examples” that the model learned from. By imitating these expert responses over and over, PH-LLM learned how to talk like a coach and reason like one too.

After this, the researchers added a second layer of training using something called a multimodal adapter. This part fascinated me because it let the model go beyond simple text input. Instead of only reading written summaries of the user’s data, PH-LLM also receives a compressed representation of the actual sensor values — the statistical patterns hidden in daily heart rate, HRV, sleep duration, respiratory rate, and other signals across fifteen days. These adapter-generated “soft tokens” get injected directly into the LLM’s internal understanding. In simple terms, the model doesn’t just read your data — it absorbs it. This is how PH-LLM can predict things like “Will this person feel tired tomorrow?” with accuracy on par with traditional machine learning approaches.

The result of these two training phases is a model that can look at your daily metrics and instantly form a holistic picture. If your bedtime keeps drifting later, PH-LLM notices. If your deep sleep drops right after your workout intensity spikes, it picks up the relationship. If your HRV has been declining all week, it connects that to stress, recovery, or the need for rest. What impressed me is that PH-LLM isn’t just matching patterns — it can articulate the reasoning behind them, almost like an expert thinking out loud.

The paper shows a great example of this. For one user, the model pointed out that the midsleep point was extremely consistent — meaning their circadian rhythm was stable — but their total sleep time was consistently too low. From there, PH-LLM suggested a gradual shift in bedtime over several days. This wasn’t a generic “try to sleep more” tip; it was a specific plan tailored to what the data actually showed. That kind of reasoning is the whole point of the system.

Another thing I appreciated is that PH-LLM adapts its responses based on how much data it has. When the researchers removed parts of the input — like today’s sleep metrics or the last week of workout logs — PH-LLM still adjusted its explanations in a sensible way. To me, this shows that the model doesn’t rely on memorized patterns but actually understands the structure of sleep and fitness behavior.

Technical Working

The foundation of PH-LLM is Gemini Ultra 1.0, Google’s flagship multimodal model. In this work, it functions mainly as a text LLM (the vision-side is not used). Structurally, it’s a Transformer decoder with extremely large context and token embedding dimensions. This gives it the capacity needed to reason across long-form case studies, sleep explanations, and multi-step fitness logic.

The base model (Gemini Ultra 1.0) already has strong performance in medical question answering and general health reasoning. But by itself, it doesn’t know how to interpret raw wearable data — it needs domain-specific training.

Finetuning

The first major training step is a full-parameter finetuning of Gemini Ultra on 857 expert-annotated sleep and fitness case studies. Each case study includes: (1) Up to 30 days of daily metrics, (2) Aggregated (mean, variance, percentiles) statistics, (3) Expert-written insights, etiologies, and recommendations. This dataset is unique because each example combines real sensor patterns with expert-level reasoning. When the researchers fine-tuned the model, they essentially taught it:

“When you see patterns like this in the data, here’s how a real sleep doctor or athletic trainer explains it.”

To teach Gemini Ultra how to behave like a real sleep and fitness coach, the researchers fine-tuned the entire model—not just a small part of it—using a large collection of expert-written examples. Each example paired a user’s real wearable data with the exact explanation a sleep doctor or athletic trainer would give. There were about thirteen hundred of these pairs for sleep and fifteen hundred for fitness, and the model was trained on them over roughly fifteen hundred optimization steps. Instead of using shortcuts like LoRA, they updated all of the model’s weights directly, following a smooth cosine schedule for the learning rate so the model gradually stabilized as it learned. After this full training process, the base Gemini model effectively “became” PH-LLM: an LLM that now understands how to read multi-day sensor patterns and talk like a personal health expert.

Multimodal Adapter for Sensor Data

Once PH-LLM can generate coaching advice, the second challenge is enabling it to interpret raw numerical sensor data for prediction tasks (like estimating how tired someone will feel).

To do this, the researchers added a custom MLP-based multimodal adapter. This is the most “technical” part of the architecture.

How the adapter works:

For each of the 20 wearable sensor signals (HRV, resting HR, sleep duration, etc.), the system collects 15 days of values.
It computes standardized mean and variance for each signal.
These 40 numbers (20 means + 20 variances) feed into a multi-layer perceptron (MLP):
- Input: 40
- Hidden layers: 1024 → 4096 → 1024 (ReLU activations)
- Output: 4 “soft tokens,” each of size 14,336 (the embedding size of PH-LLM).
These 4 soft tokens are prepended to the text input as if they were real tokens.
The LLM then processes the numerical data inside its own embedding space, letting its reasoning layers combine subjective sleep patterns with wearable readings

The LLM never sees raw numbers — it sees learned embeddings representing the person’s physiological state. This allows PH-LLM to achieve machine-learning level accuracy in predicting subjective sleep outcomes, without needing a separate ML pipeline.

Input Format and Context Handling

The system uses two kinds of input representations:

Textual Representations
- Daily tables written out in text
- Time ranges (bedtime, sleep duration)
- Percentile comparisons
- Metric summaries
Soft Token Representation via Adapter
- Encodes underlying numerical structure
- Injected directly into the model’s attention layers

Because PH-LLM was trained on long, structured case studies, it naturally handles: multi-day patterns, variations in missing data, differences in available context.

Output Structure

PH-LLM produces multi-part responses in the same structure experts use:

Insights: Patterns detected in the data
Etiology: Possible causes based on sleep medicine frameworks (like RU-SATED)
Recommendations: Personalized, SMART-style advice
Readiness Scoring (Fitness): Evaluation of fatigue, HRV trends, and recovery loads

Because the model was trained on structured expert templates, it learns to produce cohesive, medically-grounded narratives instead of generic advice.

Evaluation Architecture

The team built a secondary system called AutoEval, which is an LLM finetuned to grade PH-LLM’s responses against expert criteria. This created an automated loop for model validation, enabling: fast benchmarking, ablation studies, large-scale quality scoring.
AutoEval itself is built using Gemini Pro with LoRA finetuning

Strengths of the system

What I like most about PH-LLM is how personal it feels. Instead of giving generic sleep or fitness advice, it actually looks at your patterns over weeks and speaks to you the way a real coach would. When it notices your bedtime shifting or your deep sleep dropping after intense workouts, it doesn’t just state the numbers—it explains what those changes mean and why they matter.

The system’s expert-level reasoning also stands out. In the paper, PH-LLM performs as well as or better than trained professionals on board-style sleep and fitness exams, scoring 79% in sleep medicine and 88% in fitness. This gives its recommendations a level of credibility that most consumer health apps don’t have.

Another strength is how smoothly it blends wearable data with human-like reasoning. Using the multimodal adapter, PH-LLM can predict subjective feelings like tiredness or restfulness based purely on sensor trends, something even many traditional ML models struggle with.

Finally, PH-LLM adapts well when information is missing or incomplete. The paper shows that when certain sleep or workout metrics are removed, the model still adjusts its reasoning instead of failing outright. That flexibility makes it feel more intelligent and reliable, almost like it truly understands how sleep and fitness behaviors change from day to day.

Limitations

The biggest one, in my opinion, is its dependence on the quality of the wearable data itself. If your device misreads your sleep stages or your heart rate jumps around because the watch wasn’t snug, PH-LLM will still try to interpret that noise as if it’s meaningful. The paper even points out that the model sometimes references data incorrectly or forms conclusions that don’t perfectly match the input — small confabulations that become more noticeable when the data is messy

Another issue is that the model sometimes struggles with consistency when giving recommendations. For sleep insights, the fine-tuning helped a lot, but for fitness coaching the improvements were smaller, and in certain sections like “training load,” PH-LLM actually performed worse than the base Gemini model and human experts. That tells me the model doesn’t fully grasp every fitness scenario as deeply as it understands sleep patterns.

The paper itself admits there were demographic skews — more middle-aged users, fewer younger or older participants, and no information about race or ethnicity. That means the eva.

And finally, there’s the broader limitation that PH-LLM, no matter how smart it sounds, is not a medical device. It can give coaching-style suggestions, but it isn’t validated for clinical decision-making. Sometimes its tone feels authoritative, which can make the advice sound more medically precise than it actually is.

Final Takeaways

What impressed me most is the model’s ability to turn a messy week of habits into a simple, actionable story. It doesn’t lecture or overwhelm. It just explains what’s going on and nudges you in the right direction. At the same time, it’s important to remember that PH-LLM isn’t a medical system. It still makes small mistakes, relies heavily on the data it sees, and carries the biases of the Fitbit-dominated population it was trained on.

But the core idea feels powerful. PH-LLM represents a shift from “apps that measure you” to “systems that understand you.” This paper is a glimpse of where personal health AI is heading, and it sets a strong foundation for the next models I’ll be reviewing in this series. As I move into PHIA, the IR Explainer Agent, and later multi-agent systems, I can already see how all these ideas start to connect. PH-LLM feels like the first major building block in creating an AI that doesn’t just track your health—but helps you improve it.

AI for Personal Health — My Review, Reflections & Real-Life Understanding

Mitansh Gor — Thu, 27 Nov 2025 05:32:31 +0000

VIDEO LINK

1. Introduction

I’ve been using wearables like Fitbit and Pixel Watch for a while, and I’ve always had the same question in the back of my mind:
“These devices collect so much data… but what does it actually mean for my health?”

I could see my steps, sleep score, heart rate, all the usual numbers — but I didn’t really know how to connect them to real insights about my body. That curiosity is what pulled me into this world of AI for personal health.

As I started reading research papers and experimenting with data, I realized something important:
AI is slowly becoming the “missing link” between raw wearable data and actual, useful guidance.
It can interpret patterns, explain them in simple language, and sometimes even coach you like a personal guide.

This blog series is my attempt to share what I learned — not as a researcher writing a formal report, but as a student trying to make sense of a fast-moving field in a friendly, simple way.

What This Series Will Cover (Based on the Timeline of Publications)

I studied four papers and mapped out how AI in personal health evolved over the past few years.
Here’s the order I’ll follow in this series — from earlier work to the most recent:

Intro Blog (this one) - Why I’m exploring this topic and how the whole series is structured.

Blog 1 (2024) — PH-LLM - A large language model that gives personalized sleep and fitness coaching using wearable data.

Blog 2 (2024) — PHIA - An agentic LLM that writes code, analyzes your data, and turns it into meaningful health insights.

Blog 3 (2025) — IR Prediction + IR Explainer Agent - A combination of machine learning and an LLM-based explainer to estimate metabolic risk.

Blog 4 (2025) — Personal Health Agent (PHA) - A multi-agent system where different AI “roles” collaborate — data analyst, coach, and domain expert.

Blog 5 (My Project, 2025) - My own proof-of-concept that mixes ideas from all four systems to create a unified health agent architecture. My goal is to keep everything straightforward, even if someone is reading about AI for the first time.

Why This Topic Matters (and What I Hope You’ll Learn)

Most of us don’t think much about our health until something feels wrong. But our wearables are quietly tracking us every day — our sleep, heart rate, movement, and even stress signals. That means they’re collecting clues long before we ever feel anything.

AI can turn those clues into something meaningful by noticing patterns we might miss, predicting early risks, explaining things in plain English, and even giving small suggestions that feel personal. To me, this makes health feel more understandable and less like a set of random numbers.

Through this series, I want to share what I learned about these systems in a simple, relatable way. I’ll talk about how wearables work with AI, how different models make sense of health data, why LLMs and multi-agent systems matter, where the ethical issues show up, and how these ideas connect to real projects.

I’m writing as a student — sharing what made sense to me, what confused me at first, and what finally “clicked.” And at the end of the series, I’ll also walk through my own proof-of-concept project where I combine everything I learned into one architecture.

10. References

PH-LLM: A Personal Health LLM for Sleep & Fitness Coaching
Nature Medicine (2025)
https://www.nature.com/articles/s41591-025-03888-0
PHIA: Transforming Wearable Data into Health Insights Using LLM Agents
Google Research Blog (2024)
https://research.google/blog/advancing-personal-health-and-wellness-insights-with-ai/
SHARP Framework: Principles for Building Health & Wellness LLMs
Google (2025)
https://services.google.com/fh/files/blogs/winslow_2025_sharp_framework.pdf
Insulin Resistance Prediction From Wearables + Routine Blood Biomarkers
arXiv (2025)
https://arxiv.org/pdf/2505.03784
Personal Health Agent (PHA): Multi-Agent System for Data + Coaching + Expertise
arXiv (2025)
https://arxiv.org/pdf/2508.20148

Nested Learning — My Reflections on a Model That Learns How to Learn

Mitansh Gor — Mon, 17 Nov 2025 02:57:13 +0000

I recently came across a paper called Nested Learning: The Illusion of Deep Learning by Behrouz and the team — the same researchers behind Titans and Atlas. It really caught my attention because it challenges what we usually think “deep learning” means. The paper says that depth in neural networks isn’t just about stacking layers — it’s about how many layers of learning the system can apply to itself. Instead of just updating weights, this model learns how to improve its own learning process.

While reading it, I realized this isn’t just another optimization trick. It actually feels like a glimpse into what real intelligence could be — an AI that doesn’t just react but reflects, improves, and evolves how it learns over time. The authors even built a prototype called Hope, a model that modifies itself using feedback, learning not just what to learn but how to learn better.

Take on the Problems in Deep Learning and Transformers

Today’s deep learning systems — even transformers — are strong but still limited in how they actually learn.

Neural networks are called “deep” because of their layers, but their learning is flat — one optimizer like Adam or SGD updates everything the same way. Once training ends, the model stops learning, like a student who graduates and never studies again. :)

Transformers improved context handling, but they only have short-term memory. They remember what’s inside their context window and forget the rest. Even if I teach GPT something new, it won’t remember it next time — its knowledge is frozen.

To me, that’s what makes this paper exciting. It explores real, continuous learning, where models don’t just perform tasks but actually grow and evolve from their experiences — more like how the human brain learns over time.

How the Human Brain Actually Learns

When I compared this idea to how our brain works, the difference was clear. The brain doesn’t just store facts — it keeps updating how it learns from every experience. Each new moment changes not only what we know but how we learn next time.

Like when I study late and remember less, my brain quietly adjusts — it learns how to learn better. We also have different kinds of memory: fast, short-term memory for quick thoughts and slow, long-term memory for what truly matters.

What’s amazing is that all this happens at different speeds — reflexes form instantly, habits take time. That’s what makes our learning flexible and self-improving. This paper helped me realize that real intelligence isn’t just about storing knowledge — it’s about systems that can adapt the way they adapt.

Why Real Intelligence Learns at Many Speeds

“Nested Learning” mirrors the way our brain learns at multiple speeds. In real life, not all learning happens instantly — some lessons come from quick feedback, and others sink in over time through reflection and repetition.

For example, when I make a mistake in code, I fix it fast — that’s short-term learning. But when I notice a pattern of mistakes across projects and change how I approach debugging, that’s slow, higher-order learning. My brain is basically nesting layers of learning, one inside another.

This is exactly what the paper argues AI should do. Instead of having one rigid update rule for all situations, it should have systems that operate on different timescales — fast ones for reacting to the present and slow ones for improving how it learns in the future. Real intelligence, human or artificial, grows when it can learn fast, remember slow, and keep adjusting both.

Why Optimizers Are Still “Shallow”

One part that stood out to me was how the paper calls our current optimizers “shallow.” At first, that sounded odd — optimizers like Adam or SGD are what make models learn, right? But the point is deeper: they only operate at one level. They adjust the weights, but they never learn how to optimize better on their own.

Think about it like this — an optimizer is a rulebook. It says, “If error is high, change parameters this way.” That rule never changes, no matter how the model behaves or what patterns it encounters. It doesn’t evolve. It’s like a student who keeps using the same study method forever, even when it stops working.

Nested Learning challenges that. It treats the optimizer as something that can learn from its own history — almost like giving the optimizer memory and awareness. So instead of being a fixed rule, it becomes a learner itself. That’s why normal optimizers are called “shallow” — they only see one layer of the learning process, while true intelligence needs many.

What Nested Learning Really Means

When I finally got to the main idea — Nested Learning — it clicked for me that this isn’t just about deeper networks, but deeper learning loops. Normally, a model learns by updating its parameters once per round. But in Nested Learning, there are multiple layers of learning stacked inside each other, each operating at its own level.

The paper calls these “levels.”

Level 1 is the fast learner — it adjusts to new data right away.
Level 2 is slower — it learns how well Level 1 is learning and changes its strategy.
Level 3 and beyond keep zooming out, letting the model reflect on its own updates and tweak the process itself.

It’s like having a mind inside a mind inside a mind — each layer watching and improving the one below. What makes it powerful is that it never stops at one rule; it can always find a better way to learn. I realized that’s what makes it feel almost human — because that’s how we grow too, by not just learning facts, but by constantly refining how we learn them.

The Power of Associative Memory

In this paper, associative memory is what allows the model to connect surprise signals over time. Each time it encounters something unexpected, it doesn’t just correct the output; it stores that “surprise” pattern and learns from how surprises evolve. So instead of forgetting past mistakes, it builds a history of how it has been wrong before — and uses that as context for new learning.

I liked how this turns memory from a passive storage system into an active, learning part of the network. It’s not just remembering data; it’s remembering how learning felt last time. That’s what makes the system more adaptive and self-improving, just like how human intuition forms through repeated experiences.

HOPE — The Model That Learns to Learn

The paper introduces a model called HOPE (Hierarchical Optimizing Processing Ensemble), and it really ties everything together. HOPE is basically the first real example of Nested Learning in action. It builds on the older Titans architecture, which was already designed for smart memory management — storing “surprising” experiences and forgetting the rest. But Titans could only update its parameters in two layers, which meant it was still limited to first-order learning.

HOPE takes that concept and adds self-modification. It doesn’t just store knowledge — it rewrites how it learns based on what it experiences. The more it learns, the better it gets at learning itself. That’s what makes it “hierarchical” — every layer is optimizing the one below it, creating an infinite loop of improvement.

When I read that, it felt like looking at a prototype for true adaptive intelligence. HOPE doesn’t just grow its memory; it evolves its own way of thinking. It’s almost like the model is building its own brain architecture in real time — guided only by feedback and surprise.

CMS — Learning Across Multiple Memory Speeds

One of the coolest parts of the paper was the Continuum Memory System (CMS). This idea clicked with me right away because it’s inspired by how our brain manages memory at different speeds. We have fast, short-term memory for reacting to the moment, and slower, long-term memory for storing what truly matters. CMS brings that same principle to AI.

In HOPE, CMS creates layers of memory that operate on different time scales. Fast memory reacts instantly to new data, slow memory holds onto stable knowledge, and middle layers balance both. The system learns what to keep, what to adapt, and what to forget — automatically.

This makes the model more flexible and less likely to “forget” old knowledge when it learns something new, solving a big problem in continual learning. For me, CMS felt like giving the model an actual sense of time — letting it learn short-term lessons without losing its long-term wisdom. It’s memory that grows, refines, and stays balanced, just like ours.

From Titans to HOPE — How the Architecture Evolved

Before HOPE came along, there was the Titans architecture, which was already an interesting idea. Titans worked like a long-term memory system for AI — it didn’t try to remember everything, only what was surprising. Whenever the model saw something that didn’t match its expectations, it marked that as “important” and stored it. This made Titans good at keeping rare or unexpected experiences while forgetting routine ones, kind of like how our brain remembers unusual events more vividly than daily habits.

But Titans had a big limitation — it could only learn at two levels. It could store knowledge (Level 1) and slightly adjust how it stored it (Level 2), but it couldn’t modify its own learning process. It was stuck with fixed update rules, so even though its memory was smart, its way of learning stayed static.

That’s where HOPE (Hierarchical Optimizing Processing Ensemble) came in as the next step. HOPE keeps Titans’ “surprise-based memory,” but adds self-modification — meaning it can change how it learns over time. Instead of just remembering, it reflects on how it learned and improves that process.

In simple terms:

Titans learns what to remember.
HOPE learns how to learn better next time.

This shift from reactive memory (Titans) to reflective learning (HOPE) is what made the architecture truly recursive — an AI that can not only adapt but evolve its own learning rules.

My Takes on the Paper

This paper made me rethink what “deep learning” really means. It’s not just about adding layers — it’s about adding levels of learning. I liked how it pushed the idea that intelligence should evolve, not just perform.

What stood out to me was the mindset shift. Instead of models that just learn tasks, it showed a system that learns how to learn better. That’s the kind of loop real intelligence needs — self-awareness in its own process.

I also liked the balance between fast and slow learning. It reminded me of how humans think — reacting quickly to new events while slowly refining long-term habits. The whole idea felt less like training a model and more like nurturing an evolving mind.

GEN-AI 5 : WGAN and WGAN-GP

Mitansh Gor — Sun, 20 Apr 2025 01:19:50 +0000

Why ❌ GAN

In the ever-evolving landscape of generative models, GANs have taken center stage with their remarkable ability to generate data that mimics real-world distributions. But as with all great things, classic GANs came with caveats—training instability, vanishing gradients, and mode collapse, to name a few.

Let’s dive deeper into these powerful models: WGAN and its enhanced cousin WGAN-GP, two sophisticated upgrades that fix many of GANs' shortcomings.

What is Wasserstein distance

Wasserstein Distance, also known as the Earth Mover's Distance (EMD), is a mathematical measure of the distance between two probability distributions.

Imagine you have two piles of sand—one symbolizing real data and the other representing data generated by a model. The goal is to reshape one pile into the other by moving portions of sand. The effort required depends not only on how much sand you move but also on how far you move it. This effort is what the Wasserstein Distance measures—a cost-efficient way to transform one distribution into another.

What makes this distance especially powerful is its ability to compare distributions even when they don't overlap at all—a situation where traditional measures like JS divergence fail. It offers a continuous and interpretable signal for how "far off" the generated data is from the real data, making it incredibly useful for training generative models like WGANs.

Traditional GANs use Jensen-Shannon (JS) divergence to measure similarity between distributions.
BUT :

JS divergence becomes undefined or uninformative when distributions don't overlap.
This leads to vanishing gradients—a huge problem for training

By contrast, Wasserstein Distance:

Remains well-defined and continuous even when distributions are far apart.
Provides meaningful gradients, allowing the generator to learn effectively from the start.

Improvement in WGAN wrt GAN

WGAN is a variant of GAN that uses the Wasserstein Distance as its loss function instead of JS divergence.

Key differences from traditional GAN:
The discriminator is now called a critic. It doesn't classify data as real/fake, but scores it to reflect how “real” it looks.

The loss function is based on the difference in critic scores for real vs. fake data.

In simple terms:
Real Data → High Critic Score
Fake Data → Low Critic Score
The generator improves by producing data that the critic scores higher.

Working

Just like in standard GANs, WGAN consists of two neural networks:

A Generator (G) that tries to generate realistic data from random noise.
A Critic (C) (not called a discriminator here) that scores the realness of data, assigning higher values to real data and lower values to fake ones.

Unlike GANs, which use a sigmoid function to classify samples as real or fake, WGAN's critic outputs real-valued scores—this small shift changes everything.

The Loss Function

The heart of WGAN lies in replacing Jensen-Shannon divergence with the Wasserstein distance (also called Earth Mover’s Distance)—a metric that measures how much "effort" it takes to morph the generated distribution into the real one.

This results in the following loss functions:

Critic's Loss:

𝐿c = E[C(fake)]−E[C(real)]

The critic aims to maximize the difference, rewarding real data with higher scores and penalizing fakes.

Generator's Loss:

𝐿G = −E[C(fake)]

The generator tries to minimize the critic’s ability to detect its fakes by generating better, more realistic data.

Missing Something? 🤔

The model that we learned until now has some issues.

The critic assigns real-valued scores to data samples, so nothing stops the critic from amplifying its outputs to maximize the loss. For example, it might assign real samples a score of +10,000. And fake samples a score of -10,000. This leads to :
- The generator receiving extreme gradients, which are not helpful and can distort learning.
- Loss curves spike or crash. Generated samples look worse over time instead of better, making training unstable.
- Mode collapse and exploding gradients can occur.

How can we solve this problem?
Let's introduce a 1-Lipschitz Function. This will solve our problem.
We will add a 1-Lipschitz Function to the critic (discriminator).
In the Lipshitz function, we will give any 2 images (x1 and x2) as input.

Numerator: Absolute difference between the critic predictions
Denominator: Average pixelwise absolute difference between two images.

What we are trying to do with this function is to limit the rate at which the predictions of the critic can change between 2 images, when comparing it with the real/pixel-wise difference of the image.

You can get this using a graph by visualizing 2 (white) cones whose origin can be moved along the graph so that the whole graph always stays outside the cones.

By doing so, we are clipping the weights of the critic to lie within a small range, eliminating problems of exploding gradients.
But again NEW PROBLEM !!
Learning significantly decreases 😢 😢

A much better Solution is the WGAN-GP

Adding Gradient Penalty

As the Lipshitz function did a partial job. We need to modify/tweak the function so that it can eliminate the problem of Learning significantly decreases.
The solution should be to penalize the Gradient if it's too far away from 1.

The Lipschitz constraint is enforced by including a gradient penalty (GP) term in the loss function for the critic. The GP penalizes the model if the gradient norm deviates from 1
The GP term, which we will be using with Lipschitz constraint, will measure the square difference between the gradient of the prediction wrt input image and 1.

From the above image, it's clear that the gradient penalty increases as we move away from 1.

There are a few important things to take care of while using training WGAN-GP :

With Wasserstein loss, the Critic must be trained to converge before updating the Generator. To do so, we train the Critic several times between Generator updates. A typical ratio used is 3 to 5 Critic updates per Generator update.
Batch Normalization shouldn't be used in the critic. Batch Norm creates a correlation between images in the same batch. This makes the gradient penalty loss less effective. Experiments have shown that WGAN-GPs works great without it anyway

Results after WGAN-GP model:

Using Wasserstein distance + Lipschitz function + Gradient penalty has improved the results of the images generated significantly.

Traditional GANs suffer from vanishing gradients or mode collapse due to the Jensen-Shannon divergence. WGAN-GP, by using Wasserstein distance + gradient penalty, leads to more consistent and stable convergence. You get smoother generator updates, making training less sensitive to hyperparameters and architectural choices.
In classic GANs, generators can produce a limited variety (mode collapse). WGAN-GP encourages the generator to explore the data distribution more fully, thanks to the meaningful gradient feedback from the critic. Hence, more diverse outputs, especially noticeable in image generation tasks.
The loss correlates with sample quality, unlike traditional GAN loss, which becomes meaningless when the discriminator is too good. Hence, you can track training progress numerically, not just visually
There are no sudden spikes or flat zones in gradients. Hence, more efficient and productive generator learning.

Disadvantages of WGAN

Slower training, higher memory usage, and increased training time, especially on large datasets.
Hyperparameters are sensitive. Even a small change in hyperparameters can cause a lot of change. It needs to be tuned carefully.

Summarize

A WGAN-GP uses the Wasserstein loss
The WGAN-GP is trained using labels of 1 for real and -1 for fake
No Sigmoid activation in the final layer of the Critic
Include a GP term in the loss function for the Critic
Train the Critic multiple times for each update of the Generator
There are no Batch Norm layers in the Critic

GEN-AI 4 : GAN

Mitansh Gor — Sun, 20 Apr 2025 01:19:40 +0000

In the world of generative models, Variational Autoencoders (VAEs) were among the first to show us how machines can learn to create new data. But they came with limitations which sparked the need for a new direction in generative modeling.

Enter Generative Adversarial Networks (GANs): a powerful, game-theoretic approach that changed the landscape by producing stunningly realistic outputs and pushing the boundaries of what machines can generate.

Lets start by understanding the limitations of VAE.

Disadvantages of VAE

Blurry Outputs: In a VAE, two things are being balanced: how well the model can recreate the input data (reconstruction loss) and how closely the latent variables follow the assumed normal distribution (KL divergence). If this balance isn't right, it can cause problems. For example, if the model focuses too much on making the latent space look normal, it might not focus enough on accurately recreating the input data, leading to blurry or low-quality reconstructions. Finding the right balance is tricky, and if it's off, the output can look fuzzy or unclear.
Latent Space Constraints: When we assume a simple normal distribution for the latent variables in a VAE, it means we expect the data to follow a very basic pattern (like a bell curve). However, real-world data is often more complex and doesn't always fit this simple pattern. Because of this, the VAE might not capture all the details or variations in the data, leading to less accurate results and poor generation of new data. Essentially, the model's assumption limits its ability to understand and recreate the data’s true complexity fully.
Mode Averaging: When the data has many different patterns or "modes" (called multimodal data), VAEs can struggle to capture all of them. The model tends to average over these modes, meaning it might create a mix of features from different data patterns instead of focusing on the specific details of each mode. This can result in the model generating outputs that don’t accurately reflect the diversity of the data, often leading to a loss of important variations or details in the generated samples. Essentially, the model might not fully capture all the unique aspects of the data.

How is GAN better than VAE?

While Variational Autoencoders (VAEs) offer a powerful framework for generative modeling, they come with certain limitations. To address these challenges, Generative Adversarial Networks (GANs) provide an alternative approach. Unlike VAEs, which rely on a probabilistic framework, GANs use two neural networks—a generator and a discriminator—that compete against each other to improve the quality of generated data. The adversarial setup in GANs enables them to produce sharper, more realistic outputs, particularly in applications like image generation, where detail and realism are crucial.

Here’s how GANs are better than VAEs:

Sharper Outputs: GANs produce clearer and more detailed images because they focus on distinguishing real data from generated data, leading to more realistic results.
No Blurry Reconstructions: Unlike VAEs, which can sometimes produce blurry outputs due to their reliance on a probabilistic framework, GANs avoid this issue by directly optimizing for realism.
Better for High-Quality Generation: GANs excel in tasks where the goal is to generate high-quality data, like realistic images, videos, or audio, because the adversarial training encourages the generator to produce more lifelike results.
Flexibility in Learning Complex Patterns: GANs can learn complex data distributions better, especially when data has many modes (variety of patterns), without averaging out the details like VAEs might.

Working of GAN

A Generative Adversarial Network (GAN) is a machine learning model that consists of two neural networks, a generator and a discriminator, that work against each other in a process called adversarial training. Here's how it works in detail:

The Generator:

The generator is like an artist trying to create new data (e.g., images) that looks as real as possible. It starts with random noise and uses this as input to produce a generated output. The goal of the generator is to produce data that can fool the discriminator into thinking it’s real.

The Discriminator:

The discriminator is like a critic trying to tell whether a piece of data is real (from the training data) or fake (produced by the generator). It takes an input—either real data or a generated one—and outputs a probability that the input is real or fake. It can be considered as a supervised image classification problem.

Adversarial Training:

The key idea behind GANs is the competition between the generator and the discriminator:

The generator tries to improve by creating more convincing, realistic data to fool the discriminator.

The discriminator tries to improve by getting better at distinguishing real data from the fake data created by the generator.

This process is a game where the generator tries to "cheat" by generating better data, and the discriminator tries to become more skilled at detecting fake data. Over time, both networks improve.

The Objective:

Generator's Goal: Minimize how often the discriminator correctly identifies fake data. It wants to produce data that the discriminator can't distinguish from real data.

Discriminator's Goal: Maximize its ability to correctly classify real vs. fake data, helping it get better at spotting fakes.

The interaction between the discriminator and generator in a GAN requires a delicate balance. If the discriminator is too good, it can easily distinguish real from fake data, leaving the generator with little feedback and hindering its ability to improve. On the other hand, if the generator is too strong, it may exploit weaknesses in the discriminator, producing fake data that the discriminator wrongly classifies as real (false negatives). The ideal situation occurs when the discriminator outputs a value close to 0.5, meaning it cannot distinguish between real and fake data, indicating that the generator is producing high-quality samples. For example, if the discriminator outputs ~1, the fake images are too realistic, and the generator won't be forced to improve. If it outputs ~0, the fake images are too obvious, and the generator needs more training. However, when the discriminator outputs ~0.5, it suggests the generator is performing well, producing convincing, realistic samples.

The Loss Functions:

The generator's job in a GAN is to create fake data that looks so real that the discriminator can’t tell it apart from the real data. Instead of directly focusing on the quality of the data, the generator’s main goal is to trick the discriminator into thinking the fake data is real. The generator gets feedback from the discriminator, which helps it improve. The loss function for the generator encourages it to produce data that has a high probability of being classified as real by the discriminator

The discriminator in a GAN's job is to tell whether the data it sees is real (from the true data source) or fake (generated by the generator). It outputs a probability (between 0 and 1) that the input is real.
The goal of the discriminator is to correctly classify Real data as real and Fake data as fake.

Together, the overall training objective for the GAN can be summarized as :
GAN Loss = Discriminator Loss + Generator Loss

Training Process:

The training process of a Generative Adversarial Network (GAN) involves a series of steps that alternate between training the discriminator and the generator.

Initialize the generator and discriminator.
Train the discriminator on real and fake data.
Train the generator to fool the discriminator.
Alternate between training the discriminator and the generator.
Repeat until the generator produces convincing fake data.

We must alternate the training of these two networks, making sure that we only update the weights of one network at a time!

Initially, the generator creates very poor, random data, and the discriminator easily detects it as fake. This adversarial training process continues until the generator has learned to produce realistic data, and the discriminator has become more adept at distinguishing real from fake data.

Training can be stopped when the generator produces high-quality data that the discriminator cannot reliably distinguish from real data (i.e., the discriminator’s output approaches 0.5, meaning it cannot differentiate between real and fake data).

Situations in GAN

1) - We normalize the input range from 0, 255 to [-1, 1] instead of [0, 1] because the tanh activation function works better with inputs in the range of [-1, 1]. The tanh function outputs values between -1 and 1, and normalizing the data to this range helps ensure the gradients are stronger and more stable during training. In contrast, the sigmoid activation function has a range of [0, 1] and produces weaker gradients, which can slow down learning. Therefore, using [-1, 1] helps improve the training process.
2) - The training process of GANs can be unstable because the generator and discriminator are constantly competing. Over time, the discriminator may become too good at distinguishing real from fake data, which could cause issues. However, this isn't always a problem because the generator might have already learned enough by that point. To improve stability, we can add a small amount of random noise to the training labels, which helps prevent the discriminator from becoming too dominant too quickly

3) - When the Discriminator Overpowers the Generator,

The discriminator becomes too good at distinguishing real from fake images.
This causes the generator to receive weak feedback, and the loss signal becomes too weak to improve the generator.
The discriminator perfectly classifies real and fake images, causing the gradients to vanish, and the generator stops training.
Solutions to Weaken the Discriminator:
Increase the Dropout rate in the discriminator to reduce its ability to overfit.
Reduce the learning rate of the discriminator to slow down its training.
Reduce the number of convolutional filters in the discriminator to limit its capacity.
Add noise to the labels when training the discriminator to make it harder for the discriminator to distinguish real from fake.
Randomly flip labels of some images during training to confuse the discriminator and prevent it from becoming too powerful.

4) - When the Generator Overpowers the Discriminator:

The discriminator becomes too weak, and the generator tricks it with a small set of nearly identical images.
This results in mode collapse, where the generator produces limited variety in its outputs, focusing on a single observation that fools the discriminator.
Solutions to Weaken the Generator:
If you find that your generator is suffering from mode collapse, you can try strengthening the discriminator using the opposite suggestions to those listed in the previous section. Also, you can try reducing the learning rate of both networks and increasing the batch size.

5) - The generator's loss is evaluated based on the current discriminator, which constantly improves during training. This makes it hard to compare the generator's loss at different stages, as it may not reflect the actual quality of generated images. The loss can even increase over time, despite the images improving, because the discriminator is getting better, making it harder for the generator to fool it.

Disadvantages of GAN.

Mode Collapse: The generator may start producing limited, identical outputs, tricking the discriminator, and failing to capture the full diversity of the data.
Uninformative Loss: The generator's loss may increase even as image quality improves, making it hard to track progress.
Vanishing Gradients: If the discriminator becomes too powerful, the generator may receive weak gradients, preventing meaningful learning.
Hyperparameter Sensitivity: GANs are sensitive to choices like learning rates, dropout rates, and the number of layers, requiring careful tuning for optimal performance.

GEN-AI-3 : VAE

Mitansh Gor — Sun, 20 Apr 2025 01:19:30 +0000

So far, we’ve seen how Autoencoders (AEs) can take data—like images, audio, or text—and compress it into a lower-dimensional space, only to reconstruct it again like a digital magician pulling a rabbit out of a hat. Pretty neat, right?

But there’s a catch: while AEs are great at learning compact representations, they’re not exactly dreamers. Their latent space—the core of their compressed understanding—isn’t built for imagination. Try sampling from it randomly, and you’re more likely to get noise than meaningful data.

Now, what if we wanted a model that not only compresses data but can also generate new, meaningful samples that look like they came from the original dataset? A model that understands probability, uncertainty, and can dream up new content with finesse?

🎉 Enter the Variational Autoencoder (VAE).

In this post, we’re going to unpack how VAEs work, why they’re a major leap forward from traditional autoencoders, and how they lay the groundwork for some of the most exciting generative models in AI today.

Let’s dive in.

The Problem with Regular Autoencoders 🧩

Traditional autoencoders compress data by learning a direct mapping from input to a latent vector, and then decompress it using a decoder. While effective for feature learning, they suffer from:

Disorganized Latent Space: Nearby latent points don’t necessarily produce similar outputs.
Poor Generative Ability: Sampling randomly from the latent space usually results in noisy, incoherent outputs.
No Uncertainty Modeling: The model doesn't capture how confident it is in the latent representation.
We needed structure, smoothness, and the ability to reason probabilistically.

Variational Autoencoders

VAEs solve these issues by introducing two key ideas:

Injecting randomness into the encoding process.
Imposing constraints on the distribution of the latent space.

Instead of encoding an input to a fixed point, the VAE encodes it to a probability distribution—specifically, a multivariate normal distribution centered at a point in the latent space.
Or in other words,
Instead of encoding input into a fixed latent vector, it encodes it into a distribution (usually Gaussian), from which it samples.

By doing so, it not only compresses data, but VAEs learn to imagine variations.

🎯 Constraints on the latent space

To bring structure to the latent space, we constrain how encodings are distributed:

🌀 Centering: Each encoded distribution should be centered as close to the origin (0, 0, ..., 0) as possible.
📏 Unit Variance: The spread (standard deviation) of each distribution should be close to 1.

The further the encoder deviates from these goals, the higher the loss during training.

Why? This forces all encoded samples to live near the same area in latent space, enabling smooth interpolation and consistent generation.

Changes in VAE Encoder

The encoder no longer outputs just a point, but rather, it parameterizes a distribution from which we can sample.

x → (μ, σ) → z ~ N(μ, σ²)

We do this because we want to generate new data by sampling from a continuous, meaningful latent space.

z_mean (μ): The center of the distribution where we want our encoding to be.
z_log_var (log(σ²)): The (log of) the variance, controlling the spread of that distribution.

Together, they define a normal distribution, from which we will later sample to get a latent variable z for decoding.

Now the question is why log(σ²) and not just σ² ?
There are two major reasons: positivity constraint, numerical stability

Variance Must Be Positive: If we try to output σ directly from a neural network, we need to ensure that the network only produces positive values. But neural nets naturally output values from (−∞,∞). Hence, log(σ²) guarantees that σ>0, always.
We work in the log domain, where multiplications become additions, and exponentials become linear. It avoids premature underflow for tiny variances or instability from tiny gradients when optimizing σ directly.

Imagine we want to allow the encoder to learn variances from 0.0001 to 1000:
If we output σ directly, the network must learn to span that huge dynamic range. But if we output log(σ²), the values range from about:
log(0.0001)=−9.2 to log(1000)=6.9.
A much more manageable range!

But wait on!!
There is one problem.

Let's go through it using the example

Let’s imagine the encoder outputs:
μ=0.5
σ=1.0
If we directly sample
z = np.random.normal(mu, sigma)

This operation:
Picks a random number (say 0.23 or 1.42) with no guarantee

There's no way to know how the output Z would change if μ or
σ changed — because the randomness hides the function's slope

And without a gradient, the network can’t learn.

How can we deal with it?

Reparameterization Trick

Here’s the twist:

Instead of sampling like this:
z∼N(μ,σ2)
We reparameterize it as:
z = μ+ σ⋅ε where ε∼N(0,1)
Here :

ε = Random noise from a fixed distribution
μ, σ = Output from the encoder (learnable)
z = Latent vector to feed into the decoder

Why this works:

ε is independent of the network, so its randomness doesn’t interfere with gradient flow.
μ and σ are now involved in a deterministic operation (addition and multiplication), so gradients can be calculated
Now, you can compute:
∂z/∂μ = 1 ∂z/∂σ = ε

Backpropagation is happy again. 🎉

This tiny trick makes backpropagation work through the stochastic layer. Without it, training would collapse.

Changes in VAE Decoder wrt AE

The decoder no longer sees a fixed vector in the latent space.
Instead, it gets:
z=μ+σ⋅ε
The decoder must be able to take any nearby sample around μ and still reconstruct a very similar output.

VAE Loss Function: More Than Just Reconstruction

We used reconstruction loss for AutoEncoders. We cannot use just the reconstruction loss anymore. We’re now dealing with probability distributions, and we need to keep the latent distributions close to a standard normal (N(0,1)).

So we add a Kullback–Leibler (KL) divergence term:

Loss = ReconstructionLoss + β∗KLDivergence

KL measures how much our learned distribution (μ, σ²) deviates from the standard normal. A higher KL means our encoding is straying too far.

Think of it like this:
Reconstruction loss: "How well did we recreate the input?"
KL divergence: "How wild is our latent distribution? Should we calm it down?"

What's the use case of the β Parameter?

The β coefficient balances reconstruction and regularization.

If β is too small, the KL divergence is ignored. The latent space becomes disorganized, similar to a vanilla AE. Good reconstructions, bad generation.
If β is too large, the model prioritizes matching N(0,1) over reconstruction. All samples start looking the same—blurry outputs, poor expressiveness.

✅ Sweet spot: When β is balanced, we get coherent generation and meaningful reconstructions.

🧱 Disadvantages of VAEs

Tends to generate blurry images
KL term is tricky to balance with reconstruction loss
Not ideal for high-resolution data (GANs often outperform here)

TL;DR: VAEs are smarter but also harder to tame.

🧭 Wrap-up: When to Use What and Why It All Matters

Use Autoencoders when you need compression, denoising, or anomaly detection.
Use VAEs when you need controlled generation, diversity, and smooth interpolation.
Use GANs when you need photo-realism.

The future of generative models lies in hybrids — combining VAEs, GANs, and Diffusion models.