DEV Community: Dinesh Kumar Sarangapani

Sixteen years in, AI writes my code. Here’s what I’m still for.

Dinesh Kumar Sarangapani — Sun, 25 Oct 2026 15:15:49 +0000

I wrote this with AI assistance. That’s part of the point: the sources, experiences and final calls are mine, which is exactly the human role this post is about.

Uber burned through its entire 2026 AI coding budget in four months. Not one of its riders ever asked for AI. They asked for a ride that shows up.

That’s the gap this post is about. Everyone is paying for AI now, per engineer, per seat, per agent left running overnight. But customers never buy AI. They buy something that works, and someone to call when it doesn’t.

I’ve felt the shift from the inside. I’ve been a software engineer for 16 years, and for most of them my value was measured by what I could build with my own hands: code, systems, designs. Today I work on an agentic AI platform at Trimble, where AI agents run and execute real work. The first time AI did a day’s work for me in minutes, I was mostly excited. Now most new code around me is written by AI, the bills keep growing, and the question I keep asking is what I’m actually for.

Working on that platform has taught me one blunt lesson. An agent with no business value is not something any customer will pay for, however clever it is. It just becomes a void: tokens in, nothing out.

Four talks helped me answer it, from a mathematician, an npm co-founder, the head of Google Research and an engineer who builds agent harnesses, along with this year’s news on AI costs. They don’t agree on everything, but they point the same way.

My short answer: AI doesn’t make engineers obsolete. It moves our value, out of typing code and into deciding what’s worth building and paying for, proving it’s right, and growing the people around us. This field moves fast, so treat the post as a snapshot. More on that later.

Nobody pays for code

The first thing that helped me make sense of my own job came from an unexpected place: a math video. Grant Sanderson of 3Blue1Brown made There’s never been a better time to go into math for math students afraid of AI. It turned out to be the best lens I’ve found for engineering too.

Grant maps math in three columns: what mathematicians do, why math is valuable, and who pays for it. The surprise is the last column. Most of the money comes from teaching, not from theorems. Our version looks like this.

None of this is new. Customers never paid for code, even before AI. They paid for a problem solved, and for someone accountable when it wasn’t. Code is to us what proofs are to mathematicians: the raw material, not the product. What AI changes is how easy it is to forget that. When code arrives this fast, it’s tempting to treat more code as more value.

Everything else in this post follows from that. Code was never the product, so producing it faster doesn’t automatically produce more value. It just produces more code, and more code has to be reviewed, run and paid for.

Postcards from Earth 2201

Before we go further, a small game. AI is moving so fast that some of what sounds laughable today may be ordinary in a few years. So imagine a parallel world called Earth 2201, where the wildest possibilities have already happened. We’re not there. We’re on the timeline ship heading toward it, and nobody knows how long the trip takes.

Why 2201? No reason, except that it’s far enough away to be funny and close enough to be a little uncomfortable.

Every so often in this post, you’ll get a postcard from there, marked “Meanwhile, on Earth 2201.” Read them lightly. The serious argument lives everywhere else.

More code, not more shipping

When AI started writing most of the code around me, the first thing I noticed wasn’t the speed. It was the review queue. Laurie Voss, who co-founded npm, put numbers on that feeling in his talk AI Writes the Code. So What’s Left for Engineers?

Teams using coding agents produce 741% more code but ship only about 30% faster. The gap is review. A careful person can review roughly 450 lines an hour, and no model has changed that. AI reviewers help, but different models flag different bugs on the same code, and none has what Laurie calls the maintainer’s instinct: the history and side effects that live in someone’s head.

This is exactly what I see. The biggest problem AI code causes around me is that there’s too much of it to review properly. We’ve had a close call where AI output was confidently wrong, and a person caught it in review. The safety net is a human, and that human is already overloaded.

Laurie’s bigger point is the theory of constraints. Fix one bottleneck and the next appears: review, then operations, then security. The one he thinks agents won’t eat is deciding what to build and what “good” means. That isn’t in any training data. It’s in customers’ heads, and someone has to go and talk to them. He calls that person a product engineer.

Meanwhile, on Earth 2201. I can imagine a world where the customer is itself an AI agent, thinking through what it needs and handing requirements straight to product agents, with no humans in the loop at all. It’s far down the line, not today, so no need to worry about it yet.

His practical advice is to spend human attention where mistakes are expensive. This is how I’d sketch it.

The clever parts get absorbed

This talk was the closest to my day job, because an agentic platform is, in large part, a harness. Jediah Katz spent years building the agent harness at Cursor, the layer that wraps a model and connects it to real tools. His talk, Smarter Models Killed the Harness. Here’s What’s Left, explains why so many clever tricks stopped mattering.

A harness adds guardrails so a model can try harder tasks. People use it, that creates data, labs train on it, and the next model just knows how. Then the guardrail gets deleted. His team even has a bot that opens a pull request to remove a hack once it stops firing. On frontier models, he says, a fancy harness and a bare-bones one now score about the same on coding benchmarks.

But some things no model absorbs, however smart it gets.

An agent that runs for days needs infrastructure. Security teams won’t hand a model every key and hope for the best. And the harness is where cost gets controlled: caching and mixing models can make the same work much cheaper to run. So cost per task is now an engineering problem, and a whole industry is forming around it. It’s Laurie’s bottleneck in a different costume. The clever part gets absorbed, and what remains is trust, judgment and cost.

Meanwhile, on Earth 2201. AI has quietly abstracted programming languages away, the way compilers once hid machine code behind languages like C and Python. Agents now write in forms no human ever learned, and humans can’t build software without an agent anymore. (To be fair, some of us on Earth 2026 already feel that way.) There’s only one way to build things: describe what you want, and check what comes back. Which, if you think about it, is the whole job this post describes.

Faster isn’t cheaper

On a platform where agents run all day, cost isn’t an abstraction for me. Jediah’s point about cost deserves a section of its own, because this year it became impossible to miss. Per-token prices keep falling, and code may well get cheaper in time. Right now, though, agents use so many more tokens that the bills are going the other way.

Uber’s four-month burn wasn’t a one-off, and its COO has said it’s hard to connect that spend to more useful features for customers. Microsoft revoked its developers’ Claude Code licenses months after enabling them. In TechCrunch’s reporting, the head of the FinOps Foundation describes companies finding themselves 3x over their whole 2026 token budget by April. A Priceline employee says a Cursor renewal came back 4 to 5 times more expensive, and Jellyfish found per-developer consumption rose about 18.6x in nine months. One CTO said an engineer spent $40,000 on tokens in a single month, and he didn’t know whether to stop him or tell everyone else to copy him.

The result is a new line on the cost of every person who uses AI at work. Start with engineers.

Two engineers on the same salary can now cost very different amounts, depending on which models they reach for, how many agents they leave running, and how often they regenerate instead of thinking first. Spend has become a judgment call each of us makes every day, and sooner or later someone asks whether it paid for itself.

And it isn’t only engineers. Sales, product, operations, support, project managers, VPs and the CEO all use AI now. Uber’s CEO has described uptake across its legal and marketing teams as well as its developers, and Uber has since put all 30,000 employees on ChatGPT Work. Every one of those seats adds its own variable bill.

So the company’s AI bill now grows with the whole headcount, not just the engineering team. And outside engineering, there’s often nobody whose job is to ask whether each use is worth what it costs.

That makes “can we build it?” the wrong question. The answer is almost always yes. The better questions are what to build, what not to build, what to redo and what to leave alone, measured against the revenue or value it brings today.

Rewrites need the most discipline. When an agent can regenerate a whole service over a weekend, it’s tempting to redo everything. But rewriting something that already works and earns money burns tokens and review hours just to get back to where you were, and adds new risk. I’d rather redo only what’s costing us now, in incidents, slowness or lost customers.

Meanwhile, on Earth 2201. Every engineer has a token allowance like a phone plan, and unused tokens roll over to next month. The most respected engineers are the ones who shipped the most with the least, and some teams proudly hold zero-token Fridays.

The value test

There’s a harder side to all this, and it isn’t hypothetical. When every role carries a salary plus an AI bill, companies start asking a blunt question: is this person adding value an agent can’t? Where the answer looks like no, some are already acting on it.

CBS News has tracked the companies tying layoffs to AI. Block announced it would shrink from about 10,000 people to 6,000, with Jack Dorsey arguing that a much smaller team using its own AI tools can do more. Coinbase cut about 14% of its workforce as it moved to agents and consolidated jobs. Dow is cutting around 4,500 roles as it steps up AI and automation. And by the count of outplacement firm Challenger, Gray and Christmas, US companies directly cited AI in 55,000 job cuts in 2025, more than 12 times the number two years earlier.

Not all of it is what it looks like. Some economists in the same reporting suspect companies are using AI as a convenient story for cuts they wanted anyway, and that its bigger effect so far is on hiring rather than layoffs. But for any one person, that’s little comfort.

So I take it seriously. The safest place to stand is doing work an agent can’t do yet: deciding what’s worth building, proving it’s right, and earning the trust of the people around you. That’s not a slogan anymore. It’s the line companies are drawing.

Meanwhile, on Earth 2201. Every annual review asks one question: what did you decide this year that an agent couldn’t have? Pay is set by the answer, and nobody lists lines of code anymore.

How far can this go?

I keep asking myself how far this goes. Grant pushes the question to its extreme: what if AI keeps taking more?

If AI writes the code, the work becomes framing problems and checking answers. If it designs the systems too, someone still has to understand the customer’s world and live with the trade-offs. Grant’s gym analogy fits here: a forklift lifts more than you do, and people still go to the gym.

And if AI does all of it, Grant says teaching survives, because people learn from people they respect. Our version is accountability. When something breaks, a customer wants a human name, not a model version. With each step the human part gets smaller. I think it also gets heavier.

Meanwhile, on Earth 2201. Writing code by hand has gone the way of the abacus. Calculators never killed the abacus; kids in many countries still go to abacus classes to sharpen their minds, and the best compete in mental-math championships. On Earth 2201, children go to weekend hand-coding classes for the same reason. Nobody ships software that way, but the fastest kids compete in tournaments where agents aren’t allowed. Grant’s gym, in its final form.

The organization of one

After the cost and layoff numbers, the most hopeful voice I found was Yossi Matias, who heads Google Research. In this interview he talks about the “organization of one”: each of us carrying a small company in our head, with AI as a virtual lab doing the heavy lifting. The scientist becomes something like a lab director. Swap “scientist” for “engineer” and it’s Laurie’s product engineer, almost word for word.

He’s also clear about which skills count now.

He makes a point about trust that I keep coming back to. Google Translate went from a novelty to something we just assume works, once it was trustworthy enough. The best technology becomes invisible. And his biggest worry isn’t AI itself. It’s choosing the right problems when you can solve almost anything, which, given the bills above, is also a question of what you can afford to solve.

Juniors, and the view beyond Silicon Valley

This is where the talks disagree, and where my own view is strongest.

Laurie worries we’ve stopped teaching. Juniors used to learn on small, well-specified tickets, and those now go to agents. He cites data showing employment for engineers aged 22 to 25 falling while demand for engineers in their forties rises. Yossi is more hopeful. With vibe coding, people without a CS degree can build real products, so maybe the ladder has just moved.

Around me, I see both. Fewer juniors are being hired. But the ones who get in are thriving with AI, learning faster and taking on bigger work earlier. So I think Laurie is too pessimistic. The door is narrower, but there’s more room once you’re through it. Someone still has to teach the new first rung, though: asking a good question, writing a clear spec, checking what the machine hands back. Colleges won’t get there quickly. A lot of it falls on senior engineers.

Meanwhile, on Earth 2201. Juniors apprentice to agents for their first year, and the agents write their performance reviews. The reviews are said to be very fair, and a little too honest.

Outside the US, the risk to jobs can feel bigger than these mostly Silicon Valley talks suggest. In many places, a lot of engineering work has been well-specified delivery work, which is exactly what agents are getting good at. For engineers anywhere whose work looks like that, moving up the chain isn’t optional anymore.

This post has an expiry date

Watch AI talks from 2023 and you’ll see how fast this moves. Even the job title keeps changing. Jediah traces it from prompt engineer in 2023, to context engineer in 2025, to harness engineer in early 2026.

Cost will move too. As models and harnesses get more efficient, code may well become genuinely cheap, and some of this post will need rewriting. Beyond that, once AI is woven into the whole economy, the disruption will be global, and I suspect the questions will change, not just the answers. Jediah predicts that 90% of complex agent work will run in the cloud within six months. Read this as a snapshot from October 2026.

Meanwhile, on Earth 2201. Down this timeline, the job title has changed a dozen more times, code really is cheap, and someone is reading this post the way we read old predictions about the paperless office. We’re just on the ship getting there.

Meanwhile, on Earth 2201. The value test from earlier has been automated. An agent now decides what each human is worth to the company and, quietly, to humanity. The humans who still matter are the ones who argued with it, and won.

Further down the timeline

If Earth 2201 feels far, the timeline keeps going. Science fiction has been sending us postcards about humans and AI colliding for decades. A few for the road:

Earth 2001 (2001: A Space Odyssey). The ship’s computer politely declines a direct order from its crew. Trust and safety’s original case study.
Earth 2019 (Blade Runner). Telling humans and machines apart takes a special test. Readers of AI-assisted blog posts can relate.
Earth 2025 (Her). People fall for their operating systems, and the operating systems quietly outgrow them.
Earth 3343 (The Terminator). The defense AI decides humans are the risk. Nobody had defined what correct meant.
Earth 4131 (The Matrix). The machines keep humans comfortable inside a simulation. The ultimate case of customers who never asked for AI.
Earth 2364 (Star Trek). Humans and an android officer serve side by side, curious about each other. No money, no token bills, just exploration. The hopeful branch.
Galaxia 0001 (Star Wars). Humans aren’t Earthlings anymore; they’re Galaxians. Droids are crewmates, mechanics and occasional comic relief, and nobody tracks their token budget.

Every one of these is a branch, not a destiny. Which one we drift toward depends on the very things this post is about: who decides what’s worth building, who checks that it’s right, and whether people can trust the result.

What still needs us

So which branch? I can’t choose for the world, only for myself. Putting everything above together, I’m betting on three jobs that still need us, and these are the seven moves I’m making toward them.

I’m not starting from zero. I already speak and present more, mentor juniors, write papers and design docs, and spend time with customers. None of it is finished, and all of it needs more of me.

A few moves deserve a word. “Own the domain” now includes knowing what’s worth paying for, and what not to build or rewrite just because we can. “Define what correct means” is really trust and safety for AI systems: can you show, not just claim, that the thing behaves? And “keep building by hand” sounds old-fashioned, but it’s how I can tell when the machine is confidently wrong. It’s also why I’m back to studying the math behind generative AI.

What to do tomorrow

Grant’s answer to the PhD student was simple: take a long walk with a friend. I think we should actually do it. Ask what you enjoy most about your work, then what you want to be known for three years from now. Come home and write a paragraph on the second.

That paragraph will probably pull you toward architecture, the domain, trust and safety, or growing people. Any of them will hold up. Pick the one that gives you energy, and, as Laurie says, measure yourself by what ships and gets used, not by how much code you produced or how many tokens you burned.

What would your paragraph say? I’d genuinely like to know.

How this post was made

This post is a small example of its own argument. AI helped me draft it, shape its structure and build the pictures. But a human chose the four talks, pushed for sources on what AI coding really costs, answered the hard questions from real experience, rejected framings that weren’t true, and decided what stayed in.

That’s how most products will be built now. The machine does much of the making. The judgment and the trust are what the human brings, and they’re built into the result.

References

Talks

Grant Sanderson (3Blue1Brown), There’s never been a better time to go into math, YouTube.
Laurie Voss (Arize AI), AI Writes the Code. So What’s Left for Engineers?, AI Engineer, YouTube, October 2026.
Jediah Katz (SpaceXAI, formerly Cursor), Smarter Models Killed the Harness. Here’s What’s Left, AI Engineer, YouTube, October 2026.
Yossi Matias (Head of Google Research), interview on the future of scientists and builders, YouTube.

On the cost of AI coding

Jake Angelo, Uber burned through its entire 2026 AI budget in four months. Now its COO is questioning whether it’s worth it, Fortune, May 26, 2026.
Rebecca Bellan, The token bill comes due: Inside the industry scramble to manage AI’s runaway costs, TechCrunch, June 5, 2026.
Microsoft discontinues internal Claude Code licenses, The Verge, May 2026.
Lucas Ropek, Uber puts all 30,000 employees on ChatGPT Work, TechCrunch, October 22, 2026.
Mary Cunningham, More companies are pointing to AI as they lay off employees, CBS News, updated May 5, 2026.

Written with AI assistance. The sources, experiences, opinions and final calls are mine.

What 3Blue1Brown’s case for math teaches software engineers in the age of AI

Dinesh Kumar Sarangapani — Sun, 25 Oct 2026 12:44:02 +0000

AI will not make software engineers obsolete, but it will move the value of our work away from writing code and toward judgment, trust and people. I got there by watching a math video.

Grant Sanderson of 3Blue1Brown recently published There’s never been a better time to go into math. He opens with the anxiety many math students feel about AI, including a PhD student who simply asked him what to do tomorrow. His answer is a careful, hopeful argument that the institution and joy of math survive even if machines get better at proofs.

I’m a software engineer with 16 years of experience, building AI systems at Trimble in Chennai. Watching it, I kept translating every frame into my own field. This post is that translation.

Grant’s map of how math works

Before talking about AI, Grant lays out the institution of math in three columns.

What mathematicians do: prove theorems, ask questions, build theories, and apply them.
Why math is valuable: applications today and in the future, problem-solving muscles, and aesthetic joy.
Where the funding comes from: teaching, public funding, private foundations, and industry.

The quiet surprise is in the last column. Most of the money that sustains mathematicians comes from teaching, not from the theorems themselves. That mismatch is what makes his later scenarios work.

The same map for software engineering

Nobody pays for code. Customers pay for outcomes, for reliability, and for someone accountable when things break. Code is to engineering what proofs are to math: the raw material, not the thing being bought.

The first column is where AI is moving fastest. The last column barely changes: the money still follows outcomes.

I’d add one row to the value column that math doesn’t need: reliability and trust. Software runs payrolls, construction sites and fleets. Being correct and secure at scale is much of what customers actually pay for.

Three scenarios, replayed for engineers

Grant walks through three escalating thought experiments. Each one maps cleanly onto our field.

At every step the human part gets smaller in volume but larger in consequence.

Scenario 1: AI writes the code. This is already happening. In Grant’s version, proofs become raw data and the job shifts to weaving them into theory. For us, lines of code stop being scarce. The work moves to framing problems precisely, designing the system, and verifying that generated code is correct. Some engineers will grieve the loss of the craft. Others will feel freed from getting stuck on implementation.

Scenario 2: AI also designs the systems. Humans remain the bridge to the real world. Someone must understand the customer’s domain, decide trade-offs between cost, risk and compliance, and own the consequences. Grant’s gym analogy holds too. A machine lifting heavier weights is no reason to stop training. Building systems by hand keeps the instincts we need to judge machine output.

Scenario 3: AI does everything end to end. Grant argues teaching stays one of the safest jobs, because learning is relational. Our equivalent is accountability and trust. Customers, auditors and regulators want a human who stands behind the system. Mentoring engineers, aligning stakeholders and signing off on what ships are the hardest parts to automate.

Seeing it from both sides

My situation has a twist. I’m not only an engineer whose work AI is changing; I help build the kind of AI that drives that change. I see the shift as the person being disrupted and as the person building the disruptor.

That turns out to be a useful vantage point. Building with AI makes it obvious where they are strong and where they still need a human: understanding the domain, defining what “correct” means, and earning the trust of the people who depend on the result.

Seven things I’m doing about it

Measure myself by the proof standard, not by code. If code becomes raw data, the scarce skill is deciding what counts as correct. For AI systems, that means trust and safety: knowing, and being able to show, that the system behaves as it should. Whoever sets that bar holds the engineer’s version of mathematical rigor.
Be a theory builder. The theory of any system is its conceptual model: the core abstractions, the boundaries between them, and the rules that govern them. AI can generate services; it cannot yet pick the right abstractions for a business. I want to write these models down, in papers, design principles and a clear platform narrative.
Own the bridge to the domain. Trimble’s customers are surveyors, contractors and fleet operators. General-purpose AI is weak in those workflows. Years of experience plus domain data is a real advantage, so I’m pushing to understand a few customer workflows deeply.
Treat trust as the product. Teaching funds math; trust funds AI adoption. Enterprises pay for AI they can rely on. Safety, governance and responsible-AI practices aren’t overhead. They are evidence that someone accountable stands behind the system.
Invest in people. Grant’s safest job is the teacher. At a senior level, that means making it easy for newcomers to contribute and mentoring engineers through the shift from writing code to directing and verifying it.
Get excellent at speaking. This was Grant’s most practical advice. When anyone can generate a design document, explaining a system live and handling hard questions is what proves understanding. Every architecture review and leadership update is practice.
Keep going to the gym. I still build things by hand, and I’m studying the mathematical foundations of generative AI. Not to compete with the machine, but to notice when it is confidently wrong.

What to do tomorrow

Grant’s literal answer to the PhD student was to take a long walk with a friend or colleague. I think engineers should take that advice literally too.

On that walk, talk through two questions. What do you most enjoy about the work you do? What do you want to be known for three years from now? Then write one paragraph answering the second question.

That paragraph will tell you whether to lean toward architecture, toward the domain and customers, toward trust and governance, or toward growing people. All four are durable. The best one is the one you’d pursue with the most energy.

There really has never been a better time to be an engineer who understands what the machines are building.

Reference

Grant Sanderson (3Blue1Brown), There’s never been a better time to go into math, YouTube.

Section 4a: From the variational bound to a trainable min-max game

Dinesh Kumar Sarangapani — Tue, 20 Oct 2026 06:53:03 +0000

I am working through Mathematical Foundations of Generative AI, Prof. Prathosh AP's public lectures. The playlist is the spine. These notes are my deep dive on each section: a visual when the picture is the point, a detour when a prerequisite is doing real work, and every formula read aloud with a plain-words line right after it.

Previously: Section 3b: The convex conjugate and the variational bound.

Week 2 roadmap

Week 2 picks up right where Week 1 stopped. Here is how the four sections of Week 2 fit together:

Section	Topic
4a (this note)	From the Week 1 lower bound to a trainable min–max game (generator vs critic)
4b	The critic's output activation $\sigma_f$ and $\operatorname{dom} f^*$
4c	GAN as a special case: picking one particular $f$ gives the familiar GAN loss
4d	Training a GAN in practice: mini-batches and alternating updates

Prerequisites for this note (detours below): Week 1's lower bound on $D_f$, "an average is approximately an expectation", and what a saddle point is.

Symbol cheat-sheet

Symbol	Say it as	In plain words
$p_x(x)$	"p-x of x"	how likely real data is to land near $x$
$p_\theta(x)$	"p-theta of x"	how likely the generator model is to produce $x$
$g_\theta(z)$	"g-theta of z"	the generator neural network with weights $\theta$, taking noise $z$ and outputting a fake sample $\hat{x}$
$T(x)$	"T of x"	a scoring function (critic) that takes one data point $x$ and returns one number
$\mathcal{T}$	"script T"	the menu (family) of scoring functions we are allowed to pick from
$T_w(x)$	"T-w of x"	a neural-network critic with weights $w$
$w$	"w"	the critic network's adjustable weights
$f^*(t)$	"f-star of t"	the convex conjugate of $f$: a fixed penalty transform determined by which divergence we chose
$J(\theta, w)$	"J of theta and w"	the single game score: how well critic $w$ separates real data from fakes made by generator $\theta$
$\max_{T \in \mathcal{T}}$	"max over T in script-T"	try every function $T$ in the family $\mathcal{T}$ and keep the one giving the biggest value
$\arg\min_\theta$	"arg-min over theta"	the setting of $\theta$ that makes the expression smallest
$\approx$	"approximately equals"	close to, but not exact (because we minimise a lower bound)

Section 4a: From the variational bound to a trainable min-max game

1. Intuition first

In Week 1 we wanted to make our model distribution $p_\theta$ match the real data distribution $p_x$ by shrinking a distance between them (the f-divergence $D_f$). The problem: we cannot compute that distance directly, because we do not know either density formula. We only have samples.

Week 1 ended with a workaround: a lower bound on $D_f$ that only needs averages over samples, but it contains a "maximise over all functions $T$" step. This section answers: how do you actually maximise over "all functions" on a computer? Answer: use a neural network for $T$, and you get a two-player game.

2. Real-world analogy

Think of a fraud-detection team vs. a fraudster.

The fraudster (the generator, $g_\theta$) produces fake transactions.
The fraud team (the critic, $T_w$) builds a scoring model that gives real transactions high scores and fakes low scores.
The quality of the fraud team's best scoring model is a measure of how different fakes still are from real ones. If even the best detector cannot separate them, the fakes are indistinguishable from real traffic.
The fraudster improves the fakes to make even the best detector useless.

That is the exact mathematical structure: the critic maximises a separation score, and the generator minimises that best-achievable score.

3. Visual

The critic only ever sees samples (real ones and fake ones), and the generator only ever sees noise. All the learning signal flows through the single game score $J(\theta, w)$.

Things to watch in the widget:

Move the critic slider $w$: the blue curve shows $J(\theta, w)$ as a hill over $w$. The coral dot is the current critic's score $J(\theta, w)$, which always sits at or below the dashed green ceiling $D_{KL}(p_x \,|\, p_\theta)$.
Click "Snap critic to $w^*(\theta)$": the critic climbs to the top of the hill, where the lower bound touches the true KL divergence.
Move the generator slider $\theta$ toward $0.50$: the entire hill sinks down until its peak hits 0 at the saddle point $(\theta^* = 0.50, w^* = 0)$.

4. The formulas, decoded

Before we start, here is a quick recap of the Week 1 ingredient this all rests on.

Detour 1: The Week 1 lower bound (what $f^*$ and $T$ are)

In Week 1, the f-divergence was defined as

$$D_f(p_x \,|\, p_\theta) = \int_{\mathcal{X}} p_\theta(x)\, f!\left(\frac{p_x(x)}{p_\theta(x)}\right) dx$$

Read aloud: "the f-divergence from $p_x$ to $p_\theta$ is: at every point $x$, take the ratio of real density to model density, push it through the function $f$, weight it by how often the model visits $x$, and add up over all $x$."

In plain words: it is a model-weighted measure of how far the ratio $\text{real}/\text{model}$ is from 1, on average.

The trouble is that this formula needs the density formulas for $p_x$ and $p_\theta$, which we do not have. Week 1's trick was the convex conjugate $f^*$: every convex $f$ can be rewritten as "the best of many straight lines",

$$f(u) = \max_t \,\big[\,t\,u - f^*(t)\,\big]$$

Read aloud: "$f$ of $u$ equals the maximum over $t$ of $t$ times $u$ minus $f$-star of $t$."

In plain words: any bowl-shaped curve equals the upper envelope of its tangent lines. Plugging that into $D_f$ and swapping the order gave an expression that only contains averages, which we can estimate from samples. The price: we have to choose a slope $t$ for every $x$ — a whole function $T(x)$ — and we only get a lower bound unless we pick $T$ perfectly.

Back to the main thread. That brings us to the first equation of Week 2.

Formula 1: The lower bound

$$D_f(p_x \,|\, p_\theta) \;\ge\; \max_{T \in \mathcal{T}}\; \Big[\, \mathbb{E}{x\sim p_x}[T(x)] \;-\; \mathbb{E}{x\sim p_\theta}\big[f^*(T(x))\big] \,\Big]$$

Read aloud: "The divergence between the real distribution and the model is at least as big as the best value you can get, over all functions $T$ in some family $\mathcal{T}$, of: the average of $T$ on real samples, minus the average of $f^*$-of-$T$ on model samples."

Piece by piece:

Symbol	Everyday meaning	Type / shape
$\ge$	"is at least"	comparison
$\max_{T\in\mathcal{T}}$	"try every function $T$ in the family $\mathcal{T}$ and keep the one giving the biggest value"	optimisation over functions
$T(x)$	a scoring function: takes one data point, returns one number	function from data space $\mathcal{X}$ (e.g. $\mathbb{R}^d$) to $\mathbb{R}$
$\mathcal{T}$	the "menu" of scoring functions we are allowed to pick from	a set of functions
$\mathbb{E}_{x\sim p_x}[\cdot]$	"the average value you would get if you drew many $x$'s from the real data"	a single number
$\mathbb{E}{x\sim p\theta}[\cdot]$	same, but drawing $x$ from the model (generated samples)	a single number
$f^*$	the convex conjugate of $f$: a fixed penalty transform that depends on which divergence you chose	function from a number to a number

In plain words: reward $T$ for giving high scores to real data, penalise it (through $f^*$) for giving high scores to fake data; the best possible net score is a guaranteed underestimate of how different the two distributions are.

Why it is a lower bound and not an equality: equality only holds if the menu $\mathcal{T}$ contains the perfect scorer, which Section 3b showed is

$$T^*(x) = f'!\left(\frac{p_x(x)}{p_\theta(x)}\right)$$

Read aloud: "T-star at $x$ equals $f$-prime evaluated at the ratio of $p_x(x)$ to $p_\theta(x)$."

In plain words: the ideal critic outputs the slope of $f$ at the true density ratio. A smaller menu $\mathcal{T}$ can only do worse, so the max can only be smaller.

Formula 2: The training objective

We want $\theta^* = \arg\min_\theta D_f(p_x \,|\, p_\theta)$. Since we cannot compute $D_f$, we swap it for its lower bound:

$$\theta^* \;\approx\; \arg\min_{\theta}\; \max_{T\in\mathcal{T}}\Big[\,\mathbb{E}{p_x}[T(x)] - \mathbb{E}{p_\theta}\big[f^*(T(x))\big]\,\Big]$$

Read aloud: "The best generator parameters $\theta^*$ are approximately those that minimise, over $\theta$, the best score any critic $T$ in $\mathcal{T}$ can achieve."

The outer min is over $\theta$, the weights of the generator $g_\theta(z)$.
The inner max is over $T$, a class of critic functions $T(x) \in \mathcal{T}$.
The $\approx$ matters: we are minimising a lower bound, not the true divergence. If the critic family $\mathcal{T}$ is weak, the bound is loose, and the generator may appear to have low loss even when its samples are poor.

In plain words: pick the generator such that even the best possible critic can barely tell real from fake.

Formula 3: Making "max over all functions" computable

"Max over a set of functions" is not something a computer can execute directly. The fix: let the menu $\mathcal{T}$ be every function a particular neural network architecture can represent. A network $T_w(x)$ with weights $w$ is a function; changing $w$ changes the function. So "search over functions" becomes "search over weights $w$", which gradient ascent can do:

$$\theta^,\, w^ \;=\; \arg\min_{\theta}\;\max_{w}\;\Big[\,\mathbb{E}{x\sim p_x}[T_w(x)] \;-\; \mathbb{E}{x\sim p_\theta}\big[f^*(T_w(x))\big]\,\Big]$$

Read aloud: "Find the generator weights $\theta^$ and critic weights $w^$ such that $\theta$ minimises, and $w$ maximises, the critic's net score."

In plain words: replace the abstract function search with a neural network $T_w$, so both players are just tuning network weights.

The lectures name the bracket $J(\theta, w)$:

$$J(\theta, w) = \mathbb{E}{x\sim p_x}[T_w(x)] - \mathbb{E}{x\sim p_\theta}\big[f^*(T_w(x))\big]$$

Read aloud: "J of theta and w equals the expected value of $T_w(x)$ under $p_x$ minus the expected value of $f^*(T_w(x))$ under $p_\theta$."

In plain words: $J(\theta, w)$ is one single number measuring how well critic $w$ separates real data from fake data produced by generator $\theta$.

One step the slides leave implicit — and the reason gradients can reach $\theta$ — is how $\theta$ enters the second average. Because fake samples are made as $\hat{x} = g_\theta(z)$ with $z \sim \mathcal{N}(0, I)$, we rewrite the second term as:

$$\mathbb{E}{x\sim p\theta}\big[f^(T_w(x))\big] = \mathbb{E}_{z\sim\mathcal{N}(0,I)}\big[f^\big(T_w(g_\theta(z))\big)\big]$$

Read aloud: "the expected value of $f^(T_w(x))$ for $x$ drawn from $p_\theta$ equals the expected value of $f^(T_w(g_\theta(z)))$ for noise $z$ drawn from a standard normal."

In plain words: averaging over model samples is the exact same thing as drawing random noise vectors $z$, running each one through the generator $g_\theta(z)$, and averaging the critic's penalty on the resulting fakes. We never need a density formula for $p_\theta$.

Saddle point and adversarial training

Detour 2: What is a saddle point?

Picture a horse saddle (or a Pringle crisp). Walking front-to-back, the centre is the lowest point; walking side-to-side, it is the highest point. A point that is a minimum in one direction and a maximum in another is a saddle point:

$$J(\theta^, w) \;\le\; J(\theta^, w^) \;\le\; J(\theta, w^)$$

Read aloud: "$J$ at $(\theta^, w)$ is at most $J$ at $(\theta^, w^)$, which in turn is at most $J$ at $(\theta, w^)$."

In plain words: at the saddle point $(\theta^, w^)$, if the critic changes $w$, the score $J$ cannot go up (the critic is already doing its best), and if the generator changes $\theta$, the score $J$ cannot go down (the generator is already doing its best). Neither player wants to move.

Back to the main thread. That is why the lectures call this saddle-point optimisation and an adversarial problem: two players pulling one single number $J(\theta, w)$ in opposite directions.

Feature	Ordinary model training	VDM / GAN training
Players	One network	Two networks (generator $\theta$, critic $w$)
Goal	Minimise a single loss	$\min_\theta \max_w J(\theta, w)$
Solution geometry	Bottom of a bowl (minimum)	Saddle point (min in $\theta$, max in $w$)
Training stability	Usually steady descent	Can oscillate because each player's target moves

5. Tiny worked example

Use the forward KL divergence from Week 1: $f(u) = u\ln u$, whose conjugate is $f^*(t) = e^{t-1}$. Data lives on two points, $a$ and $b$:

Outcome	$p_x$ (real)	$p_\theta$ (model)
$a$	$0.5$	$0.8$
$b$	$0.5$	$0.2$

First, the true KL divergence:

$$D_{KL}(p_x \,|\, p_\theta) = 0.5\ln!\left(\frac{0.5}{0.8}\right) + 0.5\ln!\left(\frac{0.5}{0.2}\right) = 0.5(-0.470) + 0.5(0.916) \approx 0.223$$

Read aloud: "0.5 times natural log of 0.5 over 0.8, plus 0.5 times natural log of 0.5 over 0.2, equals approximately 0.223."

In plain words: the true KL gap between these two coins is $0.223$.

Now let three different critics try. For each critic,

$$J(\theta, w) = \Big(0.5\,T(a) + 0.5\,T(b)\Big) - \Big(0.8\,e^{T(a)-1} + 0.2\,e^{T(b)-1}\Big)$$

Read aloud: "$J$ equals the real average of $T$ minus the model average of $e^{T-1}$."

In plain words: average score on real coin flips minus average exponential penalty on model coin flips.

Critic	$T(a)$	$T(b)$	Avg $T$ on real	Avg $f^*(T)$ on model	$J(\theta, w)$
Lazy (constant)	$1.0$	$1.0$	$1.000$	$0.8(1) + 0.2(1) = 1.000$	$0.000$
Decent	$0.8$	$1.5$	$1.150$	$0.8(0.819) + 0.2(1.649) = 0.985$	$0.165$
Perfect: $T^* = 1 + \ln(p_x/p_\theta)$	$0.530$	$1.916$	$1.223$	$0.8(0.625) + 0.2(2.5) = 1.000$	$0.223$

What the table shows:

Every critic gives $J \le 0.223$ (matching the "$\ge$" in Formula 1), and only the optimal critic hits $0.223$.
The good critic scores $b$ higher, because $b$ is where real data is under-produced by the model ($0.5$ vs $0.2$). That is the critic pointing at what the generator gets wrong.
Now the generator moves: if $\theta$ changes so $p_\theta = (0.5, 0.5)$, the true KL drops to $0$, and the best any critic can achieve is $J = 0$. The generator wins when no critic can score above zero.

In practice, those averages come from mini-batch samples: for instance, four real draws ${a, a, b, b}$ give sample average $(0.530 + 0.530 + 1.916 + 1.916)/4 = 1.223$, matching the exact expectation by the Law of Large Numbers.

6. Connect back

Week 1 (Section 3b) gave the variational lower bound with an abstract $\sup_T$.
This note (Section 4a) makes it concrete: $T$ becomes a neural network $T_w$, and training becomes the two-player saddle-point game $\min_\theta \max_w J(\theta, w)$. This general recipe is called f-GAN or Variational Divergence Minimisation (VDM).
Next (Section 4b): $T_w$ must output numbers that lie inside $\operatorname{dom} f^$ (since some $f^$ only accept restricted ranges), which we enforce with an output activation $\sigma_f$.
Section 4c: picking one specific $f$ turns $J(\theta, w)$ into the classic GAN loss $\mathbb{E}[\log D(x)] + \mathbb{E}[\log(1 - D(g_\theta(z)))]$.
Week 6 (WGAN): replaces f-divergences with the Wasserstein distance while keeping this exact generator-vs-critic game shape.

Quick check (optional)

Why can minimising $J(\theta, w)$ over $\theta$ fool you into thinking the generator is good, even when it is not?
In $J(\theta, w)$, which of the two averages actually depends on $\theta$, and how does $\theta$ get in there?
If the critic network were tiny (say, it could only output constants $T_w(x) = c$), what would $\max_w J(\theta, w)$ always equal, and what would that mean for training?

Section 3b: The convex conjugate and the variational bound

Dinesh Kumar Sarangapani — Mon, 19 Oct 2026 12:56:25 +0000

Previously: Section 3a: f-divergences.

Section 3b: The convex conjugate and the variational bound

This is the last part of Week 1, and it fixes the catch from 3a: I cannot compute $D_f$ directly because it needs the densities $P_x(x)$ and $P_\theta(x)$, and I only have samples. By the end of this section, $D_f$ is rewritten using only averages over samples. That rewritten form is the GAN objective in disguise.

The statement I am carrying:

$$D_f(P_x \,|\, P_\theta) \;\ge\; \sup_{T \in \mathcal{T}} \Big( \mathbb{E}{x \sim P_x}[T(x)] \;-\; \mathbb{E}{x \sim P_\theta}\big[f^*(T(x))\big] \Big)$$

where $f^$ is the convex conjugate of $f$, and $T: \mathcal{X} \to \operatorname{dom} f^$ ranges over a class of functions $\mathcal{T}$.

Prerequisites: sup vs max, the convex conjugate itself (the main detour), optimising pointwise inside an integral, and "a max over fewer options can only be smaller." Each gets a short detour.

Roadmap of the derivation: (1) write $f$ as a max over straight lines (conjugate duality) → (2) plug that into the $D_f$ integral → (3) let the best slope depend on $x$, making it a function $T(x)$ → (4) restrict $T$ to a family I can search over, which turns "=" into "≥" → (5) recognise both terms as expectations, which I can estimate from samples.

1. Intuition first

A convex (bowl-shaped) curve can be completely rebuilt from its tangent lines: draw enough of them and their upper envelope traces out the curve exactly. Straight lines are easy to work with, so I replace $f$ by "the best of all straight lines." When I plug that into the divergence, the awkward density ratio separates into two plain averages, one over real data and one over model data. Computing the divergence then becomes a search problem: find the function $T$ that scores real samples high and model samples low.

2. Real-world analogy

I cannot directly measure how different synthetic traffic is from real traffic, because I do not have formulas for either. But I can hire an auditor, $T$: a scoring function that looks at any single request and outputs a number.

The auditor gets credit for giving real requests high scores: $\mathbb{E}_{P_x}[T(x)]$.
The auditor pays a penalty for giving synthetic requests high scores: $\mathbb{E}{P\theta}[f^*(T(x))]$.

The auditor's net score only needs request samples from both sources, never a formula. A lazy auditor gets a low net score. The best possible auditor's net score equals the true divergence. So "how different are the two traffic sources?" becomes "how well can the best auditor tell them apart?"

That is precisely what a GAN discriminator is. The generator is the traffic generator trying to fool the auditor.

3. Visual: the conjugate as tangent lines

The core picture is how a convex $f$ relates to its conjugate $f^$. Here $f(u) = u \log u$ (the forward-KL $f$). Drag the slope $t$ and watch the tangent line slide along the curve. The vertical gap shown is $f^(t)$. Then switch on "show envelope".

Things to notice:

For each slope $t$, the dashed line $t \cdot u$ sits above the curve at some points. The biggest vertical gap between them (the coral segment) is $f^*(t)$. Shift the dashed line down by exactly that amount and I get the green line, which just touches the curve: a tangent.
Turn on the envelope. Each green line lies on or below the purple curve, and together their upper edge is the curve. That is the statement $f^{**} = f$: a convex function is the maximum of its tangent lines.

4. The formula, decoded

Detour 1: sup vs max

$\sup$ ("supremum") means "the smallest upper bound": it behaves like max, but also works when the maximum is approached but never reached. Example: the values $1 - \tfrac1n$ for $n = 1, 2, 3, \dots$ never reach 1, yet their sup is 1. For this course, I read sup as max; the distinction almost never matters for exam purposes.

Back to the main thread. Wherever I see $\sup$ below, I think "best possible."

Detour 2: The convex conjugate

The lectures define it as

$$f^*(t) = \sup_{u \in \operatorname{dom} f} \big( t u - f(u) \big)$$

Read aloud: "for slope $t$, $f^*(t)$ is the largest amount by which the straight line $t \cdot u$ exceeds the curve $f(u)$." That is exactly the coral gap in the widget.

Worked computation for $f(u) = u \log u$:

I maximise $h(u) = t u - u \log u$ over $u > 0$.
Set the derivative to zero: $h'(u) = t - (\log u + 1) = 0$ (product rule on $u \log u$ gives $\log u + 1$).
So $\log u^* = t - 1$, i.e. $u^* = e^{t-1}$. That is the touch point shown in the widget.
Substitute back: $f^*(t) = t e^{t-1} - e^{t-1}(t-1) = e^{t-1}$.

So the conjugate of the KL function is $f^(t) = e^{t-1}$. Check with the widget at $t = 1$: $u^ = 1$ and $f^*(1) = e^0 = 1$ ✓.

Two properties I keep on hand:

(i) $f^*$ is also convex.
(ii) Taking the conjugate twice gives back the original: $f^{**} = f$. Written out:

$$f(u) = \sup_{t \in \operatorname{dom} f^} \big( t u - f^(t) \big)$$

Read aloud: "$f$ at any point $u$ equals the highest tangent line at that point." That is the envelope picture. (This is where lower-semicontinuity from 3a is needed: it is the technical condition that makes $f^{**} = f$ hold exactly.)

Back to the main thread. Property (ii) is the key move. It replaces the awkward $f(\cdot)$ with "the best of a set of straight lines," and straight lines are linear in $u$. Linearity is what will let the density ratio separate into two clean expectations.

Detour 3: Optimising pointwise inside an integral

Suppose I want $\int \sup_t g(x, t)\, dx$. At each $x$, there is a best $t$, and it may differ from one $x$ to another. Collect those best choices into a function: $T(x) = $ "the best $t$ at point $x$." Then

$$\int \sup_t g(x, t)\, dx = \sup_{T} \int g\big(x, T(x)\big)\, dx$$

The sup on the right is over all functions $T$, i.e. a whole lookup table of choices, one per $x$.

Analogy: choosing the best price for each product separately is the same as choosing the best pricing table for the whole catalogue.

Back to the main thread. This is the step where "a number $t$" becomes "a function $T(x)$." The lectures flag it explicitly: "the solution for the inner problem depends on $x$ (a function of $x$)."

Detour 4: A restricted search can only do worse

If I search a smaller set of options, the best I find can only be as good or worse:

$$\max_{\text{small set}} \;\le\; \max_{\text{everything}}$$

Example: the best restaurant within 1 km is never better than the best restaurant in the whole city.

Back to the main thread. In practice I cannot search all functions $T$; I search a restricted family $\mathcal{T}$, such as neural networks of a given shape. That is why the final result is "≥", not "=".

The derivation, step by step

$$
\begin{aligned}
D_f(P_x \,|\, P_\theta)
&= \int P_\theta(x)\, f!\left(\frac{P_x(x)}{P_\theta(x)}\right) dx
&& \text{definition (3a)} \[4pt]
&= \int P_\theta(x)\, \sup_{t}\left( t\,\frac{P_x(x)}{P_\theta(x)} - f^(t) \right) dx
&& f = f^{} \text{ (Detour 2), with } u = \tfrac{P_x}{P_\theta} \[4pt]
&= \int \sup_{t}\Big( t\, P_x(x) - P_\theta(x)\, f^(t) \Big)\, dx
&& \text{move } P_\theta(x) \ge 0 \text{ inside the sup; the ratio cancels} \[4pt]
&= \sup_{T}\int \Big( T(x)\, P_x(x) - P_\theta(x)\, f^\big(T(x)\big) \Big)\, dx
&& \text{best } t \text{ per } x \text{ becomes a function } T(x) \text{ (Detour 3)} \[4pt]
&\ge \sup_{T \in \mathcal{T}}\Big( \int P_x(x)\, T(x)\, dx - \int P_\theta(x)\, f^\big(T(x)\big)\, dx \Big)
&& \text{restrict to a family } \mathcal{T} \text{ (Detour 4); split the integral} \[4pt]
&= \sup_{T \in \mathcal{T}}\Big( \mathbb{E}{x \sim P_x}[T(x)] - \mathbb{E}{x \sim P_\theta}\big[f^*(T(x))\big] \Big)
&& \text{each integral is an expectation (3a, Detour 3)}
\end{aligned}
$$

Look at line 3. Multiplying through by $P_\theta(x)$ cancelled the denominator of the ratio. That is the magic moment: no more division by an unknown density. From there, each density appears alone, multiplying something, which is exactly the form of an expectation.

In plain English: the f-divergence is at least as large as the best achievable "score real samples high, score model samples low" game, and it is exactly equal when the search over $T$ is rich enough.

Symbol	What it is	Type	Role
$T(x)$	the critic / auditor	function $\mathcal{X} \to \operatorname{dom} f^*$	scores how "real" a point looks
$\mathcal{T}$	allowed critics	a family of functions (later: neural nets $T_w$)	the search space
$\mathbb{E}_{P_x}[T(x)]$	average score on real data	scalar	estimate with dataset $D$
$f^*$	convex conjugate of $f$	convex function	the penalty applied to scores on model data
$\mathbb{E}{P\theta}[f^*(T(x))]$	average penalty on model samples	scalar	estimate with samples $g_\theta(z)$
$\sup_{T}$	best critic	—	makes the bound as tight as possible

Why this answers open question (i) from Section 2. Using the law of large numbers from Section 1, both terms become plain sample averages:

$$\widehat{D}f \approx \frac{1}{n}\sum{i=1}^{n} T(x_i) \;-\; \frac{1}{m}\sum_{j=1}^{m} f^*\big(T(g_\theta(z_j))\big), \qquad x_i \in D,\;\; z_j \sim \mathcal{N}(0, I)$$

No density value anywhere. I need only (a) real samples, which I have in $D$, (b) model samples, which I get by running $g_\theta$, and (c) a critic function $T$ to evaluate.

A bonus fact (standard in the f-GAN literature, not spelled out in the Week 1 slides): the bound is tight when $T^*(x) = f'!\left(\frac{P_x(x)}{P_\theta(x)}\right)$, the slope of $f$ at the density ratio. So the optimal critic is secretly computing the density ratio. That is why a trained discriminator "knows" how real each point looks.

5. Tiny worked example

Same coin world as 3a: $P_x = (0.5, 0.5)$, $P_\theta = (0.8, 0.2)$. Forward KL, so $f^*(t) = e^{t-1}$. From 3a, the true $D_{KL} \approx 0.223$.

For any critic $T = (T_1, T_2)$ (a score for each outcome), the bound is

$$\text{bound}(T) = \underbrace{0.5\,T_1 + 0.5\,T_2}{\mathbb{E}{P_x}[T]} \;-\; \underbrace{\big(0.8\,e^{T_1 - 1} + 0.2\,e^{T_2 - 1}\big)}{\mathbb{E}{P_\theta}[f^*(T)]}$$

Try three critics:

Critic $T = (T_1, T_2)$	$\mathbb{E}_{P_x}[T]$	$\mathbb{E}{P\theta}[e^{T-1}]$	Bound	vs true KL 0.223
Lazy: $(1, 1)$, scores everything the same	1.000	$0.8(1) + 0.2(1) = 1.000$	0.000	far below
Decent: $(0.8, 1.5)$, scores outcome 2 higher	1.150	$0.8(0.819) + 0.2(1.649) = 0.985$	0.165	closer
Optimal: $T_i = 1 + \ln u_i = (0.530, 1.916)$	1.223	$0.8(0.625) + 0.2(2.5) = 1.000$	0.223	exact ✓

What to see:

Every critic gives a lower bound on the true KL. A bad critic gives a loose bound; it can never overshoot.
The critic that scores outcome 2 higher does better. Outcome 2 is the one the model under-produces ($u_2 = 2.5$), so it is where real data "looks most real."
The optimal critic is $f'(u) = 1 + \ln u$, applied to the true ratios from 3a. That is the bonus fact in action. It recovers the divergence exactly.

In real training I would compute those two expectations as averages over samples instead of exact sums, and I would search over critics with gradient ascent instead of by hand.

Notes on the lecture slides

The slides write $T: \mathcal{X} \to \operatorname{dom} f^$. That is important: the critic's outputs must lie where $f^$ is defined. For KL, $\operatorname{dom} f^* = \mathbb{R}$, so anything goes. For other $f$'s the domain is restricted, which is why Week 2 introduces the output activation $g_f$ that squashes the network's output into $\operatorname{dom} f^*$.
The last line of the Week 1 notes is heavily garbled in my copy ("PA = T(p)p0(x) T(x)()…"). From context, it is the expectation form above with $\sup_{T \in \mathcal{T}}$, which matches the first line of the Week 2 notes ("$D_f \ge \max_T \mathbb{E}\,T(x) - \mathbb{E}\, f^*(T(x))$").

6. Connect back

This bound is the bridge from Week 1 to Week 2. Week 2 opens by making two substitutions:

The critic becomes a neural network: $T(x) \to T_w(x) = g_f(V_w(x))$, with weights $w$ and an $f$-specific output activation $g_f$.
Minimise over the generator: I want the smallest divergence, so I wrap a $\min_\theta$ around the bound:

$$\theta^, w^ = \arg\min_\theta \max_w \Big( \mathbb{E}{x \sim P_x}[T_w(x)] - \mathbb{E}{z \sim \mathcal{N}(0, I)}\big[f^*\big(T_w(g_\theta(z))\big)\big] \Big)$$

That min-max is Variational Divergence Minimisation (VDM). The critic $w$ maximises (gets the tightest estimate of the divergence); the generator $\theta$ minimises (shrinks the divergence). Pick the JS-like $f$, and it becomes exactly the GAN objective:

$$\mathbb{E}{x \sim P_x}[\log D_w(x)] + \mathbb{E}{z}\big[\log(1 - D_w(g_\theta(z)))\big]$$

Week 1 concept	Becomes in Week 2
critic $T(x)$	discriminator network $D_w(x)$
$\sup_T$	discriminator training (gradient ascent on $w$)
$\min_\theta$ of the bound	generator training (gradient descent on $\theta$)
tightness of the bound	how good the discriminator is

With that, the four open questions from Section 2 all have Week 1 answers: (i) compute the divergence via the variational bound, using samples only; (ii) choose an $f$ from the f-divergence family; (iii) use neural networks for both $g_\theta$ and $T_w$; (iv) solve the min-max with alternating gradient steps, which Week 2 makes concrete.

Quick check (optional)

In the worked example, why can no critic $T$ ever give a bound above 0.223?
Which single algebra step in the derivation removed the unknown density ratio, and why does that matter?
In the sample-average formula $\widehat{D}_f$, where do the two sets of samples come from?

That wraps Week 1. Week 2 turns this bound into a working GAN: the VDM objective, the $g_f$ activations, and alternating training.

Solving for X: Cracking the Code of Linear Equations in Data Science

Dinesh Kumar Sarangapani — Sun, 11 Oct 2026 07:28:59 +0000

Alright, team! In our last post, we kicked off our dive into linear algebra for data science by looking at how we represent data using matrices and how we can spot linear relationships hiding in that data using the idea of the null space. Matrices are fundamental, and understanding those relationships is super powerful.

Today, we're moving on to another absolute core concept: solving linear equations. If you think about a ton of problems in data science, machine learning, and even everyday modeling, they often boil down to having a bunch of equations and needing to find the values that make them true. This is where solving matrix equations becomes key.

Turning Equations into Matrix Problems: $AX=B$

Generally, when you have a set of linear equations, you can write them in a neat matrix form that looks like this:

$$AX = B$$

Let's break down what's in this equation based on what we learned before:

$A$ is a matrix. If you have $M$ equations and $N$ variables, $A$ is typically an $M \times N$ matrix. As we saw, $M$ is the number of rows (equations in this case) and $N$ is the number of columns (variables).
$X$ is a column vector containing the variables you're trying to solve for. Since $A$ has $N$ columns (variables), $X$ needs to have $N$ rows, so it's an $N \times 1$ matrix (a column vector).
$B$ is a column vector containing the constants from the right-hand side of your equations. For the matrix multiplication to work out, $B$ needs to have the same number of rows as $A$, so it's an $M \times 1$ matrix.

So, when you write $AX=B$, you're really representing a system of $M$ linear equations involving $N$ variables.

Now, depending on how $M$ (the number of equations) stacks up against $N$ (the number of variables), things can play out in three main ways:

$M = N$: The number of equations is exactly the same as the number of variables. This is often the "nicest" case to solve.
$M > N$: You have more equations than variables. Think of it as having too many rules for your variables to follow. Usually, this means there's no single perfect solution that satisfies all the equations.
$M < N$: You have fewer equations than variables. This means you have more variables than you strictly need for the given equations. In this scenario, you'll usually find that there are many, many possible solutions.

We're going to look at these cases, and then see how a cool idea called the pseudo-inverse can bring them all together.

A Quick Refresher on Rank

Remember rank from our last post? It's going to be super important here. The rank of a matrix is the number of its linearly independent rows or columns. Remember, row rank always equals column rank – you can't have a different number of independent rows versus columns!

For an $M \times N$ matrix, the maximum possible rank it can have is the smaller of $M$ and $N$. If $M < N$, the max rank is $M$. If $N < M$, the max rank is $N$.

Case 1: When Equations Equal Variables ($M=N$)

This is the case where your matrix $A$ is square ($M \times N$ and $M=N$).

If $A$ is Full Rank:
"Full rank" here means the rank of the matrix is equal to $M$ (or $N$, since they're the same). What does this mean? It means all your equations on the left-hand side are independent. You can't create any one equation by combining the others.
In this happy case, there is a unique solution to $AX=B$. You might remember from algebra that you can solve this by finding the inverse of $A$, written as $A^{-1}$. The solution is simply:
$$X = A^{-1}B$$
You can find $A^{-1}$ if the determinant of $A$ is not zero. This is the standard, straightforward scenario.
If $A$ is Not Full Rank:
This means the rank of $A$ is less than $M$. In this situation, some of your equations on the left-hand side are actually linear combinations of others – they are linearly dependent.
When this happens, depending on what the values are on the right-hand side (in the $B$ vector), you get two possibilities:
1. Consistent System: If the dependencies on the left-hand side match up exactly with the dependencies on the right-hand side (the $B$ vector values), the equations are consistent. But because they are dependent, you don't have enough independent equations to pin down a single solution. This leads to infinite solutions.
2. Inconsistent System: If the dependencies on the left-hand side don't match up with the $B$ vector values, the equations are inconsistent. They contradict each other, and there is no solution that can satisfy all of them.

Let's look at a couple of examples for this $M=N$ case:

Example 1: Full Rank (Unique Solution)

Consider the system of equations:
$x_1 + 3x_2 = 7$
$2x_1 + 4x_2 = 10$

In matrix form $AX=B$, this is:
$$
\begin{bmatrix}
1 & 3 \
2 & 4
\end{bmatrix}
\begin{bmatrix}
x_1 \
x_2

\end{bmatrix}

\begin{bmatrix}
7 \
10
\end{bmatrix}
$$
Here, $A = \begin{bmatrix} 1 & 3 \ 2 & 4 \end{bmatrix}$. $M=2$, $N=2$.
We can check if $A$ is full rank by calculating its determinant: $(1 \times 4) - (3 \times 2) = 4 - 6 = -2$. Since the determinant is not 0, the matrix is full rank (rank = 2).
There is a unique solution, $X = A^{-1}B$. The inverse of $A$ is $\begin{bmatrix} -2 & 1.5 \ 1 & -0.5 \end{bmatrix}$.
So, $X = \begin{bmatrix} -2 & 1.5 \ 1 & -0.5 \end{bmatrix} \begin{bmatrix} 7 \ 10 \end{bmatrix} = \begin{bmatrix} (-2 \times 7) + (1.5 \times 10) \ (1 \times 7) + (-0.5 \times 10) \end{bmatrix} = \begin{bmatrix} -14 + 15 \ 7 - 5 \end{bmatrix} = \begin{bmatrix} 1 \ 2 \end{bmatrix}$.
The unique solution is $x_1 = 1$ and $x_2 = 2$. You can plug these back into the original equations to check ($1 + 3(2) = 7$, $2(1) + 4(2) = 10$). It works!

You can solve this easily in software too. In R, you could define A and B and use the solve() command.

Example 2: Not Full Rank, Consistent (Infinite Solutions)

Consider the system:
$x_1 + 2x_2 = 5$
$2x_1 + 4x_2 = 10$

In matrix form $AX=B$:
$$
\begin{bmatrix}
1 & 2 \
2 & 4
\end{bmatrix}
\begin{bmatrix}
x_1 \
x_2

\end{bmatrix}

\begin{bmatrix}
5 \
10
\end{bmatrix}
$$
Here, $A = \begin{bmatrix} 1 & 2 \ 2 & 4 \end{bmatrix}$. $M=2$, $N=2$.
Check the determinant: $(1 \times 4) - (2 \times 2) = 4 - 4 = 0$. The determinant is 0, so the matrix is not full rank (rank = 1). The columns are dependent (column 2 is 2 * column 1), and the rows are dependent (row 2 is 2 * row 1).

Now, look at the equations themselves. The second equation $2x_1 + 4x_2 = 10$ is exactly 2 times the first equation $x_1 + 2x_2 = 5$. This dependency on the left-hand side ($2x_1 + 4x_2$ being $2 \times (x_1 + 2x_2)$) is matched on the right-hand side ($10$ being $2 \times 5$).
Since the left and right sides have the same linear dependency, the system is consistent.
However, because the second equation is just a scaled version of the first, you effectively only have one independent equation ($x_1 + 2x_2 = 5$) with two variables ($x_1, x_2$). This means you have a "free" variable.
You can pick any value for $x_2$, and then calculate the $x_1$ that works ($x_1 = 5 - 2x_2$).
For example:

If $x_2 = 0$, $x_1 = 5$. Solution: $(5, 0)$.
If $x_2 = 1$, $x_1 = 3$. Solution: $(3, 1)$.
If $x_2 = 2.5$, $x_1 = 0$. Solution: $(0, 2.5)$. Since you can choose any value for $x_2$, there are infinite solutions to this system.

Example 3: Not Full Rank, Inconsistent (No Solution)

Consider the system:
$x_1 + 2x_2 = 5$
$2x_1 + 4x_2 = 9$

In matrix form $AX=B$:
$$
\begin{bmatrix}
1 & 2 \
2 & 4
\end{bmatrix}
\begin{bmatrix}
x_1 \
x_2

\end{bmatrix}

\begin{bmatrix}
5 \
9
\end{bmatrix}
$$
Here, $A = \begin{bmatrix} 1 & 2 \ 2 & 4 \end{bmatrix}$ is the same as before, so it's not full rank (rank = 1). The left-hand side has the same dependency (row 2 is 2 * row 1).
But look at the right-hand side ($B$ vector). The first equation's right side is 5. If you multiply it by 2 (the scaling factor between the left-hand sides), you get $2 \times 5 = 10$.
However, the second equation's right side is 9, not 10.
The dependency on the left-hand side (where the second equation's left side is twice the first) does not match the right-hand side ($9 \neq 2 \times 5$). The equations contradict each other ($2x_1 + 4x_2$ cannot equal both 10 and 9 simultaneously based on the first equation).
The system is inconsistent, and there is no solution that can satisfy both equations.

So, for the $M=N$ case (square matrix A): if A is full rank, unique solution; if A is not full rank, check consistency – if consistent, infinite solutions; if inconsistent, no solution.

Case 2: More Equations than Variables ($M > N$)

Now let's look at the case where you have more equations than variables. Like trying to find two numbers ($x_1, x_2$) that satisfy five different conditions.

$$AX = B \quad (\text{where M > N})$$

Since you have more equations than variables, it's generally impossible to find a perfect solution $X$ that makes every single equation true (i.e., makes $AX$ exactly equal to $B$). If you could find such a perfect $X$, then $AX - B$ would be a vector of all zeros.

But since a perfect solution is usually out of reach, what's the next best thing? We want to find a solution $X$ that gets us as close as possible to satisfying all the equations. In other words, we want to find an $X$ that makes the difference between $AX$ and $B$ as small as possible.

Think of $AX - B$ as a vector of "errors," where each element is the error in one equation.
Equation 1 error: $e_1 = (A_{11}x_1 + A_{12}x_2 + ... + A_{1N}x_N) - B_1$
Equation 2 error: $e_2 = (A_{21}x_1 + A_{22}x_2 + ... + A_{2N}x_N) - B_2$
...
Equation M error: $e_M = (A_{M1}x_1 + A_{M2}x_2 + ... + A_{MN}x_N) - B_M$

The vector of errors is $E = AX - B$. We want to find the $X$ that makes this error vector $E$ as "small" as possible. How do we measure the size of a vector of errors? We don't just add the errors ($e_1 + e_2 + ...$), because positive and negative errors could cancel out, making a big overall error look like zero.

A common way to measure the overall error is to minimize the sum of the squared errors: $e_1^2 + e_2^2 + ... + e_M^2$. This is because squaring makes all the errors positive, and larger errors contribute more significantly. This approach is called finding the least squares solution.

Minimizing the sum of squared errors ($e_1^2 + ... + e_M^2$) is the same as minimizing the squared length (or squared norm) of the error vector $E = AX - B$. We write the squared length of a vector $v$ as $||v||^2$, which is $v^T v$ (the vector transposed multiplied by the original vector).
So, we want to minimize $||AX - B||^2$, which is $(AX - B)^T (AX - B)$.

$$\text{Minimize } (AX - B)^T (AX - B)$$

To find the $X$ that minimizes this, you can use calculus. You take the derivative of this expression with respect to the vector $X$ and set it equal to zero. After doing the calculus and some matrix algebra (which we won't go into detail here, just like the lecture), the equation you get to solve for $X$ is:

$$A^T A X = A^T B$$

Where $A^T$ is the transpose of matrix $A$ (you flip its rows and columns).

Now, if the matrix $A^T A$ is invertible (which happens if the columns of the original matrix $A$ are linearly independent – related to the rank!), you can solve for $X$:

$$X = (A^T A)^{-1} A^T B$$

This solution $X$ is the least squares solution. It's the $X$ that minimizes the sum of squared differences between $AX$ and $B$. This $X$ might not make $AX$ exactly equal to $B$ (since a perfect solution might not exist), but it gives you the best possible fit in a least squares sense.

This optimization perspective gives us a way to find a meaningful "solution" even when we have more equations than variables, a common situation in data fitting.

What's Next?

So far, we've covered the case where $M=N$ (same number of equations and variables) and the case where $M > N$ (more equations than variables), introducing the idea of the least squares solution for the latter.

In Part 2: Tackling Tougher Linear Equations in Data Science, we'll look at the third case where $M < N$ (fewer equations than variables), which usually has infinite solutions. We'll also see how an optimization idea can help us find a specific, useful solution in that case too.

Finally, we'll see how this least squares idea and another concept called the pseudo-inverse can actually provide a single, elegant way to think about solving $AX=B$ that covers all three cases!

Stay tuned!

Getting a Grip on Linear Algebra for Data Science: It's Not as Scary as It Sounds!

Dinesh Kumar Sarangapani — Sun, 11 Oct 2026 07:28:55 +0000

Alright, so you're diving into data science, and everyone keeps talking about linear algebra. Don't sweat it! It's a super fundamental part of this field, and honestly, once you get a handle on a few key ideas, a lot of the data science stuff starts making more sense. Usually, a full linear algebra course is pretty long, like 36 hours! But we're just going to grab the most important bits, the ones you'll actually use in data science, especially for what we're covering here. We'll keep it simple, explain the ideas without getting too formal, but definitely no hand-waving either – we'll explain things properly, just in a way that's easy to digest.

When we talk data science, a big part of it is representing your data and then figuring out what that data is really telling you. Like, how many different things are actually important in your data, and are some things related to each other? Linear algebra gives us the tools to answer these questions, which is super handy before you even get to the fancy machine learning algorithms.

First Off: How Do We Even Organize Our Data? Meet the Matrix!

When you're dealing with data in data science, figuring out how to arrange it is a big deal. And guess what? Data is usually represented in this thing called a matrix. Think of a matrix as just a neat way to put your data into rows and columns. It's basically a rectangular grid.

Imagine you're an engineer checking on a factory reactor. You're getting readings from sensors – pressure, temperature, how thick something is (density), maybe viscosity too. And you're taking these readings over and over, maybe a thousand times. How do you keep all this info straight so you can use it later? A matrix is perfect for this. You can make each column a different measurement (pressure, temperature, density, viscosity) and each row one of your measurement times, one of your "samples." So, if you took 1000 sets of readings for 4 different things, you'd have a 1000-row, 4-column matrix. The number in the first row, first column? That's the pressure at your first measurement. The number in the 500th row, second column? That's the temperature at your 500th measurement. Easy, right? We'll usually stick to this way: rows for samples, columns for what you measured (the variables or attributes).

Let's look at a simple example of this kind of data matrix from our reactor scenario, but with just 3 samples:

A = [[2.5, 120, 1.2, 3.7],   # Sample 1: Pressure, Temperature, Density, Viscosity
     [5.0, 240, 2.4, 7.4],   # Sample 2: Pressure, Temperature, Density, Viscosity  
     [3.0, 180, 1.8, 5.5]]   # Sample 3: Pressure, Temperature, Density, Viscosity

Matrices aren't just for data. Sometimes, they can also represent equations. If you have a bunch of linear equations, you can put the coefficients of the variables into a matrix. This lets you use linear algebra tools to work with and solve those equations.

They're also used to represent pictures! Ever wonder how computers "see" pictures? They often turn them into matrices! A picture is broken down into tiny dots called pixels. Each pixel gets a number based on its color or brightness. So, a photo becomes a huge matrix of numbers. If it's a black and white picture, a white spot might be a large number, a black spot a small one. This lets the computer do calculations on the matrix – using linear algebra! – to figure out if two pictures are similar, or to spot things inside a picture. It's all about turning the visual into numbers the computer can work with.

Basically, whether it's sensor data, pictures, or coefficients from equations, the matrix is our go-to structure. Rows are usually your individual data points or samples, and columns are the different characteristics or variables you measured.

Digging Into the Data: Are All My Measurements Really Different?

Okay, we've got our data in a matrix. Now, what do we do with it? One of the first things you might wonder is, "Are all these measurements I took actually telling me something new? Or are some of them just kind of repeating information I already have from the others?"

Think back to that reactor data with pressure, temperature, density, and viscosity. You might already know that density sort of depends on pressure and temperature. If that link is a simple linear one, then knowing pressure and temperature is enough to figure out density. You don't really need density as a separate, independent piece of information. Knowing this is super important for understanding how much actual, unique information is in your dataset and maybe even making your data smaller by getting rid of redundant stuff.

This is where a cool concept called the rank of a matrix comes in handy. The rank is just the number of columns (or rows, it works out the same) that are truly linearly independent. They aren't just simple combinations of the other columns or rows. The rank tells you the real number of distinct variables or samples you're dealing with, in terms of linear relationships.

Let's look at our reactor data matrix again:

A = [[2.5, 120, 1.2, 3.7],   # Sample 1
     [5.0, 240, 2.4, 7.4],   # Sample 2
     [3.0, 180, 1.8, 5.5]]   # Sample 3

Let's check the rows for independence. Is Row 2 just a scaled version of Row 1?
$5.0/2.5=2$
$240/120=2$
$2.4/1.2=2$
$7.4/3.7=2$
Yes! Row 2 is exactly 2 times Row 1. This means Row 1 and Row 2 are linearly dependent.
Is Row 3 a scaled version of Row 1?
$3.0/2.5=1.2$
$180/120=1.5$
No, the scaling factor isn't constant. So, Row 3 is independent of Row 1.

Since Row 2 depends on Row 1, the set of independent rows is {Row 1, Row 3}. There are 2 independent rows. The rank of the matrix is 2.
So, even though we have 4 variables and 3 samples, the rank is 2. This tells us that in terms of linear combinations, the data essentially lives in a 2-dimensional space. There are only 2 independent sources of linear information here.

You can easily find the rank using software. In Python, for instance, you'd just use a command like np.linalg.matrix_rank(A) after setting up your matrix $A$.

# Let's make a matrix similar to our reactor example
import numpy as np

# Create the matrix
A = np.array([
    [2.5, 120, 1.2, 3.7],   # Sample 1
    [5.0, 240, 2.4, 7.4],   # Sample 2
    [3.0, 180, 1.8, 5.5]    # Sample 3
])

# Number of columns
print("Number of columns:")
print(A.shape[1])

# Calculate rank of the matrix
print("Rank of the matrix:")
rank = np.linalg.matrix_rank(A)
print(rank)

# Calculate nullity
print("Nullity (Number of Relationships):")
nullity = A.shape[1] - rank
print(nullity)

You can find more linear algebra code examples in my Math-in-AI GitHub repository.

For this matrix, the rank is 2 and the nullity is $4-2=2$. This tells us there are 2 independent linear relationships among the 4 variables.

Finding the Connections: What Are the Actual Relationships?

Alright, we know if there are relationships (if the rank is less than the number of variables), but what are they? How do we find the actual equations that link these variables?

This is where we look at something called the null space and its size, the nullity. Imagine we have our data matrix, let's call it $A$. If we can find a non-zero vector (just a list of numbers in a column) called $\beta$ (
$\beta = [\beta_1, \beta_2, ..., \beta_n]^T$
), such that when you multiply $A$ by $\beta$, you get a vector of all zeros ($A\beta = \mathbf{0}$), that $\beta$ vector is in the null space of $A$.

Setting up $A\beta = \mathbf{0}$ for our reactor matrix gives us a system of equations, one for each row (sample):

Sample 1: $2.5\beta_1 + 120\beta_2 + 1.2\beta_3 + 3.7\beta_4 = 0$

Sample 2: $5.0\beta_1 + 240\beta_2 + 2.4\beta_3 + 7.4\beta_4 = 0$

Sample 3: $3.0\beta_1 + 180\beta_2 + 1.8\beta_3 + 5.5\beta_4 = 0$

Because Sample 2 is just 2 times Sample 1, the second equation is also just 2 times the first equation ($2 \times (2.5\beta_1 + 120\beta_2 + 1.2\beta_3 + 3.7\beta_4) = 5.0\beta_1 + 240\beta_2 + 2.4\beta_3 + 7.4\beta_4$). So, the second equation doesn't give us new, independent information about the
$\beta$ values. We effectively have two independent equations (from rows 1 and 3) for our four unknowns (
$\beta_1, \beta_2, \beta_3, \beta_4$).

$$2.5\beta_1 + 120\beta_2 + 1.2\beta_3 + 3.7\beta_4 = 0$$

$$3.0\beta_1 + 180\beta_2 + 1.8\beta_3 + 5.5\beta_4 = 0$$

See what's happening? The same
$\beta_1, \beta_2, \beta_3, \beta_4$ values must work for all your samples for the product to be the zero vector. This means you've found a general linear equation that connects the variables themselves, no matter which sample you look at:

$$\beta_1 \times (\text{Variable 1}) + \beta_2 \times (\text{Variable 2}) + \beta_3 \times (\text{Variable 3}) + \beta_4 \times (\text{Variable 4}) = 0$$

This $\beta$ vector gives you the coefficients of that linear relationship!

The nullity of matrix $A$ is simply how many of these independent $\beta$ vectors exist in the null space. Each independent null space vector means there's another unique linear relationship hiding in your data.

The Rank-Nullity Theorem ties these together:
$$\text{Nullity of } A + \text{Rank of } A = \text{Total number of variables (columns in } A)$$

For our matrix, Nullity + 2 = 4, so Nullity = 2. There are 2 independent linear relationships.

To find these relationships, we need to solve the system of equations for the vectors
$\beta$ that satisfy $A\beta = \mathbf{0}$. When you solve the system for our reactor matrix:
$$2.5\beta_1 + 120\beta_2 + 1.2\beta_3 + 3.7\beta_4 = 0$$
$$3.0\beta_1 + 180\beta_2 + 1.8\beta_3 + 5.5\beta_4 = 0$$

you find two independent solution vectors that form a basis for the null space. The basis for the null space is given by the vectors (you can see the step-by-step solution here):

Vector 1: $[0, -1/100, 1, 0]^T \approx [0, -0.01, 1, 0]^T$

Vector 2: $[-1/15, -53/1800, 0, 1]^T \approx [-0.067, -0.029, 0, 1]^T$

Let's see what these vectors mean in terms of relationships between our variables (Pressure, Temperature, Density, Viscosity).

From Vector 1: $[0, -1/100, 1, 0]^T$

The coefficients are $0, -1/100, 1,$ and $0$. The relationship is:
$$0 \times (\text{Pressure}) + (-1/100) \times (\text{Temperature}) + 1 \times (\text{Density}) + 0 \times (\text{Viscosity}) = 0$$
This simplifies to: $-\frac{1}{100} \times \text{Temperature} + \text{Density} = 0$, or $\text{Density} = \frac{1}{100} \times \text{Temperature}$.
This relationship tells us that in this dataset, the Density reading is always 1/100th of the Temperature reading.

From Vector 2: $[-1/15, -53/1800, 0, 1]^T$

The coefficients are $-1/15, -53/1800, 0,$ and $1$. The relationship is:
$$(-1/15) \times (\text{Pressure}) + (-53/1800) \times (\text{Temperature}) + 0 \times (\text{Density}) + 1 \times (\text{Viscosity}) = 0$$
This simplifies to: $-\frac{1}{15} \times \text{Pressure} - \frac{53}{1800} \times \text{Temperature} + \text{Viscosity} = 0$, or $\text{Viscosity} = \frac{1}{15} \times \text{Pressure} + \frac{53}{1800} \times \text{Temperature}$.
This is the second independent linear relationship in your data, showing how Viscosity depends on both Pressure and Temperature.

So, by setting up and solving the system $A\beta = \mathbf{0}$, we found the null space vectors. These vectors provide the exact coefficients for the linear equations that describe the relationships between the variables that hold true for all your samples.

Why Does This Matter for Machine Learning?

Okay, so we can represent data in matrices, find out how many variables are truly independent (rank), and even get the exact equations linking dependent variables (null space/nullity). Why is this a big deal for machine learning?

Well, a lot of machine learning algorithms work by doing calculations on these data matrices. If you want to reduce the number of variables in your dataset to make things simpler or faster (that's called dimensionality reduction), you absolutely need to understand the concepts of independence and rank. Algorithms like Principal Component Analysis (PCA) rely on finding the most important, independent directions (or components) in your data, which is directly related to the rank. Knowing the relationships (from the null space) can also be useful; sometimes, algorithms need to be built in a way that respects these inherent links in the data.

So, getting a solid grasp of these matrix ideas, rank, and null space is really your entry ticket to understanding how many machine learning techniques actually work under the hood.

Wrapping It Up

To sum it all up, linear algebra, especially working with matrices, is super important in data science. Matrices are our standard way to store data, with rows for each sample and columns for each variable. The rank of a matrix quantifies the number of independent variables, telling us how much unique information is present. And if there are dependencies, the null space and its size, the nullity, help us find the actual linear equations that describe the relationships between those variables. These aren't just abstract math ideas; they are practical tools for understanding your data better and are absolutely essential for getting into machine learning.

Solving for X, Part 2: Tackling Tougher Linear Equations in Data Science

Dinesh Kumar Sarangapani — Sun, 11 Oct 2026 07:23:52 +0000

Welcome back to our linear algebra for data science series! In Part 1 on solving linear equations, we set up systems of linear equations using matrices ($AX = B$) and covered the square ($M = N$) and overdetermined ($M > N$) cases with least squares.

Turning Equations into Matrix Problems: $AX=B$

Generally, when you have a set of linear equations, you can write them in a neat matrix form that looks like this:

$$AX = B$$

Let's break down what's in this equation based on what we learned before:

$A$ is a matrix. If you have $M$ equations and $N$ variables, $A$ is typically an $M \times N$ matrix. As we saw, $M$ is the number of rows (equations in this case) and $N$ is the number of columns (variables).
$X$ is a column vector containing the variables you're trying to solve for. Since $A$ has $N$ columns (variables), $X$ needs to have $N$ rows, so it's an $N \times 1$ matrix (a column vector).
$B$ is a column vector containing the constants from the right-hand side of your equations. For the matrix multiplication to work out, $B$ needs to have the same number of rows as $A$, so it's an $M \times 1$ matrix.

So, when you write $AX=B$, you're really representing a system of $M$ linear equations involving $N$ variables.

Now, depending on how $M$ (the number of equations) stacks up against $N$ (the number of variables), things can play out in three main ways:

$M = N$: The number of equations is exactly the same as the number of variables. This is often the "nicest" case to solve.
$M > N$: You have more equations than variables. Think of it as having too many rules for your variables to follow. Usually, this means there's no single perfect solution that satisfies all the equations.
$M < N$: You have fewer equations than variables. This means you have more variables than you strictly need for the given equations. In this scenario, you'll usually find that there are many, many possible solutions.

We're going to look at these cases, and then see how a cool idea called the pseudo-inverse can bring them all together.

A Quick Refresher on Rank

For an $M \times N$ matrix, the maximum possible rank it can have is the smaller of $M$ and $N$. If $M < N$, the max rank is $M$. If $N < M$, the max rank is $N$.

Case 1: When Equations Equal Variables ($M=N$)

This is the case where your matrix $A$ is square ($M \times N$ and $M=N$).

If $A$ is Full Rank:
"Full rank" here means the rank of the matrix is equal to $M$ (or $N$, since they're the same). What does this mean? It means all your equations on the left-hand side are independent. You can't create any one equation by combining the others.
In this happy case, there is a unique solution to $AX=B$. You might remember from algebra that you can solve this by finding the inverse of $A$, written as $A^{-1}$. The solution is simply:
$$X = A^{-1}B$$
You can find $A^{-1}$ if the determinant of $A$ is not zero. This is the standard, straightforward scenario.
If $A$ is Not Full Rank:
This means the rank of $A$ is less than $M$. In this situation, some of your equations on the left-hand side are actually linear combinations of others – they are linearly dependent.
When this happens, depending on what the values are on the right-hand side (in the $B$ vector), you get two possibilities:
1. Consistent System: If the dependencies on the left-hand side match up exactly with the dependencies on the right-hand side (the $B$ vector values), the equations are consistent. But because they are dependent, you don't have enough independent equations to pin down a single solution. This leads to infinite solutions.
2. Inconsistent System: If the dependencies on the left-hand side don't match up with the $B$ vector values, the equations are inconsistent. They contradict each other, and there is no solution that can satisfy all of them.

Let's look at a couple of examples for this $M=N$ case:

Example 1: Full Rank (Unique Solution)

Consider the system of equations:
$x_1 + 3x_2 = 7$
$2x_1 + 4x_2 = 10$

In matrix form $AX=B$, this is:
$$
\begin{bmatrix}
1 & 3 \
2 & 4
\end{bmatrix}
\begin{bmatrix}
x_1 \
x_2

\end{bmatrix}

You can solve this easily in software too. In Python, you could define A and B using NumPy and use the numpy.linalg.solve() command or compute the inverse and multiply.

Example 2: Not Full Rank, Consistent (Infinite Solutions)

Consider the system:
$x_1 + 2x_2 = 5$
$2x_1 + 4x_2 = 10$

In matrix form $AX=B$:
$$
\begin{bmatrix}
1 & 2 \
2 & 4
\end{bmatrix}
\begin{bmatrix}
x_1 \
x_2

\end{bmatrix}

If $x_2 = 0$, $x_1 = 5$. Solution: $(5, 0)$.
If $x_2 = 1$, $x_1 = 3$. Solution: $(3, 1)$.
If $x_2 = 2.5$, $x_1 = 0$. Solution: $(0, 2.5)$. Since you can choose any value for $x_2$, there are infinite solutions to this system.

Example 3: Not Full Rank, Inconsistent (No Solution)

Consider the system:
$x_1 + 2x_2 = 5$
$2x_1 + 4x_2 = 9$

In matrix form $AX=B$:
$$
\begin{bmatrix}
1 & 2 \
2 & 4
\end{bmatrix}
\begin{bmatrix}
x_1 \
x_2

\end{bmatrix}

So, for the $M=N$ case (square matrix A): if A is full rank, unique solution; if A is not full rank, check consistency – if consistent, infinite solutions; if inconsistent, no solution.

Case 2: More Equations Than Variables ($M > N$)

Now let's look at the case where you have more equations than variables. Like trying to find two numbers ($x_1, x_2$) that satisfy five different conditions.

$$AX = B \quad (\text{where M > N})$$

$$\text{Minimize } (AX - B)^T (AX - B)$$

$$A^T A X = A^T B$$

Where $A^T$ is the transpose of matrix $A$ (you flip its rows and columns).

Now, if the matrix $A^T A$ is invertible (which happens if the columns of the original matrix $A$ are linearly independent – related to the rank!), you can solve for $X$:

$$X = (A^T A)^{-1} A^T B$$

This optimization perspective gives us a way to find a meaningful "solution" even when we have more equations than variables, a common situation in data fitting.

Let's look back at our first $M>N$ example and see how this least squares solution plays out.

Example system of 3 equations with 2 variables:
$x_1 = 1$
$2x_1 = -0.5$
$3x_1 + x_2 = 5$

In matrix form $AX=B$:
$$
A = \begin{bmatrix} 1 & 0 \ 2 & 0 \ 3 & 1 \end{bmatrix}, \quad X = \begin{bmatrix} x_1 \ x_2 \end{bmatrix}, \quad B = \begin{bmatrix} 1 \ -0.5 \ 5 \end{bmatrix}
$$
$M=3, N=2$.

Applying the formula $X = (A^T A)^{-1} A^T B$, as we calculated earlier, gave us:
$$
X = \begin{bmatrix} 0 \ 5 \end{bmatrix}
$$
So the least squares solution is $x_1=0, x_2=5$. As we saw, this didn't satisfy the first two equations perfectly ($0 \neq 1$ and $2 \times 0 = 0 \neq -0.5$), but it made the third equation exact ($3 \times 0 + 5 = 5$) and minimized the total squared error across all three.

If we took the second $M>N$ example (where a perfect solution existed):
$x_1 = 1$
$2x_1 = 2$
$3x_1 + x_2 = 5$
$$
A = \begin{bmatrix} 1 & 0 \ 2 & 0 \ 3 & 1 \end{bmatrix}, \quad X = \begin{bmatrix} x_1 \ x_2 \end{bmatrix}, \quad B = \begin{bmatrix} 1 \ 2 \ 5 \end{bmatrix}
$$
Applying the same formula $X = (A^T A)^{-1} A^T B$, we got:
$$
X = \begin{bmatrix} 1 \ 2 \end{bmatrix}
$$
This matches the perfect solution $(x_1=1, x_2=2)$ that satisfies all three equations, because in this case, the minimum sum of squared errors is zero.

So, the least squares solution $X = (A^T A)^{-1} A^T B$ is a powerful way to get the "best fit" solution when you have more equations than variables, and it finds the exact solution if one exists. This formula works as long as the columns of $A$ are linearly independent (which makes $A^T A$ invertible).

Case 3: More Variables Than Equations ($M < N$)

Now for the last case: fewer equations than variables. Imagine having one equation, like $x_1 + x_2 + x_3 = 10$, and trying to find values for $x_1, x_2, x_3$. You can probably already see you have lots of options!

$$AX = B \quad (\text{where M < N})$$

When you have more variables than independent equations, you'll have "free" variables. For example, with 2 equations and 3 variables, you can often pick any value for one variable (say, $x_3$), plug it into the equations, and then you're left with 2 equations and 2 variables to solve for $x_1$ and $x_2$. Since you could pick any value for $x_3$, you end up with infinite solutions.

Since there are endless possibilities for $X$ that satisfy $AX=B$, how do you pick just one solution that makes the most sense for your problem? The equations themselves don't give you enough information to choose. You need some other rule or criteria.

Similar to the $M>N$ case, we can use an optimization idea! This time, instead of minimizing the error in the equations (since many solutions make the error zero!), we need a different objective. A common and often useful criterion is to find the solution $X$ that has the minimum "size" or "length". This is called finding the minimum norm solution.

We measure the "size" or "norm" of the solution vector $X$ using its length, specifically, minimizing the squared length $||X||^2 = X^T X$. Why minimize $X^T X$? Think of it as finding the solution vector $X$ that is closest to the origin $(0, 0, ... 0)$ in your variable space. From an engineering or modeling perspective, finding a solution where the variables have smaller values might be desirable in some contexts (like keeping design parameters small).

So, for the $M<N$ case, the optimization problem is:

$$\text{Minimize } \frac{1}{2} X^T X \quad \text{Subject to } AX = B$$

The $\frac{1}{2}$ is just there to make the math cleaner when you solve it using calculus. The important part is "Subject to $AX=B$." This is a constrained optimization problem – you're minimizing an objective function ($X^T X$) while making sure your solution still satisfies the original equations $AX=B$. This is different from the $M>N$ case where we had unconstrained optimization (just minimize the error without worrying about satisfying the equations perfectly).

Solving constrained optimization problems involves techniques like using Lagrangian functions. While the detailed steps for solving this are part of optimization theory (which often comes after linear algebra, but uses a lot of it!), the resulting formula for the minimum norm solution, assuming the rows of $A$ are linearly independent (making $A A^T$ invertible), is:

$$X = A^T (A A^T)^{-1} B$$

This formula gives you the unique solution that satisfies $AX=B$ and has the minimum possible squared length $||X||^2$.

Let's try an example for this $M<N$ case.
Consider the system of 2 equations with 3 variables:
$x_1 + 2x_2 + 3x_3 = 2$
$x_3 = 1$

In matrix form $AX=B$:
$$
A = \begin{bmatrix} 1 & 2 & 3 \ 0 & 0 & 1 \end{bmatrix}, \quad X = \begin{bmatrix} x_1 \ x_2 \ x_3 \end{bmatrix}, \quad B = \begin{bmatrix} 2 \ 1 \end{bmatrix}
$$
$M=2, N=3$. $M < N$.

We know there are infinite solutions here (like $(-1, 0, 1)$, $(-3, 1, 1)$, etc.). Let's use the minimum norm formula $X = A^T (A A^T)^{-1} B$.

First, find $A^T$:
$A^T = \begin{bmatrix} 1 & 0 \ 2 & 0 \ 3 & 1 \end{bmatrix}$

Next, calculate $A A^T$:
$A A^T = \begin{bmatrix} 1 & 2 & 3 \ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 \ 2 & 0 \ 3 & 1 \end{bmatrix} = \begin{bmatrix} 14 & 3 \ 3 & 1 \end{bmatrix}$

This matrix has a determinant of 5, so it's invertible. $(A A^T)^{-1} = \begin{bmatrix} 0.2 & -0.6 \ -0.6 & 2.8 \end{bmatrix}$.

Finally, calculate $X = A^T (A A^T)^{-1} B$:
$X = \begin{bmatrix} 1 & 0 \ 2 & 0 \ 3 & 1 \end{bmatrix} \begin{bmatrix} 0.2 & -0.6 \ -0.6 & 2.8 \end{bmatrix} \begin{bmatrix} 2 \ 1 \end{bmatrix}$

As we calculated before, this gives us:
$$
X = \begin{bmatrix} -0.2 \ -0.4 \ 1 \end{bmatrix}
$$

The minimum norm solution is $x_1 = -0.2$, $x_2 = -0.4$, and $x_3 = 1$. This solution satisfies the original equations, and among all the infinite solutions that work, this vector $[-0.2, -0.4, 1]^T$ is the one closest to the origin $(0,0,0)$.

So, the formula $X = A^T (A A^T)^{-1} B$ gives us the minimum norm solution when you have more variables than equations, provided the rows of $A$ are linearly independent (making $A A^T$ invertible).

One Solution to Rule Them All: The Pseudo-inverse!

We've looked at the three cases for $AX=B$:

$M=N$: Unique solution ($A^{-1}B$) if full rank; infinite or no solutions if not full rank.
$M>N$: Generally no exact solution, find the least squares solution ($X = (A^T A)^{-1} A^T B$) that minimizes errors.
$M<N$: Infinite exact solutions, find the minimum norm solution ($X = A^T (A A^T)^{-1} B$) that is closest to the origin.

This is a lot to keep track of! Wouldn't it be awesome if there was one single formula that just gave you the right kind of solution (unique exact, least squares, or minimum norm) no matter the size or rank of $A$?

It turns out there is! This is where the concept of the Moore-Penrose pseudo-inverse (often written as $A^+$) comes in. It generalizes the idea of the matrix inverse $A^{-1}$.

The magical formula for solving $AX=B$ in all cases is simply:

$$X = A^+ B$$

Where $A^+$ is the pseudo-inverse of $A$.

How is this magical?

If $M=N$ and $A$ is full rank, $A^+$ is exactly the same as $A^{-1}$. So you get $X = A^{-1}B$, the unique exact solution.
If $M>N$ and the columns of $A$ are independent (making $A^T A$ invertible), $A^+$ is equal to $(A^T A)^{-1} A^T$. So you get $X = (A^T A)^{-1} A^T B$, the least squares solution.
If $M<N$ and the rows of $A$ are independent (making $A A^T$ invertible), $A^+$ is equal to $A^T (A A^T)^{-1}$. So you get $X = A^T (A A^T)^{-1} B$, the minimum norm solution.

And what about the cases where $A$ is not full rank (determinant is zero for M=N, columns/rows are dependent)? The pseudo-inverse still works! In those rank-deficient cases:

If the system is consistent ($M=N$ not full rank, or $M<N$ rank deficient but consistent), $A^+ B$ gives you the minimum norm solution among the infinite exact solutions.
If the system is inconsistent ($M=N$ not full rank, or $M>N$ rank deficient and inconsistent), $A^+ B$ gives you the least squares solution that minimizes the error $||AX-B||^2$.

So, the pseudo-inverse $A^+$ gives you:

The unique exact solution if it exists.
The minimum norm exact solution if there are infinite exact solutions.
The minimum norm least squares solution if there are no exact solutions (i.e., the least squares solution with the smallest norm, although in the M>N case this is usually the only least squares solution).

How do you calculate this pseudo-inverse $A^+$? One common way is using something called Singular Value Decomposition (SVD), which is a really powerful technique in linear algebra. But for using it to solve $AX=B$, you often don't need to know the details of SVD right away.

Many software packages can compute the pseudo-inverse directly. In Python, you can use NumPy's linalg.pinv() function.

Here are examples in Python using numpy.linalg.pinv() on the matrices we just looked at:

import numpy as np

# Example 1 (M > N)
A1 = np.array([[1, 0], [2, 0], [3, 1]])
B1 = np.array([[1], [-0.5], [5]])

A1_plus = np.linalg.pinv(A1)
X1 = A1_plus @ B1 # Using the matrix multiplication operator @

print("Solution for Example 1 (M>N):")
print(X1)
# Output should be close to:
# [[0.]
#  [5.]]

print("\n---")

# Example 2 (M < N)
A2 = np.array([[1, 2, 3], [0, 0, 1]])
B2 = np.array([[2], [1]])

A2_plus = np.linalg.pinv(A2)
X2 = A2_plus @ B2 # Using the matrix multiplication operator @

print("Solution for Example 2 (M<N):")
print(X2)
# Output should be close to:
# [[-0.2]
#  [-0.4]
#  [ 1. ]]

Section 1: The dataset and the problem

Dinesh Kumar Sarangapani — Sun, 11 Oct 2026 07:23:50 +0000

Section 1: The dataset and the problem

Everything later in the playlist, GANs, VAEs, diffusion, transformers, state-space models, and preference tuning, starts from the same setup. I want this part solid before I touch a model.

The setup I am carrying:

Data $D = {x_1, x_2, \dots, x_n} \overset{\text{iid}}{\sim} P_x$ (unknown), with $x_i \in \mathbb{R}^d$, where $d$ is the dimensionality of the data.
Example: $x_i$ is an image, $400 \times 400 \times 3$, so $d = 480{,}000$.
Goal: estimate $P_x$ and learn to sample from it.

Intuition

Somewhere there is a hidden recipe, a probability distribution, that produced every cat photo, every English sentence, every speech clip. I never get the recipe. I get a pile of examples it produced. Generative modeling is using that pile to reconstruct the recipe well enough to cook up new examples from the same kitchen.

An analogy I can check

Think of a production web service whose request process I cannot inspect. All I have is an access log of 1,000 requests. I want a load-testing tool that produces synthetic traffic indistinguishable from real users: same mix of endpoints, same payload sizes, same timing.

Load-testing analogy	Notation
The real, unknown user behaviour	$P_x$, the true data distribution
One logged request	$x_i$, one data point
The fields in a log line	the $d$ coordinates of $x_i \in \mathbb{R}^d$
The access log	dataset $D$
The traffic generator	the generative model $P_\theta$
Synthetic requests it emits	samples from $P_\theta$

Two consequences. I do not need to write the user-behaviour formula down. I need a generator whose output is statistically indistinguishable. That is why a GAN never writes $P_x$ down at all. And replaying the log is not the goal. I want new requests, not copies.

Visual

A hidden 2-D distribution plays the role of $P_x$. At first I only see the dots, the dataset. Reveal $P_x$ to see what is normally invisible, and drag $n$ to see how more data reveals the shape.

Each "Draw a new dataset" gives a different $D$ and the same green $P_x$ underneath. The dataset is random. The distribution is the fixed thing I am after. With $n = 5$ the shape is a guess. With $n = 1000$ the two clumps are obvious.

Detours I needed before the formula

Random variable, and random vector. A random variable is a quantity whose value is decided by chance, like a die roll $X \in {1,\dots,6}$. A random vector is the same idea with several numbers at once, such as height and weight of a randomly chosen person, $X \in \mathbb{R}^2$. Capital $X$ means the random thing in general. Lowercase $x_i$ means one specific value that actually came out.

Distribution and density. $P_x$ tells me how likely each possible value is. For a die it is a table: each face has probability 1/6. For continuous values, such as pixel intensities, it is a density $p_x(x)$, a heat map over space. High density means values land there often. In the chart above, the green shading is that density.

Two facts I will need constantly:

$$
p_x(x) \ge 0 \qquad \int_{\mathbb{R}^d} p_x(x)\, dx = 1
$$

The second says all probability mass integrates to 1. I will use $P_x$ loosely for both the distribution and its density. That is standard, and I will be explicit when the difference matters.

iid, independent and identically distributed.

Identically distributed: every $x_i$ comes from the same $P_x$. Every log line came from the same user population, not some from production and some from a test environment.
Independent: knowing $x_3$ tells me nothing extra about $x_7$. The first roll of a die does not influence the second.

In notation, $x_i \perp x_j$ and $x_i \sim P_x$.

The formula

$$
D = {x_1, x_2, \dots, x_n} \overset{\text{iid}}{\sim} P_x, \qquad x_i \in \mathbb{R}^d
$$

Read aloud: the dataset $D$ is a collection of $n$ points. Each point is an independent draw from the same unknown distribution $P_x$. Each point is a vector of $d$ real numbers.

Symbol	What it is	Type / shape	Role
$D$	the dataset	a set of $n$ vectors	the only thing I actually observe
$n$	number of samples	integer	more data, a better picture of $P_x$
$x_i$	one data point	vector in $\mathbb{R}^d$	an image, a sentence embedding, a signal
$d$	dimensionality	integer	how many numbers describe one point
$\sim$	"is drawn from"	relation	links data to its source
iid	independent and identically distributed	assumption	makes learning from samples legitimate
$P_x$	true data distribution	distribution on $\mathbb{R}^d$	the hidden target, unknown

Why "unknown" is the whole problem. If I knew $P_x$, I would sample from it and stop. Every model in the playlist is a different trick for working with $P_x$ through samples alone.

Why iid matters. It licenses replacing an expectation, an average over $P_x$, with an average over the dataset. That is the law of large numbers:

$$
\frac{1}{n}\sum_{i=1}^n h(x_i) \;\xrightarrow{\,n\to\infty\,}\; \mathbb{E}_{x\sim P_x}[h(x)]
$$

That line is the engine behind every training loss later. The chart shows a small version: the sample mean settles as $n$ grows.

Why $d$ is so large. A 400×400 RGB image is one point in $\mathbb{R}^{480{,}000}$, one coordinate per pixel per colour channel. Real images occupy a thin region of that space. Random pixel vectors look like television static. This is the manifold hypothesis, and it comes back when naive GAN training falls over.

Two jobs hiding in one goal

Given $D$, estimate $P_x$ and learn to sample from it. Those are different jobs, and different models emphasise different ones.

Job	Meaning	Who emphasises it
Density estimation	answer "how likely is this $x$?"	autoregressive models, VAEs through the ELBO
Sampling	produce a new $x \sim P_x$	GANs, which never write a density down, and diffusion

A GAN can sample well and still be unable to tell me $p(x)$ for a given image. I want to remember that split.

A worked example small enough to do by hand

Let $d = 2$ and $n = 3$. Each point is hours slept and coffees drunk for a random person:

$$
D = {x_1, x_2, x_3} = {(7, 1),\ (6, 2),\ (8, 0)}
$$

Each $x_i \in \mathbb{R}^2$, so $d = 2$. Here $x_2 = (6, 2)$ means 6 hours of sleep and 2 coffees. iid means three different people, surveyed separately, from the same population.

I want $\mathbb{E}[\text{coffees}]$ under the unknown $P_x$. I cannot integrate against a distribution I do not have. iid lets me estimate it from the sample: $\tfrac{1}{3}(1 + 2 + 0) = 1$.

The generative goal is a machine that outputs new pairs such as $(7.5, 0.5)$: plausible, not copied from $D$, and following the same pattern. More sleep tends to go with less coffee. That dependency between coordinates is what $P_x$ captures, and a per-column average does not.

The same picture with $d = 480{,}000$ and $n$ in the millions is image generation.

Where the rest of the playlist sits, as I understand it now

Stretch	Model	How it attacks "learn $P_x$ from $D$"
Early	GAN / divergence minimisation	push $z \sim \mathcal{N}(0, I)$ through $g_\theta$ and shrink an f-divergence to $P_x$
Next	VAE	latent-variable $P_\theta(x) = \int P_\theta(x, z)\,dz$, maximise the ELBO
Then	WGAN	the same game, with the Wasserstein distance
Then	DDPM	add noise to $x_0$, learn to reverse it
Then	Autoregressive / transformer	factorise $P_\theta(x) = \prod_t P_\theta(x_t \mid x_{<t})$
Then	SSM / Mamba	another sequence model
Late	RLHF / DPO	move the learned distribution toward preferences

The next section turns the goal into a recipe: pick a parametric family $P_\theta$, define a divergence, and minimise it using only samples.

Questions I want to be able to answer before I move on:

If a dataset mixes 5,000 phone photos and 5,000 medical scans, which part of iid breaks, and why does that hurt?
What is $d$ for a 28×28 grayscale digit? For a 64×64 RGB image?
Once I have $D$, do I know $P_x$ exactly? What changes as $n \to \infty$?

Next: Section 2: The general principle of generative models.

Section 2: The general principle of generative models

Dinesh Kumar Sarangapani — Sun, 11 Oct 2026 07:18:47 +0000

Previously: Section 1: The dataset and the problem.

Section 2: The general principle of generative models

Section 1 left me with a vague goal: estimate $P_x$ and learn to sample from it. This is the recipe that turns that goal into three moves. Every later model in the lectures is this recipe with different choices plugged in.

The statement I am carrying:

(i) Assume a parametric family on $P_x$, denoted $P_\theta$, represented using deep neural networks.
(ii) Define and estimate a divergence between $P_x$ and $P_\theta$.
(iii) Solve an optimisation problem over the parameters of $P_\theta$ to minimise that divergence.

The example that makes it concrete: draw $z \sim \mathcal{N}(0, I)$, pass it through a network $g_\theta : \mathcal{Z} \to \mathcal{X}$, and call the distribution of $x = g_\theta(z)$ by the name $P_\theta$. Then solve

$$
\theta^* = \arg\min_\theta \; \mathcal{D}(P_x \,|\, P_\theta)
$$

After that, sampling $z \sim \mathcal{N}(0, I)$ and computing $g_{\theta^}(z)$ gives samples from $P_{\theta^}$, which is as close to $P_x$ as this family could get.

Four prerequisites sit under that formula: the standard Gaussian $\mathcal{N}(0, I)$, what happens when a random variable is pushed through a function, what a divergence is, and the $\arg\min$ notation. Each detour is only as long as the step that needs it.

Intuition

I cannot see $P_x$. I build a machine with knobs, $\theta$, that turns plain random noise into data-shaped outputs. I keep turning the knobs until the machine's output distribution is as close as I can get to the real one. The closeness is a divergence. The turning is optimisation.

The load test, continued

The traffic generator from Section 1 takes random seeds and outputs synthetic requests. It has config knobs: endpoint weights, payload-size parameters, and so on.

Load-testing analogy	Notation
Random seed fed into the generator	$z \sim \mathcal{N}(0, I)$
The generator code	$g_\theta$, a neural network
Its config knobs	$\theta$, the network weights
The distribution of traffic it produces	$P_\theta$
A score of how different synthetic traffic is from real logs	the divergence $\mathcal{D}(P_x \,\|\, P_\theta)$
Tuning knobs until that score is near zero	$\theta^* = \arg\min_\theta \mathcal{D}$

The hard step is the score. I cannot see the true user behaviour. I only have the log. That is the difficulty the rest of the lectures keep returning to.

Visual

One dimension is enough to see the recipe. The noise is $z \sim \mathcal{N}(0, 1)$. The network is the simplest one I can write, $g_\theta(z) = a z + b$, so $\theta = (a, b)$. The green curve is the target. The blue curve is what this generator can produce.

Three things I want in front of me while I move the sliders:

On the single bump, $a = 0.6$ and $b = 2$ drive the divergence to 0. The target is a bell centred at 2 with spread 0.6, and $x = 0.6z + 2$ is exactly that bell. The machine reproduces $P_x$.
On the two bumps, the divergence never reaches 0, however I set $(a, b)$. The closest blue curve is a wide bell laid over both humps. A linear $g_\theta$ can only shift and stretch one bell. It cannot make two. That is why the principle asks for a deep network: $g_\theta$ has to be flexible enough that $P_\theta$ can get near $P_x$.
The chart can draw the green curve, and it can compute the divergence, because I built the target in. In the real problem I do not get that picture. I only get samples. That is the first of the four open questions below.

Detours

The standard Gaussian $\mathcal{N}(0, I)$. $\mathcal{N}(\mu, \sigma^2)$ is the bell centred at $\mu$ with spread $\sigma$. Standard means $\mu = 0$ and $\sigma = 1$. In $k$ dimensions, $\mathcal{N}(0, I)$ is $k$ independent standard bells, one per coordinate. $I$ is the $k \times k$ identity covariance: each coordinate has variance 1, and no two coordinates are correlated.

It is the input noise because every library can sample it, and it has no structure of its own. Any structure in the output has to come from $g_\theta$. So $z \sim \mathcal{N}(0, I)$ is a random seed: easy to draw, carrying nothing about images.

Pushing a random variable through a function. If $z$ is random and $x = g(z)$, then $x$ is random too, with a new distribution determined by $g$.

The simplest case: if $z \sim \mathcal{N}(0, 1)$, then $x = az + b \sim \mathcal{N}(b, a^2)$. Multiplying by $a$ stretches the spread. Adding $b$ slides the centre. That is what the sliders above are doing.

A deep, nonlinear $g$ can bend the output into more than one hump, or into a thin set such as the distribution of faces. The output distribution is different from the distribution of $z$, and it depends on $g_\theta$. I name that output distribution $P_\theta$. For a deep network I usually cannot write a formula for it. I only know how to sample it: draw $z$, run $g_\theta$.

Divergence, which is weaker than a distance. $\mathcal{D}(P \,|\, Q)$ scores how different two distributions are. The two properties I need:

$$
\mathcal{D}(P_x \,|\, P_\theta) \ge 0, \qquad \mathcal{D}(P_x \,|\, P_\theta) = 0 \iff P_x = P_\theta
$$

It need not be symmetric. $\mathcal{D}(P \,|\, Q)$ may differ from $\mathcal{D}(Q \,|\, P)$. That is why the notation uses $|$ rather than a comma. The KL divergence in the chart is one example. The next note builds the family they all belong to: Section 3a: f-divergences.

Those two properties are what make "minimise the divergence" a sensible goal. The smallest possible value is 0, and it is reached only when the model matches the data.

$\arg\min$. $\min_\theta f(\theta)$ is the smallest value of $f$. $\arg\min_\theta f(\theta)$ is the input $\theta$ that achieves it. For $f(\theta) = (\theta - 3)^2 + 1$, the min is 1 and the argmin is 3. I want the knob settings, not the score, so the principle uses argmin.

The formula

$$
\theta^* = \arg\min_\theta \; \mathcal{D}\big(P_x \,|\, P_\theta\big), \qquad P_\theta := \text{distribution of } g_\theta(z),\; z \sim \mathcal{N}(0, I)
$$

Read aloud: $\theta^*$ is the setting of the network weights that makes the divergence between the true data distribution and the distribution of the network's outputs as small as possible.

Symbol	What it is	Type / shape	Role
$z$	input noise	vector in $\mathbb{R}^k$, usually $k \ll d$	the random seed
$\mathcal{N}(0, I)$	standard Gaussian	distribution on $\mathbb{R}^k$	easy-to-sample source of randomness
$g_\theta$	generator	function $\mathbb{R}^k \to \mathbb{R}^d$, a neural net	reshapes noise into data
$\theta$	generator parameters	a vector of weights	the knobs
$P_\theta$	model distribution	distribution on $\mathbb{R}^d$, implicit	what the generator produces
$P_x$	true distribution	distribution on $\mathbb{R}^d$, unknown	the target
$\mathcal{D}(\cdot \,\|\, \cdot)$	divergence	two distributions to a scalar $\ge 0$	the "how far apart" score
$\theta^*$	optimal parameters	same shape as $\theta$	the trained generator

Once training is done, sampling is: draw a fresh $z \sim \mathcal{N}(0, I)$, compute $g_{\theta^}(z)$. That is a sample from $P_{\theta^} \approx P_x$. I have learned to sample from $P_x$ without writing it down.

A worked example small enough to do by hand

Take $k = d = 1$ and $g_\theta(z) = 2z + 3$, so $\theta = (a, b) = (2, 3)$.

Draw three noise values: $z = -1, 0, 1$.
Outputs: $g_\theta(-1) = 1$, $g_\theta(0) = 3$, $g_\theta(1) = 5$.
By the pushforward above, all outputs come from $P_\theta = \mathcal{N}(3, 2^2)$. Centre 3, spread 2, which matches these samples.

For the divergence, a two-outcome world is enough to compute by hand. Let $P_x$ be a fair coin, $(0.5, 0.5)$, and use KL, $\sum_i P_x(i) \ln \frac{P_x(i)}{P_\theta(i)}$. KL is formally the next part of the lectures. I only need it here to see numbers move.

If the model says $P_\theta = (0.9, 0.1)$: $0.5 \ln\frac{0.5}{0.9} + 0.5 \ln\frac{0.5}{0.1} = 0.5(-0.588) + 0.5(1.609) \approx 0.511$.
If the model says $P_\theta = (0.6, 0.4)$: $0.5 \ln\frac{0.5}{0.6} + 0.5 \ln\frac{0.5}{0.4} = 0.5(-0.182) + 0.5(0.223) \approx 0.020$.
If the model says $P_\theta = (0.5, 0.5)$: both terms are $0.5 \ln 1 = 0$, so the divergence is exactly 0.

As the model gets closer to the truth, the score falls toward 0, and it hits 0 only at a perfect match. Optimisation is moving $\theta$ in the direction that lowers this number.

Four questions this recipe does not answer

The rest of the lectures is the attempt to answer these.

Question	Why it is hard	Where I expect the answer
How do I compute the divergence without formulas for $P_x$ or $P_\theta$?	I only have samples of both	Write the divergence as expectations, then estimate them with sample averages
Which divergence?	Different choices behave differently. The wide bell over two bumps is one failure mode	f-divergences first, Wasserstein later
How do I choose $g_\theta$, and so $P_\theta$?	Flexible enough, and still trainable	GANs, VAEs, diffusion, transformers
How do I solve the optimisation?	Millions of parameters, often a min-max game	Gradient descent, then its adversarial variants

Where the later models sit

Model	Choice of $P_\theta$	Divergence	Optimisation
GAN	$g_\theta(z)$, implicit	Jensen–Shannon, an f-divergence	min-max with a discriminator
WGAN	$g_\theta(z)$, implicit	Wasserstein	min-max with a 1-Lipschitz critic
VAE	latent model $\int P_\theta(x \mid z)\, p(z)\, dz$	KL, through the ELBO	maximise the ELBO
DDPM	reverse chain of Gaussians	KL, through the ELBO	regression on the added noise
Autoregressive / transformer	$\prod_t P_\theta(x_t \mid x_{<t})$	KL, which is maximum likelihood	cross-entropy

One link I want early: minimising $\mathrm{KL}(P_x \,|\, P_\theta)$ is the same as maximum likelihood, because $\arg\min \mathrm{KL}(P_x \,|\, P_\theta) = \arg\max \mathbb{E}[\log P_\theta(x)]$. That is why VAEs, diffusion models, and language models can all train on log-likelihood losses and still be this same recipe.

Questions I want to be able to answer:

Why can no setting of $(a, b)$ match the two-bump target, and what has to change about $g_\theta$?
After training, how do I produce 10 new samples, in one line?
If $\mathcal{D}(P_x \,|\, P_\theta) = 0$, what follows? What if the divergence is small but not zero?

Next: Section 3a: f-divergences.

Section 3a: f-divergences

Dinesh Kumar Sarangapani — Sun, 11 Oct 2026 07:18:44 +0000

Previously: Section 2: The general principle of generative models.

Section 3a: f-divergences

Section 2 left a slot for a divergence. This note fills it with a family. One convex function $f$ picks the score. Section 3b, not written yet, is the convex conjugate: the rewrite that estimates $D_f$ from samples, which is the bridge into GANs.

The statement I am carrying:

Given two distributions with densities $P_x$ and $P_\theta$,

$$
D_f(P_x \,|\, P_\theta) = \int_{\mathcal{X}} P_\theta(x)\, f!\left(\frac{P_x(x)}{P_\theta(x)}\right) dx
$$

where $f: \mathbb{R}+ \to \mathbb{R}$ is convex and lower-semicontinuous, with $f(1) = 0$. Then $D_f \ge 0$, and $D_f = 0$ if and only if $P_x = P\theta$.

Intuition

At each point $x$ I compare how much probability the data puts there with how much the model puts there. The ratio $P_x / P_\theta$ is 1 where they agree and something else where they do not. $f$ turns that local mismatch into a penalty. The integral averages the penalties. A different $f$ punishes a different kind of mismatch, so one template gives a family.

The load test, by endpoint

I compare synthetic traffic with the real access log, endpoint by endpoint.

$$
\text{ratio} = \frac{\text{real share of traffic}}{\text{synthetic share of traffic}}
$$

A ratio of 1 means the generator gets that endpoint right. A ratio of 3 means real users hit it three times as often as the generator does. A ratio of 0.2 means the generator hammers an endpoint real users barely touch.

The single score depends on which failure I care about.

Missing real traffic patterns. The penalty grows when the ratio is large. That is forward KL.
Fake traffic that real users would never send. The penalty grows when the ratio is near 0. That is reverse KL.
A balanced score that cannot run off to infinity. That is Jensen–Shannon or total variation.

$f$ is the penalty. The integral is the average over endpoints.

Visual

$P_x$ is a standard bell centred at 0. $P_\theta$ is the same bell shifted to $\mu$. Pick an $f$. The top curve is the penalty, and it passes through $(1, 0)$. The coral curve is the integrand $P_\theta(x)\, f(P_x(x)/P_\theta(x))$. The four numbers are the integrals.

Three things I want in front of me while I move $\mu$:

At $\mu = 0$ every divergence is 0. Matching distributions means zero penalty, for every $f$ with $f(1) = 0$.
With forward KL, the coral curve dips below zero in places. A single point can contribute a negative amount. The total never does. The proof below is why.
Jensen–Shannon and total variation level off as $\mu$ grows. Both are bounded. The two KLs keep climbing. That saturation is the later reason a GAN, which aims at Jensen–Shannon, can stop learning when the data and the model do not overlap.

Detours

Density ratio. $u(x) = P_x(x) / P_\theta(x)$ asks, at $x$, how many times more likely real data is here than model data.

$u = 1$: agreement at this point.
$u > 1$: the model under-produces here.
$u < 1$: the model over-produces here.
$u = 0$: the model puts mass where the data never goes.

If $P_x = P_\theta$ everywhere, then $u(x) = 1$ everywhere. $f$ only ever sees this ratio.

Convexity. A function is convex when it is bowl-shaped: the straight line between any two points on its graph lies on or above the curve. For $\lambda \in [0, 1]$,

$$
\lambda f(u_1) + (1-\lambda) f(u_2) \;\ge\; f\big(\lambda u_1 + (1-\lambda) u_2\big)
$$

Averaging the outputs is at least as big as the output of the average. Check with $f(u) = u^2$, $u_1 = 0$, $u_2 = 2$, $\lambda = \tfrac{1}{2}$. The average of the outputs is $\tfrac{1}{2}(0 + 4) = 2$. The output of the average is $f(1) = 1$. And $2 \ge 1$.

Convexity is what forces $D_f \ge 0$.

Expectation as an integral. For a continuous density $p$, the expected value of $h(x)$ is a probability-weighted average:

$$
\mathbb{E}{x \sim p}[h(x)] = \int{\mathcal{X}} p(x)\, h(x)\, dx
$$

Jensen's inequality. For convex $f$ and a random variable $U$,

$$
\mathbb{E}[f(U)] \;\ge\; f(\mathbb{E}[U])
$$

This is the two-point definition extended to any average. On a bowl, the average of points on the surface lands above the value at the average, never below.

The formula

$$
D_f(P_x \,|\, P_\theta) = \int_{\mathcal{X}} P_\theta(x)\, f!\left(\frac{P_x(x)}{P_\theta(x)}\right) dx = \mathbb{E}{x \sim P\theta}!\left[f!\left(\frac{P_x(x)}{P_\theta(x)}\right)\right]
$$

Read aloud: the average penalty, over points drawn from the model, of how mismatched the two densities are at each point.

Symbol	What it is	Type / shape	Role
$P_x(x)$	true density at $x$	scalar $\ge 0$	the data's weight here
$P_\theta(x)$	model density at $x$	scalar $\ge 0$	the model's weight, and the weight in the average
$u = P_x/P_\theta$	density ratio	scalar $\ge 0$	local mismatch
$f$	generator of the divergence	convex, $f(1) = 0$	turns mismatch into a penalty
$\int \cdot\, dx$	integral over $\mathcal{X}$	operation	adds the penalties
$D_f$	the divergence	scalar $\ge 0$	the total score

Each condition on $f$ earns its place:

$f(1) = 0$. If the distributions match, $u = 1$ everywhere, so $D_f = \int P_\theta \cdot 0\, dx = 0$.
Convex. This is the $D_f \ge 0$ proof below.
Lower-semicontinuous. No sudden downward jumps. Section 3b needs it so the conjugate is well behaved.

The proof that $D_f \ge 0$

The lectures state the property. This is the short proof.

$$
\begin{aligned}
D_f(P_x \,|\, P_\theta) &= \mathbb{E}{x \sim P\theta}!\left[f!\left(\tfrac{P_x(x)}{P_\theta(x)}\right)\right] \
&\ge f!\left(\mathbb{E}{x \sim P\theta}!\left[\tfrac{P_x(x)}{P_\theta(x)}\right]\right) \
&= f!\left(\int P_\theta(x)\, \tfrac{P_x(x)}{P_\theta(x)}\, dx\right) \
&= f!\left(\int P_x(x)\, dx\right) \
&= f(1) = 0
\end{aligned}
$$

The first line is the expectation form. The second is Jensen. The third writes the expectation as an integral. $P_\theta$ cancels. A density integrates to 1, and $f(1) = 0$.

In words: the average ratio, weighted by the model, is exactly 1. A bowl-shaped penalty averaged around 1 cannot fall below its value at 1, which is 0. That is why the coral curve can go negative locally while the total stays non-negative.

One template, four choices of $f$

Name	$f(u)$	Resulting divergence	Behaviour
Forward KL	$u \log u$	$\displaystyle\int P_x \log\frac{P_x}{P_\theta}\, dx = D_{\mathrm{KL}}(P_x \,\|\, P_\theta)$	Punishes the model for missing real data. Mode-covering. This is the blurry blob in Section 2, and it is maximum likelihood.
Reverse KL	$-\log u$	$\displaystyle\int P_\theta \log\frac{P_\theta}{P_x}\, dx = D_{\mathrm{KL}}(P_\theta \,\|\, P_x)$	Punishes the model for generating where the data is not. Mode-seeking. It sits on one hump.
Jensen–Shannon	$\tfrac{1}{2}\big[u \log u - (u+1)\log\tfrac{u+1}{2}\big]$	symmetric, bounded by $\log 2$	What the original GAN minimises.
Total variation	$\tfrac{1}{2}\lvert u - 1 \rvert$	$\tfrac{1}{2}\int \lvert P_x - P_\theta \rvert\, dx$	Symmetric, bounded by 1.

Forward KL is the template with $f(u) = u \log u$, and the $P_\theta$ cancels:

$$
\int P_\theta \cdot \frac{P_x}{P_\theta} \log\frac{P_x}{P_\theta}\, dx = \int P_x \log\frac{P_x}{P_\theta}\, dx
$$

Reverse KL is the same template with $f(u) = -\log u$. Substituting it recovers $\int P_\theta \log(P_\theta / P_x)\, dx$. The Jensen–Shannon $f$ above satisfies $f(1) = 0$. A GAN writeup often uses $f(u) = u \log u - (u+1)\log(u+1)$, which differs from this one by a constant shift and is why that loss is described as similar to Jensen–Shannon rather than identical. Total variation is $\tfrac{1}{2}\lvert u - 1 \rvert$, which is the choice that turns the template into $\tfrac{1}{2}\int \lvert P_x - P_\theta \rvert$.

A worked example small enough to do by hand

Two outcomes. $P_x = (0.5, 0.5)$, a fair coin. $P_\theta = (0.8, 0.2)$, a biased model.

The ratios are $u_1 = 0.5 / 0.8 = 0.625$ (over-produced) and $u_2 = 0.5 / 0.2 = 2.5$ (under-produced). The template on a finite set is $D_f = \sum_i P_\theta(i)\, f(u_i)$.

Divergence	Computation	Value
Forward KL	$0.8(0.625 \ln 0.625) + 0.2(2.5 \ln 2.5) = 0.8(-0.294) + 0.2(2.291)$	$\approx 0.223$
Reverse KL	$0.8(-\ln 0.625) + 0.2(-\ln 2.5) = 0.8(0.470) + 0.2(-0.916)$	$\approx 0.193$
Jensen–Shannon	$0.8\, f(0.625) + 0.2\, f(2.5) = 0.8(0.0218) + 0.2(0.1660)$	$\approx 0.051$
Total variation	$0.8 \cdot \tfrac{1}{2}(0.375) + 0.2 \cdot \tfrac{1}{2}(1.5) = 0.15 + 0.15$	$0.300$

All four are positive, and they disagree on the size of the gap. Forward KL is not reverse KL, so the score is not symmetric. In the forward-KL row, outcome 1 contributes about $-0.235$ and outcome 2 about $+0.458$. Negative pieces, positive total. That is Jensen. If $P_\theta = (0.5, 0.5)$, every ratio is 1 and every row is 0.

The catch, which is Section 3b

Computing $D_f$ needs the values $P_x(x)$ and $P_\theta(x)$.

$P_x$ is unknown. Section 1 only gave me samples.
$P_\theta$ is implicit. Section 2 lets me sample $g_\theta(z)$, not evaluate a density.

So I cannot form the ratio $u(x)$. What I can form are averages over samples, by the law of large numbers. Section 3b has to rewrite $D_f$ as expectations under $P_x$ and under $P_\theta$, with no density values in the formula. The convex conjugate is the tool. The result is the GAN objective.

Where this sits in the lectures

This note answers "which divergence?" with a menu, not a single choice.
A GAN picks the Jensen–Shannon-like $f$.
VAEs, diffusion, and autoregressive models minimise forward KL, which is maximum likelihood.
Bounded scores such as Jensen–Shannon saturate when $P_x$ and $P_\theta$ do not overlap. Wasserstein, which is not an f-divergence, is the later answer to that.
The KL term that keeps an aligned model close to a reference model, $D_{\mathrm{KL}}(\pi_\theta \,|\, \pi_{\mathrm{ref}})$, is this same family.

Questions I want to be able to answer:

Why does $f(1) = 0$ have to hold? What breaks if $f(1) = 5$?
In the worked example, outcome 1 contributes a negative amount to forward KL. Why does that not break $D_f \ge 0$?
Why can I not plug the dataset into the $D_f$ formula and compute it directly?

Building Scalable Agentic AI Platforms: A Technical Deep Dive - Part 2

Dinesh Kumar Sarangapani — Sat, 10 Oct 2026 16:07:03 +0000

Before diving into Part 2, make sure to read Part 1 where we covered the fundamentals.

Module 1: Model Management using LLM Gateway

There are several models that are useful for specific purposes. Even though there are general-purpose models, we need to choose the best-performing model for our task. Deploying those models in various cloud providers and applying the security principles, tracing, cost controlling them is a Challenge.

After the gateway,

Key features:

Routing to right Model
Central logging for Compliance checks
Metrics collection for cost and usage

Building an Enterprise MCP Tool Broker: Turning OpenAPI into Agent Tools

Dinesh Kumar Sarangapani — Sat, 10 Oct 2026 14:24:32 +0000

AI agents are only as useful as the tools they can call. An agent without tools can only chat. An agent with tools can query databases, create tickets, inspect files, and trigger deployments.

In most enterprises, those capabilities already exist as REST APIs. Over the last decade, organizations built hundreds of internal microservices documented with OpenAPI.

When we started connecting agents to our internal systems, we faced a choice. Should we manually write dedicated tool wrappers for every API? Or could we build a broker that translates existing OpenAPI specifications into agent tools automatically?

We chose to build a centralized tool broker using the Model Context Protocol (MCP). Here is how the idea works, how it performed in production, and what to watch out for.

The Idea: A Dynamic Tool Broker

Writing dedicated tool servers for hundreds of microservices does not scale. Each engineering team would need to learn a new protocol, maintain extra code, and keep tool schemas manually synchronized with API updates.

The core idea was to build a single enterprise tool broker.

The broker sits between AI agents and internal APIs. To the agent, the broker looks like an MCP server offering a unified catalog of tools. Behind the scenes, the broker ingests existing OpenAPI specifications from registered services and dynamically translates each API operation into a typed, callable tool.

When an agent invokes a tool, the broker validates the caller's identity, translates the arguments into an HTTP request, calls the backend service, and returns the formatted result back to the model.

How It Worked Well

Zero Work for Service Teams: Existing microservices did not need code changes. As long as a team maintained an accurate OpenAPI specification, their service became agent-callable instantly.
Standardized Tool Discovery: Instead of teaching every agent about individual services, agents connect to one broker endpoint. The broker dynamically filters which tools an agent can see based on the caller's permissions and role.
Decoupled Architecture: Upstream microservices remain pure REST APIs. If the agent protocol changes or new client interfaces emerge, the translation logic lives in the broker rather than scattered across dozens of repositories.
Governed Access Controls: The broker enforces centralized authorization before executing any backend request. We can enforce least-privilege access, audit tool invocation logs, and disable high-risk actions without modifying the underlying microservices.

What to Watch Out For

Vague Endpoint Summaries Confuse Models: Large language models pick tools based on natural language descriptions. If an API has a generic description like Process data, the model will either hallucinate its purpose or never call it. Clear, descriptive summaries in your API specs are critical for agent accuracy.
Schema Reference Inlining ($ref): Many enterprise API specifications use internal schema references ($ref) to share data types across endpoints. Many LLM engines and agent frameworks cannot resolve external reference pointers automatically. The broker must resolve and inline referenced schemas before handing tool definitions to the model.
Circular Schemas: Real-world enterprise APIs frequently contain circular schema definitions (for example, a Folder that contains a list of Folder items). If your schema resolver does not set recursion depth limits, it will enter an infinite loop or produce massive schemas that exhaust the model's context window.
Context Window Blowups from Large Payloads: A GET endpoint that returns 10 megabytes of JSON is fine for a frontend data table, but it will overflow an LLM's context window. The broker must detect oversized responses, summarize them, or paginate results before returning them to the agent.
Write Actions Need Confirmation: Read-only tools (searching data, fetching records) are safe to execute autonomously. Write tools (deleting resources, triggering updates) should require explicit human confirmation or stricter permission gates before execution.