DEV Community: Kay Ashaolu

Decide where the error lands

Kay Ashaolu — Sun, 18 Oct 2026 01:04:01 +0000

How do I handle errors in distributed systems?

Wrong instinct. I watched the right-looking answer fail on my own site this week: a purchase that paid, said thank you, and delivered nothing. The buyer was me, so nobody got hurt. The next one would not have been me.

Here is what happened. A buyer paid $99 from a phone. Stripe took the money and sent my server the message it always sends: this session is paid. My server went to create the buyer's account. The account already existed, so the auth server said so, in slightly different words than my code was expecting. My code caught the error, wrote one line to a log, and told Stripe everything was fine.

Stripe believed it. The buyer landed on a page that said access granted. Nothing had been granted.

No crash. No retry. No alert. One tidy log line that nobody was reading. I only saw it because I was doing what a real customer does, buying from inside the Facebook app on a Friday afternoon and looking at the page they would see.

Look at what the try/catch did. It did its job. It caught the error. That is the whole problem: catching is not handling, and the code had no idea where the error should go next, so it went nowhere.

Errors are not edge cases you catch. They are a force you design around. The payment processor will go down. The maps API will rate-limit you. The webhook will arrive seven times, or once, with a different sentence in the body than the one you tested against. The question is never how do I catch this. The question is which block absorbs it.

A Queue absorbs a slow consumer. The producer keeps moving while the work waits in line.

A Worker absorbs a flaky call. It retries on its own schedule while the user moves on. Idempotency is the Worker's contract: run it twice, get the same result. That contract is what makes the retry safe.

And the handshake with an External Service absorbs the rest, as long as you tell the other side the truth. If my webhook had answered "not done" instead of "fine", Stripe would have knocked again, on a schedule, for three days. The error would have landed in a retry. Instead it landed on the customer, which is the one place an error must never land.

So trace it. For every failure your system can have, ask where it lands. On a Queue, to be picked up later. On a Worker, to be retried. Back on the caller, who will try again. In front of a person who can fix it. Or on the customer, silently. Only the last one is a bug, and it is the one a try/catch produces by default.

Now the part that matters more this year than last. Ask an AI coding agent to make a handler "more robust" and it will do exactly what my code did. Wrap every call. Log every failure. Return 200. It will do this with total confidence, and the result will pass every test you have, because every test checks the happy path and the one error you thought of. You get robust boxes and a silent system.

The question to put to the agent, and to yourself, is one line per failure: if this fails, who finds out, and when? If the answer is a log line, you have not handled the error. You have hidden it.

In Course 0 that line is part of the plan you write with the agent before it writes any code. The agent builds from the plan, so the failures have a place to land before the first function exists. It is a short list. It is also the list that would have granted my buyer the course.

You do not catch errors. You decide where they land.

Kay

P.S. The Queue is the block that absorbs the most failures with the least code, and the one AI forgets first. The full page on it is here: https://systemthinkinglab.ai/learn/building-blocks/queues/

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

The bug is in the handshake

Kay Ashaolu — Sun, 11 Oct 2026 01:03:41 +0000

Most bugs are not in the code. They are in the boundaries between the boxes.

You can read every function and find nothing wrong. The Service is fine. The Queue is fine. The Worker is fine. The bug lives in the seam, where two parts quietly disagree about who does what.

Here is one I have watched happen more than once.

A store takes an order. The Service writes the order to the Database, puts a "send the receipt" job on the Queue, and returns a confirmation to the customer. A Worker picks the job up and calls the email provider. Three boxes, two handoffs, all of it reviewed. Then one night the email provider times out. The Worker crashes mid-send. The customer never gets a receipt, and nobody knows until the support ticket lands on Monday.

Read the Service and it is correct: it put the job on the Queue, and the Queue is supposed to redeliver. Read the Queue and it is correct: it was configured to mark a job done the moment a Worker picked it up, because that is the default. Read the Worker and it is correct: it assumed the Queue would hand the job back if it died. Every box passed. The receipt is still gone.

Nobody wrote down who owns the retry. The Service thought the Queue did. The Queue thought the Worker did. The Worker thought the Queue did. Three confident answers, zero owners.

Every system is built from the same seven blocks, and the system is the handoffs between them. A Service calls an External Service: who owns the timeout? A Worker reads a row the Service just wrote: who promises it is there yet? A Queue hands a job to a Worker: who owns the retry, and how many times? The boxes are easy to read. The seams are where the questions live, and most of them are never asked out loud.

This matters more now, not less. An AI coding agent writes each box with total confidence. It will build you a clean Service, a clean Worker, a clean Queue client, and it will not see the handshake between them, because the handshake was never in the prompt.

The common wrong move is to ask it to "make this more robust." It will. It will add a retry to the Service and a retry to the Worker, and now the receipt goes out three times. Robust boxes do not make a robust system. A named owner at every seam does.

So before you build, write the seams down. One line each. Who owns the retry. Who owns the clock. What each side assumes the other guarantees. That list is short, it is boring, and it is the review that finds the bug the code review cannot.

In Course 0 that list is the first thing you write with the agent, before it writes a line of code, and the plan it builds from is mostly a list of handshakes.

You do not review boxes. You review handshakes.

Kay

P.S. The plan-first loop, with the four steps and the study behind it, is written up here: https://systemthinkinglab.ai/learn/plan-first-loop/

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

The decision you should not hand to an AI

Kay Ashaolu — Sun, 04 Oct 2026 01:03:37 +0000

Say your AI coding tool is about to write the piece of code that decides what happens when a job has failed every retry and has nowhere left to go: log it and move on, page a human, or something else entirely. The code itself might be ten lines. Nothing about its size tells you whether you should be the one deciding what those ten lines actually do.

That is a gap most engineers do not know exists yet. Whether a task is small enough to hand off is one question. Whether the decision inside it is yours to make is a completely different one, and a task can pass the first test clean and still fail the second. Size tells you the shape a task needs to be handed off well. It says nothing about whether handing it off is the right call in the first place.

There are four signs that a decision stays with you, no matter how small or well scoped the task around it is. Any one of them alone is enough. You do not need two to agree, and when more than one seems to fire, look for the one causing the others.

One: it touches something credentials or security sensitive, a real key, a real login, a real system the code can get into. Two: the decision is a genuine judgment call, with a real tradeoff on both sides, and no test can settle which answer is right, only a person reading it can. Three: the decision is architectural rather than cosmetic, hard to walk back once other code starts depending on the answer. Four: you want to understand this code yourself, before it becomes something you supervise instead of something you can explain. That one is not about risk: the return is your own understanding, not a mistake avoided.

Take the failed-job example. Where the record of that failure gets stored, in memory or on disk, is architectural in shape, but nothing in the system depends yet on which answer you picked, only on the fact that you picked one. That part is safe to hand off. What an operator, the human keeping the system running, should actually do when that record shows up: page someone, retry it by hand, just count them and move on, has a real tradeoff on both sides and nothing you can write a test for. That is sign two, live, in a task you might hand off today without a second thought, and it is yours to decide, not the tool's.

This decision, whether a task should get handed off at all, is part of the same method behind Course 0. It is not the same question as whether what got built is any good, judged live, inside the loop. It is not the same question as whether the brief you wrote before the loop started actually pointed at the right thing, judged from what comes back. It comes before either of those ever starts. It is the one almost nothing teaches directly, and the cost of skipping it does not show up until later, on someone else's shift.

The course is called Course 0, and it opened this week. Same approach I use in a class I teach at UC Berkeley: build a real thing with AI, then defend the design decisions behind it, not just the output.

Eight lessons and two hands-on labs, where you run the method yourself instead of watching someone else run it. Then ten short written questions, with feedback on your answers and unlimited retakes. Then three challenges on a small chat app you did not write: real code, real seams, a local chatbot you take over and make your own, real enough to break and rebuild, where you prove your read of a decision someone else already made before you touch a line. Then three wrapups after that.

Course 0 is $99. Courses I through IV are $299, one bundle, open to anyone.

P.S. Course 0 opened this week: https://systemthinkinglab.ai/course-0

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

Course 0 is open: ask the AI what it thinks first

Kay Ashaolu — Sun, 27 Sep 2026 21:43:38 +0000

You ask the AI to build the feature. It writes something that looks right, the checks you thought to run pass, and you ship it before lunch. For a while, that feels like winning.

Then production breaks at 2 AM, or code review asks why you built it this way, or an interviewer asks you to walk through a project with your name on it, and you go quiet. The code is yours. The understanding never was, because you were never really in the loop while it got built.

That gap is not a discipline problem. It is the shape of the job right now, whatever your title says.

If you are building software with AI, you are doing a senior engineering job. Not eventually. Now. An AI that builds in seconds still leaves the deciding to you: breaking the work down, questioning the plan before it builds, judging whether what comes back is good enough to ship. Those are the calls a senior engineer makes every day, and you are making them whether anyone told you or not.

Picture a senior at the whiteboard before a single line gets written. She does not ask, can this be built? She asks, what is actually being decided here? And how would we know this plan is wrong before it ships? AI changed what gets typed. It did not change who has to ask that question.

One move catches most of what goes wrong: before you let an agent build anything, ask it what it thinks. Then make it prove it. A weak answer just restates the plan. A real one points at the exact thing that would pass or fail. Ask what happens when two people hit this in the same second. A weak plan says it handles that. A real one names the exact row both writes land on. It costs you almost nothing. Skip it enough times, and you get the 2 AM version: fast code, no debugging trail, no answer when someone asks why.

I built a course around that one move. I think it is the actual skill gap right now, not typing speed. Nobody had put it in one place. It teaches you to command an AI agent the way a senior engineer would. That is how you code. The four courses after it, I through IV, build the judgment to know whether what it built is any good. That is how you grow. It is called Course 0, and it opens today. That is why this letter is arriving on a Sunday instead of the usual Saturday.

Self-paced: eight lessons and two hands-on labs where you run the method yourself instead of watching someone else run it. Ten short written questions close it out: feedback on your answers, not just a score, and unlimited retakes. Then three challenges on a small chat app you did not write, real code, real seams. A local chatbot you take over and make your own, real enough to break and rebuild: you complete two of the seven building blocks and work inside two more that are already running.

It is built for wherever you actually are. The one who ships fast with AI and cannot explain how it works when asked. The one who checks every line by hand, and it is starting to feel like a fight instead of craft. The one who plans well with the AI and loses everything the moment the context clears. And the vibe coder, still building one specific rep: sitting in the loop and pushing back on a plan before it builds.

Course 0 is $99, one time. Courses I through IV are $299, one bundle, open to anyone.

The seven building blocks stay free, same as always. Course 0 is where they stop being something you read and start being something your hands do.

P.S. Course 0 is open. If you want the details, they are here: https://systemthinkinglab.ai/course-0

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

Review is a rep, not a gate

Kay Ashaolu — Sun, 27 Sep 2026 01:03:37 +0000

The hard part of engineering moved this year. It is no longer writing the code. It is deciding whether to trust it.

Addy Osmani put a rule on that in June, and the rule is good: tier your review by risk, not by who wrote it. A config change gets a linter. A payments path gets a human reading every line. Spend attention where being wrong is expensive. For a team shipping fast, that is exactly right.

Here is the change it tells you to skip.

Monday morning. Your agent opens a pull request: "Add retry to the receipt email worker." Sixty lines. Tests green. It is a background job, not a payment path, so the rule says the linter can have it. You have eleven other things open. Approve.

Now read it instead. The job pulls a message off the Queue, renders the receipt, calls the email provider, and marks the order as notified. The retry the agent added wraps the whole job. A timeout anywhere inside it runs the whole job again from the top.

Trace one bad afternoon. The email provider accepts the send, then the connection drops before it answers. The Worker sees a timeout. The retry fires. The customer gets two receipts for one order, and the second one arrives ninety seconds after the first, which is exactly the shape of a charged-twice email. Support gets the ticket. Nobody can find a bug, because there is no bug in any single line. There is a bug in where the retry boundary sits.

The fix is small. Retry the render and the send separately, and make the send safe to repeat by handing the provider an idempotency key built from the order id. Four lines. But you only get to write them if you read the sixty.

That is what the risk rule cannot see. It scores the change by what it touches. It cannot score what reading it would have taught you. The linter would have passed this pull request every time, forever, and you would have learned nothing from it every time, forever.

This is the same shape in every system you will ever work on. Instagram's feed is a cache that has to be told when a post changes, and the interesting question is who tells it. Uber's dispatch is a Queue with a clock on it, and the interesting question is what happens to a ride request when the Worker holding it dies. None of those questions live in the code. They live at the seams between building blocks, and you only find seams by reading across them.

So here is the thing to add to the rule if you are early in your career. Review is not only a gate. It is a rep. Reading code you did not write, deciding whether it holds, finding the edge the test did not cover: that is the whole exercise that builds judgment, and there is no other one. The throughput math says move on. The growth math says read it anyway.

The wrong move is the obvious one. Approving faster looks like seniority. It is the opposite. A senior engineer is not someone who trusts more code. It is someone who has read enough code to know exactly which lines to distrust. Trust is not a setting you turn up. It is a residue that reading leaves behind.

You do not have to read everything. Read the ones that would teach you something if they were wrong, which is most of them when you are starting. Five a day, for a year, is more system reading than most engineers do in five.

In the seven building blocks I teach juniors to read systems before they write them: to trace one request across a Service, a Queue, a Worker, and a store, and to name where it can break. The reading is where the seeing gets built.

You do not build trust by clicking approve. You build it by reading until you could have written it yourself.

P.S. If you want reps that are not your own pull requests, I wrote up three real systems block by block, Netflix, Instagram, and Uber, and the seams are marked: systemthinkinglab.ai/learn/case-studies/

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

Three ways to undercharge a customer

Kay Ashaolu — Sun, 20 Sep 2026 01:03:43 +0000

Here is a ticket I use whenever someone asks me what senior actually means. Add a discount code field to checkout. It reads like twenty minutes of work: a text box, a lookup, a percentage off. You could hand that line to an agent and have working code back before your coffee cools.

Before you write anything, walk the ticket past four stops it never mentions. Three of them are ways this same ticket can quietly undercharge someone. One of them, finance's ledger, is where the other three get found out.

The first stop is the price the buyer sees. The code has to apply before tax, not after, and it has to round the same way every other price on the page rounds, or two customers checking out a minute apart see two different rules for the same 15 percent off.

Then there is the card that actually gets charged. The number on the screen and the number the processor charges have to be the same number, captured at the same moment, even if the buyer sits on that page for ten minutes with the tab open before hitting pay.

Then finance's ledger. A sale made at a discount is not a full-price sale with a note attached. Revenue, tax, and any commission owed on it all have to be calculated on what was actually paid, not on the sticker price, and the money has to get booked correctly even if "finance" is one person with a spreadsheet, or the monthly numbers stop matching the bank and someone spends an afternoon finding out why.

The fourth stop is every refund that ticket will ever cause. Six months from now, a customer who used that code returns the item. Do you refund what they paid, or the full price on the listing? Get that wrong and you have quietly become a business that sometimes pays customers more than they gave you.

None of that is written in the ticket. All of it is the part of the job that is still yours.

The same shape shows up anywhere a ticket looks smaller than it is. A field that looks like one column touches five. A toggle that looks like a single boolean touches billing, permissions, and an email that fires the moment it flips. The ticket never says any of that. Seeing it anyway, before the first line, is the job now.

An agent will happily write the discount field. It validates the code, applies the percentage, and returns a clean number to the page, because that is exactly what the ticket asked for. It has no way to know your refund window, how your finance team books discounted revenue, or what happens the day two codes are allowed to stack on the same order. That knowledge does not live in a tutorial anywhere. It lives in your product, your data, and the way your company already handles money.

That is the part of engineering a tool cannot hand you. Clean code is a skill a tool can now point at and match. Seeing the four stops before you write the first line is not.

Here is where it gets easy to measure yourself by the wrong number. If you spend the next few months grading your own worth by how clean the discount field turns out, how tight the diff is, how fast you shipped it, you are grading yourself on the part of the job that just went free. You built real things with that code, and that is a real skill. But an agent produces it now in less time than it takes you to describe the field, and if that is still where you look for your value, you will be measuring none of the leverage you are actually gaining.

I built a course around practicing exactly this: running the discount-code ticket, and ones like it, with an agent doing the typing while you do the seeing.

Shipping the field faster was never the point. Naming the four things the ticket never mentioned, before anyone has to ask, is.

P.S. The framework for spotting everything a ticket actually touches, before you write the first line, is free: https://systemthinkinglab.ai/learn/building-blocks/decision-framework/

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

Why your AI agent keeps starting from zero

Kay Ashaolu — Sun, 13 Sep 2026 01:03:47 +0000

Picture handing a brand-new engineer the raw Stripe API documentation on their first morning and telling them to ship a working payment flow by five o'clock. Nobody does this. A new hire is supposed to get a kickoff meeting, an onboarding doc, and a senior engineer who walks them through how this particular codebase works before they touch a line of it.

Now picture what we do to an AI coding agent instead. Every session starts the same way that new hire's first morning would, except nobody runs the kickoff meeting. The agent opens a codebase it has never seen, guesses at your naming conventions, and writes code that works but matches nothing else in the repository. We call the result disappointing. The result was never the problem.

Here is what that gap actually costs, across three sessions on one codebase. Session one: you ask an agent to add a webhook handler to a service with a dozen existing endpoints. It writes reasonable code. It also invents its own naming convention for the file, skips the retry wrapper every other endpoint in the service already uses, and never notices the codebase already has a small library built for exactly this kind of validation. You catch it in the diff, fix the naming by hand, and explain the retry pattern in the chat. The session ends. The explanation ends with it.

Session two, new task, new context window, same codebase. The agent remembers none of it. It reinvents the naming convention, wrong in a different way this time. So instead of correcting it again and moving on, you write the two rules it keeps missing, naming and retries, into a short file the agent reads before it starts anything. You also ask it to add one line to a second file describing what it just learned about how this service handles errors. Two files. Ten minutes.

Session three, same codebase, a different feature. This time the agent reads both files before writing anything. It matches the naming on the first try. It does not ask how to handle retries. It asks whether the new endpoint belongs in this service at all, or whether the work is slow enough that it should go through the queue the rest of the service already uses for background jobs. That is not a better answer. That is a better question, the kind a new hire only asks after someone has actually shown them how the system fits together.

The two files were not a trick. They were the onboarding a new hire gets and an agent almost never does. The first file, a kickoff skill, is the version of that first-morning meeting: your naming rules, how errors get handled, how services are supposed to talk to each other, loaded before the agent writes a line. The second file, a wiki, is the version of a senior engineer explaining how this particular codebase works, except the agent writes it down itself, and it is there again waiting for the next session. Handing an agent the raw Stripe docs and expecting a shipped payment system by five o'clock was never going to produce clean work. The two files are what you would have given the new hire instead.

The tempting move, especially the first time an agent gets something wrong, is to write a better prompt. Spend twenty minutes getting the instructions exactly right, ship the task, feel clever about it. That cost gets paid again next session, because a prompt lives inside one conversation and disappears with it. The kickoff skill and the wiki live outside the conversation. Session one is expensive because the agent is learning your codebase from nothing, the same as any new hire's first week. Session three is fast because the learning from session one and two is sitting in a file, waiting to be read instead of relearned. That is the entire difference between a cost you pay once and one that compounds.

I am building this habit into a course I am finishing.

You do not get a better agent by writing a better prompt. You get one by building the memory it reads before it writes the next line.

P.S. If you want the framework behind how I think about designing systems in the first place, it starts here: https://systemthinkinglab.ai/learn/building-blocks/7-building-blocks/

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

The seat that sold twice

Kay Ashaolu — Sun, 06 Sep 2026 01:03:37 +0000

Two strangers bought the same seat. Not seats near each other. The same seat. Row 14, seat 7, sold twice in the same second, to two people who have never met, and both of them are holding a confirmation email that says it is theirs.

Here is how a system gets there. Picture the site selling tickets for an arena show, and picture yourself as the engineer who built the part that sells the seats. The design is the obvious one: a seats table in the database, one row per seat, with a status column. When a buyer clicks Buy, your code checks the row. Is the status open? If yes, charge the card and mark the seat sold. If no, show "seat taken." It reads cleanly. It passes every test. It works every time you click through it yourself.

Then Friday at 10 AM arrives and the tickets go live. Ten thousand people are on the seat map at the same moment, and two of them click row 14, seat 7, inside the same tenth of a second. Buyer one's request reads the row: open. Buyer two's request reads the row a few milliseconds later, before buyer one's update has landed: still open. Both requests pass the check. Both charge a card. Both write sold, the second landing quietly on top of the first. Both buyers get the email. Your code did not misfire once. It did exactly what you wrote, twice.

The failure costs nothing for three weeks. Then the show happens. Two people walk up to row 14, seat 7, each holding a ticket that looks completely real, and an usher gets to decide which one of them paid for a seat that exists. Behind that moment is a support queue, a refund, an apology email, and a buyer who will never quite trust the site again. The worst part is that nothing in your logs looks broken. Every request came back a success. The system never went down. It just quietly promised the same thing to two people.

Look at where the failure lives. It is not in the check, which was correct, and not in the write, which was also correct. It is in the gap between them. Your code checked the seat at one moment and claimed it at a slightly later moment, and treated the world as if nothing could happen in between. On a quiet Tuesday, nothing does. During an on-sale, that gap is a door, and ten thousand simultaneous buyers means somebody walks through it.

The way out is not to make the gap smaller. It is to hand the job to the one layer that can make the gap not exist. That layer is the Relational Database, and the tool is called a constraint: a rule you declare about what is allowed to be true in your data, which the database enforces at the moment of every write, inside the write itself. To use one here, change the shape of the sale. Selling a seat stops being an edit to a status field and becomes the creation of a claim: buying inserts a row into a tickets table, and the rule you declare says no two rows may ever name the same seat for the same show. Now when two claims race for row 14, seat 7, the database lets the first one through and refuses the second at the instant it tries to land. There is no check at one moment and claim at another. The check is the claim. Your code's job shrinks to catching the refusal and telling buyer two, "that seat just went."

Notice what actually happened to the collision. It did not disappear. Ten thousand people still clicked at once, and two of them still wanted the same seat. The collision moved down a layer, out of your application code and into the database, which is the one place every claim must pass through, and which resolves competing writes to the same data by lining them up and taking them one at a time. Refereeing simultaneous claims on shared data is not a feature the Relational Database happens to have. It is a large part of what the block is for.

Once you see the shape, you will find it everywhere. Two people signing up for the same username in the same second: the designs that survive it put exactly this rule on the username column, which is why the second person sees "already taken" instead of two accounts existing. A flash drop where the last hoodie in stock sells to two carts. Two meetings landing in the same room at 2 PM. Anywhere a design checks first and writes second, across a gap, two actors can pass the same check before either write lands.

Now watch the fix that gets reached for first, because it is the wrong one: keep the check in application code and reinforce it. Check the seat again right before the write. Add a background job that sweeps for double-sold seats. Hold a lock in your server's memory while the purchase runs. Every one of these feels like defense, and every one of them is another racer. Your application does not run as one copy. It runs on several servers at once, each running the same checks, and no copy can see what another copy is about to write. Doubling the checks doubles the checkers, and the checkers are the thing that races. More checking makes the gap shorter. Application code cannot make it zero. At on-sale traffic, shorter still loses.

The database can make it zero, because for any one seat it is one place, taking one write at a time. That is the whole trick. Not smarter checking. One referee.

In Course 1 I teach the Relational Database as one of the seven building blocks. Every block earns its place by doing a job the others cannot. Refusing the second claim is this one's.

You do not close the gap by checking faster. You close it by making the check and the claim one move.

P.S. The constraint is one reason the Relational Database earns its spot among the storage blocks. The free walkthrough of the three storage extremes, what each one is built to maximize and what each gives up, is at https://systemthinkinglab.ai/learn/building-blocks/storage-extremes/

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

Your next song was already chosen

Kay Ashaolu — Sun, 30 Aug 2026 05:25:07 +0000

One song on a playlist ends and the next one starts before you notice a gap. That is worth stopping on. Something chose the exact next song out of Spotify's entire catalog, in the space between one note ending and the next one beginning. It did not pause to think about it.

Here is the part most people get wrong. They picture the choice happening at the moment the song ends: some process wakes up, looks at your history, scores a few million candidate songs, and picks the best one, all in the half second before the next track starts. If that were true, you would hear it. A beat of dead air. A spinner. Some evidence that work was happening.

That is not what happens. The real work happened hours earlier. A background process, what I call a Worker, had already taken your recent listening and turned it into a compact summary of your taste. That summary got compared against millions of other songs, ranked by fit, and the winner got written down and held ready, sitting in a Queue or a Key-Value Store tied to your account, the kind of storage built for exactly one job: answer fast when asked.

So when the song actually ends, the app on your phone does not compute anything. It reads a value that was already sitting there waiting for it. The transaction is not "figure out the next song." It is "look up the answer to a question that was already answered."

Once you see that shape, you cannot stop seeing it. Gmail does not scan every email in your inbox the moment you hit search. A Worker already built a search index while you were doing something else, and your query reads that index instead of your inbox. Google Maps does not calculate current traffic the instant you open the app. Probe data from other drivers gets folded into an estimate continuously, in the background, so the ETA is sitting there the moment you ask for it. Your feed, on whichever app you check first each morning, did not start from nothing when you opened it. Most of the ranking work was already done.

Every one of these is the same trade: a Worker does expensive work on its own schedule, and a Queue or Key-Value Store holds the result until the exact moment someone needs it. The part the user sees is only ever a read.

The tempting move, the one that feels smart, is to compute the answer when the question actually arrives. That is when you know the most: the real time of day, the real device, the real thing the user just did three seconds ago. Waiting feels like the more accurate choice, and for a junior engineer it usually is the first instinct.

The senior move is the opposite. Spend the compute before the question shows up, and accept that your answer is built on slightly older information in exchange for one thing: the moment someone actually needs it, there is nothing left to do but hand it over.

In Course 1 I teach this shape by name, the Worker that does the early work and the Queue or Key-Value Store that holds the answer until it is needed.

You do not wait for the question to begin the work. You begin the work and wait for the question.

P.S. If this is the kind of thing you want to start seeing everywhere once you know to look for it, the free course walking through all seven pieces is at https://systemthinkinglab.ai/learn/. This is one shape out of many, and it keeps showing up.

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

All four courses are finished. Here's what's next.

Kay Ashaolu — Sun, 23 Aug 2026 01:03:33 +0000

Thank you for reading this letter, many of you since before the series was finished. Systems Thinking in the AI Era is the four-course series I teach: seven repeating primitives that every real system is built from, and how to design with them instead of just coding around them. You kept showing up while half of it was still unrecorded. That is not a small thing, and I do not say it enough.

This week I recorded the last video. Every lesson, every discovery lab, every case study, every challenge, and every assessment across all four courses is now live. That part of the series is complete, and I am genuinely excited about what comes next.

Because the four courses were never the whole plan. They were the first half.

If you are writing code with an agent's help, you have been handed a senior engineer's job on day one. Nobody announced it. You are not writing every line anymore. You are breaking a task into pieces the agent can execute, judging what it hands back before you trust it, and deciding whether the design actually holds. That is senior work. The agent is your junior engineer. And nobody hands you the structure a senior spends years building to do it well.

Course 0, coming late September, is that structure. It teaches how to hand a task to an agent with a plan it can actually execute instead of a vague ask, how to verify what comes back before it becomes production code instead of after it breaks, and how to set up an agent's memory so it gets better at your codebase every session instead of starting from zero each time.

But structure alone will not save you. You can hand off a task perfectly and still have no idea whether the result is any good. That is the part Courses 1 through 4 exist for, and it is why I built them first. Course 1 is the framework: the seven building blocks themselves. Courses 2 through 4 are the case law, the same seven blocks traced through content systems like Instagram and Netflix, real-time systems like Slack and Discord, and business systems like Stripe and Shopify. You do not build judgment by reading a definition. You build it by watching the same primitive solve the same kind of problem enough times that you recognize it on sight.

Put together, that is the pathway this series was always meant to be. Not a course you finish and file away. A way to work with AI that makes you sharper while it makes you faster, on the same work, at the same time. Course 0 gets you moving. Courses 1 through 4 make sure what you are moving toward is good.

Four ways in, depending on what you already own.

If you already own all four courses: Course 0 is free for you the day it ships. Nothing to buy, no extra step. You are covered already, and I wanted you to hear that directly instead of guessing.

If you own some of the four courses but not all of them: complete the set for $300, and Course 0 is free the day it ships. This one is for finishing what you started.
https://checkout.systemthinkinglab.ai/b/8x2fZhfm43Jg2WMeaV7ok08

If you have not bought a course yet: the full catalog, all four, is $399, and Course 0 is free at launch too. This one is for taking the whole thing at once.
https://checkout.systemthinkinglab.ai/b/14A6oH3Dmgw21SIaYJ7ok09

Both of these offers end when Course 0 ships.

If you would rather start with one course and see how it fits: every course is still $149 individually, at systemthinkinglab.ai.

No pressure on any of it. If you are just here for the letter every Saturday, that is genuinely enough, and I am glad you are here for it.

P.S. Course 0 is coming late September. Already own all four? It is free for you, no extra step. Most of the way there? Finish the set before it ships and it is free too.

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

The most boring service in the stack

Kay Ashaolu — Sun, 16 Aug 2026 01:03:38 +0000

The most reliable part of a stack is rarely the part anyone shows off. It has no clever layer to point at. That is the whole reason it never breaks.

A payment service I want to walk through this week is a clean example of that. One queue sat in front of it, catching every incoming charge. One database sat behind it, recording each charge exactly once. That was the entire system, and it had looked that way for four years.

Engineers rotated onto the team and asked the same question every time. Why is this so primitive? No event mesh. No service constellation. No clever caching layer sitting in front to make it feel faster.

Twice a year, someone pitched a redesign. Twice a year, the answer was no.

Then Black Friday arrived. The recommendation engine buckled under the traffic and stopped returning results. Search slowed to a crawl and started serving stale listings. The mobile app threw errors across half its screens, and the support queue backed up.

The payment service kept working. Every single order that reached it got processed, exactly once, without a single duplicate charge.

Boring was never the absence of ambition. It was the design.

The parts doing the real work here have names. Use them. A Service is the piece that answers you while you are standing there waiting on it, the way a waiter takes your order and comes back with a plate. A Queue is a line for work that does not need to happen the instant it shows up, so a sudden rush of it lands somewhere safe instead of getting dropped. A Relational Database is the record book: the one place that can say, with certainty, this specific charge happened, once, and only once. One Service, one Queue, one Relational Database. Three pieces, not ten.

You can see the same restraint inside Stripe's own payment API. Every charge you send Stripe can carry an idempotency key, a value that means "this is the same request if you see it twice." Your network drops the response, you retry the request, and Stripe uses that key to recognize the retry and hand back the original charge instead of billing your customer again. Stripe did not hand you a new service to run to catch duplicates. It handed you a key, and made one place the last word on whether a charge already happened. Same restraint. Same refusal to add a fourth. You do not need Stripe's traffic to use the same idea. You need one place your data trusts to be the last word on whether something already happened, and the discipline to let it do that job alone.

The way a system like this actually gets worse is rarely neglect. It is ambition wearing a good disguise. An engineer joins the team, wants something to show for the quarter, and pitches the event mesh, or the extra caching layer, or a second message broker sitting in front of the first one. Every one of those pitches sounds like seniority. On a slide, an event mesh reads like foresight. Under real load, it is a new component that has to come up cleanly, hold a connection, and survive Black Friday: a new thing that can time out, drop a message, or disagree with the database about what actually happened, at the exact hour agreement matters most. Saying no twice a year was never stubbornness. It was refusing to add one more way to fail during the one week failure would have cost the most.

In Course 1 I teach these seven building blocks before anyone draws a single box, because the skill this payment service is showing off is not stacking parts well. It is knowing when three of them are enough, and having the discipline to stop there.

You do not protect a system by adding to it. You protect it by refusing what it does not need.

P.S. Knowing when three blocks are enough is a skill you can learn deliberately. The decision framework that teaches it is free here: systemthinkinglab.ai/learn/building-blocks/decision-framework/

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.

The two opposite ways AI gets your system wrong

Kay Ashaolu — Sun, 09 Aug 2026 01:03:46 +0000

You asked an AI to help you build something, and it put the whole thing inside one service. I have written about this before: a Service is a waiter, it takes the order and comes back with an answer. Ask AI to write the code for your app and it puts everything behind that one order: validate the input, hit the database, send the email, all before the response goes back. It works in the demo. In production the email step is slow, so every user stares at a spinner while your server waits on someone else's mail service.

Ask the same AI to design that system before you write a line of code, and you get the opposite mistake. Same AI. Same app. Opposite failure. You and one other engineer are building an app that lets roommates split rent and bills. Ask AI to design it, and it hands you five services: auth, billing, notifications, an API gateway in front of all of them, and a message broker connecting them.

Five services sounds manageable until you count what is actually inside each one. Auth is not one thing. It is a Service, a Relational Database of user accounts, and a Key-Value Store holding sessions: three building blocks. Billing is a Service plus its own Relational Database of charges: two more. Notifications is a Service, a Queue, and a Worker to send the email: three more. The API gateway is a fourth Service sitting in front of the rest, and the message broker is its own Queue. Ten building blocks, minimum, spread across five separate deployments, for a team of two.

What the app actually needs looks nothing like that. Someone logs in and adds a bill: one Service. The bill has to know which users it belongs to and how it is split: one Relational Database. When someone marks a bill paid, you want to email the other roommates, and that does not need to happen while anyone is watching a spinner, so it goes on a Queue, and a Worker picks it up a few seconds later. One Service, one Relational Database, one Queue and Worker: the whole system. Four blocks, one deployment, against ten blocks spread across five.

Ask AI to write code and it defaults to the plainest possible shape: one function, one service, request in, response out. It never reaches for a queue, because a queue is a decision about what can wait, not the plain shape of code. Ask AI to design a system on its own and it defaults to the most-discussed shape it has read about, which is how large companies solve problems you do not have. Same root cause both times: it is matching the genre of your prompt, not your actual constraints.

This is why the blocks matter more than the word "service." A microservice is not a building block. It is a whole deployment that can contain two or three of them. Count in services and "five" sounds like five things to build. Count in blocks and it is ten pieces you have to build, wire together, and keep alive, spread across five deployments you run separately. Against four pieces in one.

Microservices earn their cost when separate teams need to ship without waiting on each other, when one part needs different hardware, or when one failure cannot be allowed to take down everything else. Two people building one app have none of that pressure. What you need is one place to change things, one deploy, one thing to reason about at 11pm: a monolith. Not the beginner version of something you graduate out of. At your stage, it is the correct design.

Run every piece of the AI's answer, whether it built too little or too much, through one question: does the pressure that justifies this exist on my team, right now? Multiple teams stepping on each other's deploys. A component that needs different hardware. Traffic you can already measure, not traffic you are imagining for next year. If that pressure is not real, the piece comes out, whether it is a missing queue or an extra service.

That is why I teach the seven building blocks in Course 1 before anyone draws a single box. Know what each one is for, and you can look at any answer, AI's or your own, and ask what it is actually built to survive.

You do not fix the answer the AI gave you. You check the pressure behind every piece of it.

P.S. If you want the repeatable version of that question, I wrote up the full decision framework: one question per building block, so you can point at any requirement and know exactly what it needs. https://systemthinkinglab.ai/learn/building-blocks/decision-framework/

This letter goes out by email every Saturday. Subscribe here or read the full archive at systemthinkinglab.ai/newsletters.