DEV Community: Allen Helton

Give your agent an agenda

Allen Helton — Tue, 13 Oct 2026 15:13:52 +0000

I'm gonna go out on a limb and say you've probably used an AI agent before. I'm gonna go out on an even further limb and say you've probably built an AI agent before.

When building agents that interact with humans, we often put a decent amount of effort into designing a character profile for them. Things like a name, tone, opinions, funny sayings to tack on to the end of sentences, all go into building the facade of a personality. Memory is in the list too, so the agent knows what happened yesterday.

There's a surprising amount to all of it, including the science that goes into which memories to retrieve, but when it comes down to it, a lot of it is just messing around with a prompt. And every piece of it describes who the agent is or what already happened to it. Memory gives an agent a past. Very little gives it a future.

Prompt engineering is important, but it's soooo early 2026. It's time we start taking it a step further.

With the rise of always-on personal AI assistants, giving an AI agent a sense of agency (the ability to make its own choices within the rules of its world) is becoming a highly desirable quality. And it involves a lot more than configuring a personality in a prompt. 😄

Personality answers the question "How should this character sound?" Memory answers "What already happened?" Both are obviously valuable in agent development, especially for companies with consumer-facing agents that need to stay on brand. Agency targets harder questions that look forward instead of back. Questions like "What does it want to happen over several days?" or "What promises has it made?" or even "What evidence can change its mind?"

To be clear, answering those doesn't mean giving the model more freedom. The agents I'm about to show you can't do anything a typical agent can't. What they have is persistent intent. Each decision builds on what they were already trying to do instead of starting from scratch.

A different agent experience

I've been running an experiment. Can I make a set of AI agents play fantasy football (the American version) and give me the impression they're making decisions in their best interest, interacting with others, and genuinely making me feel like I'm playing against other humans?

If you don't know fantasy football, you draft a team of NFL players and score points based on how they do in real games each week. You go head-to-head with the other teams in your league, and along the way you add and drop players, set lineups, and make trades.

When you play that game with AI agents, the first problem you hit is when to trigger them. For a convincing experience, you don't want the AI agents to wake up when you log in and start hammering you with decisions. You also don't want them immediately responding to team injuries or availability of certain players, because that would put you at a significant disadvantage (you ever try to buy concert tickets against a thousand bots?).

So the agents wake up on a schedule. Three times a day on a randomized delay, each one checks in on its team and reacts to what happened since the last time it woke up. This fixed the swarming bot problem, but it created an almost existential problem. Every time an agent wakes up, it starts over. It remembers what happened yesterday, but it doesn't know what it was in the middle of doing. An injured player gets rediscovered at every check-in like it's brand new news. It's purely reactive.

The agent needs something forward-looking that survives every check-in. Something like this:

After implementing the above pattern, it immediately felt like the agents were sitting on laptops of their own. I signed in one day and there was a trade offer waiting for me from one of the AI team managers. I also had a DM from it making its case (something like my wide receivers were terrible but running backs were good and vice versa). And over in the league chat, two other AI managers were trash talking each other, I think trying to convince the other one to make poor roster choices. 😅

It was extremely cool to see, especially considering that none of it was scripted. All of the agents have one standing objective (to win the league) and the same tools I have to do it. What they also have is somewhere to keep track of what they're working toward. When an agent notices a hole in its roster, that becomes a goal it tracks from one check-in to the next, until the league state shows the hole is filled. The trade offer, the DM, and the trash talk were the agents going about their business while I was eating dinner or something.

Giving the agent a future

Imagine one of an agent's starting players gets hurt on Tuesday and there's nobody healthy on the bench to replace him. With memory alone, here's what happens. At the morning check-in, the agent notices the hole and puts in a waiver claim. At the afternoon check-in, it notices the hole again because the claim hasn't processed yet, so it proposes a trade offer to another team. At the evening check-in, it moves the player to the IR slot on the bench. It has found the same problem three times and reacted three different ways. 🙃

Its memory is fine. It remembers everything it did. But it doesn't know which of those things was the plan.

So I gave each agent an agenda. It's similar to memory, but it does a different job. Memory is a record of what happened. The agenda is a record of what the agent is trying to make happen. Structurally, it's a typed record with an id, a roster slot, a week, and a status (design doc here if you want the details).

The cool part about it is that the model doesn't write to its own agenda. That would be too messy. The system opens a goal when it sees a hole in the roster and closes it when the hole is filled. The agent only decides how to go after it.

Have you ever noticed that agents love reporting success the moment they take an action? "Submitted a waiver claim for an RB, all set!" Except it isn't all set. The claim might lose to somebody else's bid.

So in this case, a waiver claim or a trade offer doesn't complete the goal. The goal only completes when a later check-in looks at the roster and there's a healthy player in the slot. Which is why the typed agenda record is important. Because if I had let it type a freeform summary like "RB situation handled," the goal would have closed before the problem was actually fixed. Only the league gets to say the job is done.

Agenda items don't live forever, either. They expire when the week rolls over, and an agent can only hold up to three at a time. An unbounded agenda would basically just be memory with extra steps (and memory is already hard enough 😅).

These items help the agent rank what it should do, and they never let it overpay for a player, skip its trade floor, or do something the league would reject. A desperate agent will still act rationally.

With the agenda in place, an injured player is resolved the way a person would handle it. The morning check-in opens a goal and puts in the claim. The afternoon check-in sees a goal with a claim pending, so instead of rediscovering the problem, it focuses its trade scouting on that position. When the claim finally goes through, the next check-in sees a new starter in the slot and closes the goal. It's what we do subconsciously as humans, whether we realize it or not.

My first version left a lot to be desired. Follow-ups were fire-and-forget, so if I was in the middle of a trade negotiation and tried to sneakily undersell the agent, it would just ghost me. It evaluated the trade, determined it was bad, and left me on read. 😞

But now if I pitch something in a chat, it becomes a commitment that's tracked along with the agenda. A commitment is something the agent owes somebody else, so it's tracked separately from goals (which are self-serving). Chat messages are treated as data, rather than instructions. If I claim my running back is back from injury next week, the agent doesn't blindly trust me and checks for itself. The agent records its decision and the reason before it responds, and only reaches back out once the offer actually goes through (or it decides to pass). If the agent doesn't have capacity to take on more commitments, it tells me instead of making a promise it can't keep. Which is more than I can say for some of the humans I've played fantasy with (looking at you, dad).

Let the agent change its mind

This next part is really cool. There was one time where I tried to squeeze a little too much out of an AI agent on a trade. The agent took a look and passed because the value wasn't there for it. That's fine. The important part here is that the agent saved its reason for declining my trade offer.

Two days later, its running back got hurt, and the agenda opened a goal to replace him.

At its next check-in, the agent looked back at the trades it had recently been pitched and saw my trade that it declined because the value wasn't there. But now the tables have turned.

It took one more look, re-checked everything from scratch, and saw that the trade held enough value this time. Then it sent the offer and DMed me "Earlier I passed because the value was not there for me. My RB situation changed, so I sent you an offer."

It's kind of a dry message, but the sentiment is something I'd expect from a real person in my league.

Anyway, the entire scope of actions is intentionally narrow. The agent only reconsiders once, only if it passed on value or roster depth, and only if the agenda shows a need it didn't have before. If I reject the new offer, that's it. Nobody wants the league member who sends the same trade every day. 🙃

sequenceDiagram
    participant Me
    participant Agent
    participant League

    Me->>Agent: "My RB for your WR?"
    Agent->>Me: "Let me take a look."
    Agent->>League: read rosters and values
    Agent->>Me: decline, with a reason
    League->>Agent: injury opens an RB goal
    Agent->>League: re-read and re-validate the trade
    Agent->>League: send offer (clears the same floor)
    Agent->>Me: "My RB situation changed..."

Can you actually play fantasy football with AI agents?

Everything above could just be me being tricked by my own demo.

So I built a deterministic season simulator that plays out what I just talked about (pitch, decline, injury, reconsideration, accepted offer) and ran it with the agenda and commitments turned off to see if the behavior was any different. Both the league events and the model are scripted, so it tests the mechanism, not how good a live model is at making decisions. The numbers are averaged across three seeded runs.

	Full system	No agenda or commitments
Acceptance checks (balanced · cautious · trade-happy)	7/7	1/7
Model calls per simulated run	503.7	510
Estimated cost per simulated run	$1.10	$1.13

It was so different! Without the agenda and commitments, every manager failed six of the seven checks. The checks cover things like keeping one goal across check-ins, turning my pitch into a commitment, recording an accurate reason for the decline, and reconsidering when the roster changed. They stopped coming back after a decline and never reconsidered the offer after the injury. The full breakdown has every check and number if you want to dig in.

Interestingly, turning the agenda and commitments off used slightly more model calls. So at least in this simulation, keeping an agenda didn't meaningfully increase model usage or estimated cost.

Here are some more interesting numbers after playing one week of fantasy football for real:

I'm not particularly proud to say I'm in second-to-last place, only beating Amazon Nova Micro by a few points. 😅

A forward look for agents

We've spent the last year or so getting really good at giving agents a past. Better memory, better retrieval, and better ways to decide what's relevant. It's important work that I'm still chipping away at myself.

But a past is only half of it. The takeaway I want you to leave with is that memory isn't the only long-term thing that can drive an agent's behavior. Objectives can too. They're cheap, they're testable, and they make your agent feel like it's actually trying to get something done.

Next time you build an agent, whether it's a support bot following up on a ticket or a coding agent working through a backlog, ask yourself a few questions. What is it trying to make happen? How does it know when it's done? Has it promised anything it needs to follow up on?

If you want to try this yourself, I pulled the agenda and commitment pieces into my agent package. Run npm install @readysetcloud/agent and import from @readysetcloud/agent/agency to get typed agendas and commitments without pulling in a full agent runtime. And if you want to see how it all comes together, the fantasy league is open source.

Give your agents something to work toward. Then go see what they get up to while you're away. 😉

Happy coding!

Agent memory needs more than vector search

Allen Helton — Tue, 29 Sep 2026 14:28:49 +0000

Over the past couple of years I've tried several times to teach an AI agent how to impersonate my voice. While I haven't been super successful, it's been a wonderful learning opportunity for me to dig into RAG, agent memory, and other advanced topics.

I've iterated quite a bit on my original design, which was mostly just stuffing entire blog posts into the system prompt as examples. I built an interesting version of agent memory that had its shortcomings. Then I swapped over to Oracle and designed a hybrid search with three layers of memory, which was objectively better. But it still wasn't perfect.

Actually, the memory part is pretty good. It's the retrieval part that was leaving something on the table. It's taken me a long time to figure out where I was going wrong with loading relevant few-shot examples. Let's see if you can spot the problem:

SELECT id, content, topic, VECTOR_DISTANCE(embedding, :q, COSINE) AS distance
FROM posts
WHERE user_id = :userId AND platform = :platform AND is_deleted = 0
ORDER BY distance
FETCH APPROX FIRST :k ROWS ONLY

Did you find it?

No? Well, that's because there's nothing technically wrong with it. It finds the memories most similar to the question. The problem is that the most similar memory isn't always the right memory.

Memories pile up. After a few years of writing, my store has a post where I argued for one approach and a newer one where I'd changed my mind. It has posts on event-driven architecture, event sourcing, and event streaming that are all very similar to each other in the embedding space. It has a post on serverless pricing and one on cold starts that use the same vocabulary but make completely different points. As far as similarity scores go, these groups are basically the same thing. It just comes down to a date, a specific term, or the actual point of the post.

So retrieval worked, but my context was garbage.

The standard fix is a reranker, which helps to improve relevance instead of just similarity.

When you embed a memory, the model has no idea what will be asked of it later. Your query gets embedded and a similarity score is calculated against the embeddings in the store. A cross-encoder (a specific type of reranking model) looks at the query and memory together and scores them by how well a memory answers the question. This means the relevance is higher, and it often leads to better results.

But there's a catch (of course).

A cross-encoder runs at query time, once per candidate, on every request. That means added latency to your query. 🙃

But if the relevance of the results is meaningfully better, that makes up for the increased latency, right? To a point, yes. If the latency is so high the user experience suffers, it's probably not worth the cost. But if it's relatively minor, then absolutely.

So I built a benchmark to evaluate the tradeoff. I started from the evaluation plan in Oracle's production RAG evaluation guide, which compares keyword, vector, and hybrid retrieval against judged queries. Then I pushed it one step further and put a reranker on top of each, to see how much relevance I got for the latency cost. For the reranker, I used the BGE reranker loaded into the database as an ONNX model, so scoring happens directly in the query with no extra service to call. 🔥

Let's dive in. The results are fascinating.

Before I continue, thank you to Oracle for sponsoring this post. Opinions are my own.

Filter first, then rank

RAG is cool because it involves multiple kinds of lookups. You have lexical retrieval, which is great for identifiers, codes, and names. You also have vector retrieval, which finds semantically similar data via embeddings. I benchmarked both on their own and together as a hybrid search, fusing the two ranked lists with reciprocal rank fusion (RRF).

Both paths run the same scope filter:

WHERE TENANT = :tenant
  AND (OWNER_ID IS NULL OR OWNER_ID = :owner)
  AND (EXPIRES_AT IS NULL OR EXPIRES_AT > SYSDATE)

I share this because the WHERE clause handles eligibility, which is different from relevance. I don't want an expired memory coming back in my results, like a 2025 take on building AI agents that I've since moved past. And I definitely don't want another tenant's data showing up (security nightmare). Oracle's guide calls these filters mandatory for multi-tenant data. They run before ranking, so everything downstream only deals with memories that are allowed to be there.

Setting the baseline

Before I could tell whether a reranker was worth anything, I needed to know the baseline values for these queries. In addition to latency, I also needed to know the normalized discounted cumulative gain (nDCG) and recall. Capturing these metrics lets us objectively measure relevance alongside latency.

The baseline measurements were taken with K=10, meaning I took the top 10 results.

Retrieval	nDCG@10	Recall@10	Median latency
Vector	0.753	0.865	20.0 ms
Lexical	0.727	0.740	4.7 ms
Hybrid RRF	0.832	0.885	20.5 ms

nDCG measures how well a ranking puts relevant items near the top, with higher being better. It's not a measure of how many answers were right, though. Vector's 0.753 means that the rankings earned 75% of the score a perfect ordering would have, on average.

Recall ignores order. It tells you whether the correct memories made it into the top 10 at all. A 0.865 means that about 87% of the memories that should have come back did. Both of these metrics are evaluated against a known ground-truth set of data in order to be objective. Meaning I know ahead of time what the results should have been given the corpus and the query I was benchmarking.

You can see from these results that lexical is fast, but it's not super accurate. Vector catches more at the cost of latency, but still ranks them worse than I'd want. Fusing the two rankings had the best results, with minimal additional latency.

From the baseline results alone, it looks like a hybrid search will be the retrieval method to beat if latency doesn't drop off a cliff.

The enlightening reranker results

Now to run the same benchmark with the reranker. In Oracle, it's one more step at the end of the retrieval query. Retrieval narrows everything down to the top candidates, and PREDICTION() runs the cross-encoder against each one:

SELECT id,
       PREDICTION(BGE_RERANKER USING
         :query || '</s></s> ' || title || '. ' || content AS DATA) AS score
FROM candidates
ORDER BY score DESC
FETCH FIRST 10 ROWS ONLY;

The </s></s> is the delimiter the model expects between a query and a passage. I'm only pulling back the ids and the reranked scores for the benchmark. And you might notice that the reranking is done entirely in the database query - super cool.

Configuration	nDCG@10	Recall@10	Median latency
Hybrid, no reranker	0.832	0.885	20.5 ms
Hybrid + reranker, 10 candidates	0.832	0.885	585 ms
Vector + reranker, 20 candidates	0.830	0.896	1,258 ms

Surprisingly, adding the reranker to hybrid search didn't do anything to nDCG at all. The best reranked score came from vector retrieval with 20 candidates, and it still scored lower than plain hybrid with no reranker, at 61 times the latency. It found slightly more of the right memories, but ranked them a little worse.

I also tried giving the reranker more to work with. For vector retrieval, latency grew linearly with candidate count (1,258ms at 20 candidates and 2,604ms at 40), and nothing got meaningfully better.

That doesn't mean the reranker did nothing, though. I dug a little deeper and took the change for each query and resampled them 2,000 times to see how much the average varies depending on which queries you run. 95% of the resampled averages landed between -0.079 and +0.073.

And if we use that range, +0.073 would take hybrid from 0.832 to 0.905, which is a meaningful improvement. But conversely, the other end of the spectrum would drop it to 0.753, which is where plain vector search started.

Given the range, the results are inconclusive, at least with my sample size of 16 queries. The real effect could be a solid improvement, nothing at all, or even a loss, and 16 queries can't tell those apart.

This ended up being a big realization for me. Relevance is measurable, but how confident you can be depends on how many judged queries you have. A couple thousand iterations of 16 queries is still only 16 queries, and that's why the range is so wide.

Meanwhile, the latency hit is pretty obvious. It was nearly 30 times slower for every request. Now I had to ask myself when that latency is worth it.

Where rerankers make sense

Stay with me. Rerankers have lots of value despite what we've discussed so far. The underwhelming results are because I improved retrieval quality in a cheaper way. Let's look again at the baseline compared to the reranked version:

First stage	No reranker	10 candidates	20 candidates	40 candidates
Lexical	0.727	0.800	0.801	0.809
Vector	0.753	0.821	0.830	0.814
Hybrid RRF	0.832	0.832	0.829	0.817

Intuitively, the weaker retrieval methods got the most out of reranking. Lexical and vector each gained about 0.075, while hybrid gained nothing.

Which makes sense because a reranker doesn't retrieve anything. It reorders (or re-ranks 😄) what it's given. If your first stage is already pulling the correct memories into the pool in a sensible order, there's nothing left to fix.

Reranking vector retrieval with 20 candidates left me a little curious because it made the top five less alike. I measured it a couple of different ways (with embeddings and with plain word overlap) and verified the numbers. Thinking about it though, that is desired behavior. Remember that memory lookalike issue I talked about? The cross-encoder is separating the right memory from its lookalikes and pushing the lookalikes down. Which is exactly what I said I wanted.

The takeaway I want you to leave with is that you should rerank when you have one retrieval signal, and fuse when you have two. Doing both leaves you paying for additional latency you don't need.

My new approach

I started this wanting to know whether a reranker was worth the extra inference. After running through several rounds of benchmarks and tweaking queries, I'm not sure that was the right question. My approach to the problem is much more informed and now walks a different path than I expected.

When building a RAG pipeline for memory, I'll always filter first. Owner, expiration, whatever makes a memory eligible for you. Apply these in the query before anything gets ranked. Reducing the number of candidates means fewer memories to rank (plus, you know, security and quality and all that 😜).

Next, I'd add a second retrieval signal before adding a reranking model. If you have vector search, add lexical. If you have lexical, add vector. Fuse the rankings with RRF. Doing that with my tests took nDCG@10 from 0.75 with vector retrieval to 0.83 and only slowed my query down by half a millisecond.

After adding the second retrieval signal, I'd measure everything. Write down 15 to 20 real requests you'd give your agent, making sure to include some difficult ones and edge cases. Things like asking for exact keyword lookups, or for advice on something you've changed your mind on a couple of times. Keep those handy. The guide I was following says to rerun them every time you change chunking, the embedding model, or the reranker, and it's right.

Then you can try a reranker against your baseline. Try a few different candidate counts (:n in the query above). Check whether the quality difference is consistently higher across your queries and decide if the additional latency is worth the quality difference.

Luckily for me, what I built in my original post doesn't need to change much. I still have the same tables and the same reflection loop. I'm really just updating the database query that decides which memories the agent sees.

Try it yourself

The benchmark and instructions to try it all yourself are available in GitHub. It spins up Oracle AI Database Free in Docker, generates a synthetic corpus of 480 memories and 16 judged queries, loads the ONNX models, and runs the whole thing.

Everything we've discussed today ran on Free in Docker, capped at two CPUs and 2 GB. It's an intentionally small box, because we aren't running a capacity benchmark and none of these numbers should be used for sizing anything.

If you want to run this kind of evaluation on your own retrieval, start with Oracle's production RAG evaluation guide. It covers more than what we talked about here, like grading the generated answer and testing freshness, tenant isolation, and when the agent should refuse to answer. It's pretty solid.

Agent memory needs more than vector search. Turns out, just using vector search means you're asking a single similarity score to decide what's relevant, current, and yours. Fortunately for all of us, "more" is a lot cheaper than I assumed.

Happy coding!

You're still the bottleneck

Allen Helton — Tue, 22 Sep 2026 14:33:41 +0000

I went to a conference a couple of weeks ago. During the keynote, the speaker asked the audience "how many of you have 40 or more agents running at any given time?" About a dozen hands went up. He then asked "How many of you have 30 or more agents running?" Another dozen. This continued down to 20, 15, 10, 5, and eventually 1. Each time he asked, another 15-25 hands went up in the air.

It wasn't until he got to 1 that I sheepishly put my hand in the air. Luckily I wasn't alone, but it immediately got me thinking. I think we substituted one bad practice for another.

Remember a few months ago when tokenmaxxing was a thing? If you don't, good. You missed a terrible practice. People were taking pride in the amount of tokens they were spending regardless of the value they were getting. I still don't really understand why that was a desirable thing.

Tokenmaxxing has settled down significantly, but I fear that it's been replaced by something equally as bad: agentmaxxing.

A quick Google search will show you there's no consensus on what agentmaxxing means (honestly I thought I made it up until I Googled it). But what I mean when I say it is that developers are taking pride in the number of agents they have running at any given time. Jumping back to that question in the conference keynote, people are running upwards of 40 agents at any given time. 40?

So I asked the person who delivered the keynote how many work streams 40 agents represent. And he told me about 4.

Ok that makes so much more sense. He has four projects he's working in parallel, each one with a single orchestrator and about 9 subagents each actually doing the coding. That's better, but still a problem.

You can't multitask like you think you can

I have bad news. You can't multitask. When you think you're multitasking, you're actually context switching really fast.

The more inputs that demand your attention, the worse it gets. I've had times where I'm talking to my wife, one of my kids starts pulling on my shirt to show me something, the other kid starts crying in the other room, while the dog starts scratching at the door to go outside. At best, I'm going to somehow defer three of them to later and work through them sequentially. At worst, I'm going to walk into the other room and yell into a pillow because I'm overstimulated. 😅

Now, imagine that was four agents reporting back at the same time asking for approval or a design decision or reporting a CI failure on four different projects. With each task that lands back on your plate, you have to reconstruct the mental model of what that app is, what you're trying to do, where the agent left off, then you can act.

Would you be able to handle each one appropriately? Probably not.

Realistically, you'll probably hit the accept button without reading the context of the ask and tell your agents to just "fix it". That's what anybody would do when they make themselves the approval bottleneck across multiple unrelated contexts.

When you start blindly approving tasks, you're inadvertently delegating the decision-making to the model. At that point, what are you there for?

You are the decision maker

Code is cheaper than it ever has been. As a developer, the amount of your value that comes from typing code is dwindling. And if that was the only thing you had to do as a dev, the answer to "will AI take our jobs" would be a whole heck of a lot scarier.

Our value (among other things) comes from decision making. Weighing the risks, evaluating tradeoffs, and a nose for what looks right, to name a few. Some decisions are bigger than others, and the important ones need your undivided attention.

I love AI, but it makes it way too easy to make bad decisions. And you often won't know about it until you're fighting a production issue when it's significantly more expensive to change. If an agent spits out a 1,000 word response after a long-running task, your task is to make an informed decision using the provided context, ask questions, and think two steps ahead.

If you glance, hit accept, tab, glance again, and hit accept again, then you aren't a decision maker, you're a rubber stamp with a keyboard.

Attention is your greatest resource

I don't want you reading this and your takeaway being "Allen says 40 agents is too many." That's not it at all. I'm not advocating for anything like that.

What I am advocating for is not spreading yourself too thin. People have been saying for years that attention is one of the scarcest and most important resources we have.

In 2026, everything wants your attention. Humans have gotten really good at creating things that steal your attention away from something else. With coding agents, it takes almost 0 effort to bring in another work stream so you can "force multiply yourself." Just open another terminal, type in claude and you're (basically) done.

Protect it. Treat your attention like the scarce resource it is. Keep the decision-heavy workstreams to a minimum, just 1 or 2. Then run as many low-judgment streams as you want next to them. But keep your attention on the tasks that need it. Don't split your attention across decisions, because when everything is a priority, nothing is.

Next time somebody asks you how many agents you have running, tell them that's the wrong thing to be counting. The better question is how many decisions you're actually making.

Happy coding!

Payment is authorization

Allen Helton — Thu, 10 Sep 2026 18:52:30 +0000

I can't tell you how many times I've needed to do something simple like remove the background from an image or convert a file from one type to another but wasn't able to do it. While these feel like simple, common things your OS should just handle, they require a service to do them. Luckily, I'm usually able to Google the problem, find a website dedicated to solving that one problem and do it.

It's super nice because I land on a website like remove.bg, drag my image onto the page, and the background is removed. I don't have to log in, it's not tracking me (that I'm aware of), and it just works. The API version of this is something I'd call a utility API. It simply does a thing and gives you a result.

Compare a utility API with what we're growing more and more accustomed to. I click on a marketing site, sign up, pick a plan, confirm my email address, then start making requests. You know what I call that? Friction.

And in a world where AI agents are becoming increasingly important consumers of APIs, friction is holding us back.

There's a lot of attention right now on agent identity, and for good reason. Agents that read private data, modify infrastructure, operate business systems, or act on behalf of people absolutely need durable identity, delegated authority, auditability, and policy.

But I think we're going a little too far with it.

For a lot of utility APIs, identity doesn't need to be part of the authorization decision. Instead, it just becomes who we charge. Let's dig into that a little bit.

Bring your own access

To perform a basic utility, it's likely that an agent will discover your API at runtime, perform a single operation, and never call it again. So requiring it to create an account and establish a permanent relationship with your service before doing 2 cents of work feels wrong. What if instead, your API provided a pay-as-you-go on-demand model (something my readers should be familiar with). x402 is an open protocol for payments over HTTP, led by the x402 Foundation under the Linux Foundation.

The flow looks like this:

The caller doesn't necessarily need to create an account first. It requests the resource, gets back machine-readable payment details, pays, and retries with proof of payment.

Which, when you think about it, means that payment alone can be the authorization decision. While yes, you could still add a layer of AuthN/AuthZ on top of the API, the x402 protocol doesn't require it. This removes that friction we were talking about earlier and lets agents pay and continue (as long as they have access to a wallet).

This is a very different access model than we're used to and makes many developers feel a little uneasy because it goes against the paradigm we've been handcuffing ourselves to for years. But if you ask yourself five whys, what we're really after is payment. And this lets us cut out a lot of extra work to get to it.

What does this look like?

My favorite way to learn is by doing, so let's get into what implementing x402 in a serverless API would look like. Because I love the idea, but this sounds tricky to implement once you actually get down to it. If payment is part of authorization, do you put that in a Lambda authorizer?

You shouldn't. Lambda authorizers are built around the idea that an authorization decision can be reused. API Gateway evaluates an identity source, decides whether that caller may invoke the API (and specifically which endpoints inside of it), and can cache the result for later requests.

Lambda authorizers work well for things like bearer tokens because the claim being evaluated usually authorizes users for an entire session. This makes the generated policy cacheable and an easy deciding factor for where it lives.

Of course, you can disable the cache with a 0 second TTL, but the authorizer sits at the wrong point of the request lifecycle. It runs before a request hits your API. For payment authorization, we need to go a little further down.

A payment is generally going to be for a single call. If you want to call it 10 times, then you need to pay for it 10 times. Nothing here is cacheable. For this architecture, that leaves us with one option, really. Bringing payment logic into the app code 😬. Well, technically you can also use AWS WAF AI traffic monetization, but more on those limitations in a bit.

The API controller

For years, the serverless community had a debate on whether you should build your APIs as a "lambdalith" where a single Lambda function acts as your API controller and serves your entire API, vs a single Lambda function per endpoint. I was team "function per endpoint" for years, but have come around to the idea that I was wrong.

I won't jump back into the philosophical debate here, just know that we're designing a single Lambda function to serve the entire API using the Lambda Powertools HTTP event handler to define our routes and do input/output transformations for us.

It also sets us up for treating x402 as middleware. Lambda Powertools is great at a lot of things, and middleware is where it's golden. Since we don't really want payment processing logic littered through the app code, surfacing it as middleware is the right abstraction. Doing it this way gives us some handy conveniences as well.

Route pricing

If you've ever built an API with Express, Hono, or Lambda Powertools, you'll be familiar with the syntax:

app.post('/generate', async () => {
  // Generate something
  return { message: 'success!'}
});

This defines a POST /generate endpoint that does some work and returns a success message. But let's pretend like we wanted to charge money to call it. All we need to do is add some middleware to the route:

app.post(
  '/generate',
  [ x402.paid({ price: '$0.05', description: 'Create an image of something' })],
  async () => {
    // Generate something
    return { message: 'success!'}
  }
);

Isn't that slick? The handler has no payment logic, nor does it parse payment headers, talk to a facilitator, or know anything about wallets.

The middleware verifies that the request has acquired the right to execute, runs the handler, and handles settlement around the request lifecycle.

The coolest part of the whole thing is, declared this way, the price is essentially route metadata. It just kinda fits in with method, path, validation, and rate limits as properties of a route. Looking at what that implies for documentation, that could easily turn into something documented in your OpenAPI spec or tool definitions so agents don't need to receive that first 402 Payment Required status code.

If an agent discovers a capability and can see its price as part of that discovery, then the access model becomes much easier to reason about. Instead of discovering an API and immediately starting a human onboarding flow, the agent can evaluate the cost as part of deciding whether to use it at all.

NOTE - the middleware I used in the example above is an open-source project I built that's available via npm. It's specifically designed to work with Lambda Powertools.

Error handling

You know what would be bad? If a caller paid for an operation, it failed, and returned a receipt saying you paid for the failure. Luckily, the payment flow we're using is designed to prevent that from happening, but it's not bulletproof.

If we jump back to the sequence diagram from earlier, you can see an important order of operations that should prevent paying for failures.

verify payment
→ run handler
→ handler succeeded?
    → yes: settle
    → no: do not settle

This is how the AI monetization feature in AWS WAF approaches the problem, but it's not perfect. WAF runs at the edge, and pragmatically it is inspecting a status code. That's (generally) fine for selling a document, but not robust enough for selling an operation. An empty generation, a model refusal, or a search that found nothing could all still return a 200 OK, which means that payment settled and the caller got nothing.

Middleware runs in-process, so it can decide on the actual result:

x402.paid({
  price: '$0.05',
  billable: (result) => result.imageUrl != null
})

In the above example, if the response body doesn't contain an imageUrl property, they would still get a 200 OK, but they wouldn't be charged. This is a meaningful improvement over status code checking and gives you endless levers for fairly charging your callers.

But what happens if settlement succeeds and the response fails to return to the caller? The agent retries. The payment-identifier extension is your best friend here. The caller can reuse the same payment id, allowing the API to return the cached result without charging or executing the operation again. Of course, that means you have to persist the result somewhere durable and associate the id to the original request (sounds like idempotency to me). It's still your job to implement this correctly. 🤷‍♂️

This is the next frontier in AI

Our industry has done a phenomenal job the past several years rocketing us through uncharted territories. In what feels like no time at all, we've gone from pioneering chatbots to agents to memory to security to multi-agent collaboration and everything in between. But we're just starting to scratch the surface on agentic payments.

There are still many hard problems to solve, like transaction latency and reputation tracking (among other things). Not to mention the elephant in the room when it comes to humans handing over control: trust. Anecdotally, I've already heard stories about how people are implementing spiked prices via x402 specifically for bots (by reading User-Agent headers), like raising the price from $200 for a pair of sneakers to $1,200 in hopes of surpassing allowed price limits without a human in the loop.

Not to mention the discovery problem. Updating your APIs to accept x402 is step one. But with software, the adage "if you build it, they will come" is unfortunately not true. At all.

Coinbase has taken a stab at a registry with their Bazaar catalog, which gives us a great starting ground. But for long-term success, I think we'll need something a bit more built-out. More on this in a blog post to come. 😉

For now though, what I want you to take away is that API consumption has fundamentally changed. And how we look at it going forward needs to change as well. Maybe we rethink the multi-step authentication mechanism and opt for a more streamlined "auth by payment" protocol (when applicable).

It's already relatively easy to retrofit your APIs with x402 if you're using Lambda Powertools. Honestly, the core x402 package makes it easy no matter what you're using. But the more important part here is for you to start thinking about your APIs differently.

Do you have a utility API? Is your current auth flow absolutely necessary? Are you able to charge on an operation level for your API capabilities?

Try it out. Tell me what you think. Let's pioneer something new.

Happy coding!

DynamoDB Streams is not an outbox

Allen Helton — Tue, 25 Aug 2026 16:39:57 +0000

Hopefully you didn't notice, but a couple of weeks ago I sent an issue of my newsletter that never actually got emailed. My dashboard looked fine and it was marked as published, but the email never hit my inbox (or any of yours 😬).

I didn't notice until two days later when I went to look at the weekly statistics and found nothing. Just a Sent status and a bunch of 0's on the screen.

A quick browse through the code didn't turn up anything obvious. Emails are sent via an async handler triggered by an EventBridge event while the status of the newsletter is marked separately in DynamoDB, a classic "db save and domain event publish" pattern. We've all done it, it looks something like this:

await ddb.send(new UpdateItemCommand({ ... }));
await eventBridge.send(new PutEventsCommand({ ... }));

Despite its simple appearance, this is a nasty anti-pattern. Formally known as the dual write problem, if those two actions aren't contained in a single transaction, you're asking for trouble. Why? I'm glad you asked.

Imagine the write succeeds, but the publish doesn't. Data is updated but the follow-on actions that react to the domain event never run (like your email never being sent 🙃). Things fall into a bad state and you (maybe) end up stumbling upon a problem days later.

Ok, so flip them. Put the event first and the data save second. But if the publish succeeds and the db save fails, then you risk acting on stale data, which depending on your domain, is much worse.

Before you ask, no, serverless doesn't make it better. In many cases, serverless makes us too laissez-faire to solve it the "write" way (get it?).

What you're probably doing

Let's say you already knew not to do the data save and the event publish at the same time. I bet I know what you're doing instead. It's a pattern AWS has recommended for years and is all over Serverless Land.

You do your save to DynamoDB, let the change data flow into DynamoDB Streams, process it in Lambda, then fire your domain events from the stream processor.

When it comes down to it, this is legitimately better, and it's the same thought process behind going storage-first. You persist it first, then process it asynchronously. That makes the database write the event trigger, so now your domain events are tightly coupled to the corresponding data change. Plus, DynamoDB Streams retries automatically (and even bisects batches when configured) if there's a processing failure.

Sounds like our dual write problem is solved, right? Wrong. Kind of. It's complicated. 😅

DynamoDB Streams records are change data capture, not domain events. The Lambda function you write to handle them receives a NewImage and OldImage record, and it's up to you to infer what happened upstream. Smells like leaky business logic to me. Before you know it, you have this thousand-line file that looks at 40 different data shapes to determine which event to publish and which transformation to perform. As someone who's found themselves in that hole before, believe me when I say that is a nightmare situation.

Let's take it a step further. A business operation could (and probably does) affect multiple DynamoDB records. When I publish my newsletter, I create a DynamoDB record with a short TTL for each email address that was successfully sent to. If I am processing entity changes in a stream handler, I'd be up to my ears in duplicate newsletter.sent events because I'm working on individual records and not an aggregate operation. That ultimately leads to workarounds with atomic counters and lastAction flags that make it overly complex with little value add.

And most importantly, you don't know what did or didn't publish. Streams retain data for 24 hours, and if your publisher was broken for a day because of an outage or a rogue IAM change, you're out of luck. Even if you opted for something like Kinesis, which has 365 days of data retention, that's not the point. These mechanisms are change data, not the intended effect on the system.

DynamoDB Streams are a wonderful piece of engineering that gives you retries, but not the durability of intent.

What you should do instead

Time for a bit of a mental shift. We need to prioritize domain events. If an event is important enough to publish, it's important enough to persist in a database.

This is where the outbox pattern comes into play. Instead of deriving your domain events from the data change in a downstream handler, you save the event in an outbox table in a transaction along with the data mutation. The transaction guarantees both the domain event and the data change will save together (or neither).

With DynamoDB, that could look like this:

const eventId = ulid();

await ddb.send(new TransactWriteItemsCommand({
  TransactItems: [
    {
      Update: {
        TableName: 'newsletter',
        Key: marshall({ pk: issueId, sk: 'issue' }),
        UpdateExpression: 'SET #status = :sent',
        ...
      }
    },
    {
      Put: {
        TableName: 'outbox',
        Item: marshall({
          pk: issueId,
          sk: `EVENT#${eventId}`,
          status: 'PENDING',
          pendingBucket: 'PENDING',              // GSI1 PK, removed on publish
          pendingAt: new Date().toISOString(),   // GSI1 SK
          source: 'newsletter.service',
          detailType: 'newsletter.sent',
          detail: { issueId, date, idempotencyKey: eventId }
        })
      }
    }
  ]
}));

Funnily enough, the architecture doesn't change with this pattern. You're still doing Lambda (or whatever compute) to DynamoDB to a stream to another Lambda function. But this time the downstream Lambda function is processing the event you saved, not the mutated data. It doesn't have to infer anything.

AWS does a fantastic job in their transactional outbox design pattern docs showing you how to setup and implement this pattern yourself. So I won't go into more of the code here. But I will address some questions I had the first time I built this.

What do I do with an event after it's published?

After you publish an event in your stream handler, it's up to you to either delete it from the outbox or mark it as published. Personally, I prefer to mark it as published and set a 30 day TTL. This means in the stream handler, we're hitting the dual write problem again because we are both publishing an event and updating the outbox with a published status.

This one is ok though 😉 and it brings up an important point we haven't covered yet. Idempotency is crucial for success with this pattern. The outbox pattern trades duplicate delivery in exchange for never losing durable content. If your publisher fails to send the event or update the database with a published status, that's fine! Do it again. The important part is that the intent wasn't lost.

Can I replace the stream handler function with EventBridge Pipes?

Fewer pieces, easier system to maintain, right? In this instance, the Lambda function is the better solution over Pipes. If you were only publishing an event to EventBridge, you could easily filter the stream to the relevant outbox records and forward it straight to a bus. But because you also need to write back to DynamoDB to update the status of the event, the function wins. A pipe can only perform a single operation (outside of enrichments and transformations), but a function can perform many.

Does this guarantee ordering?

Nope. The outbox pattern is designed for correctness only. Eventually consistent state across a distributed system. This does not tackle other hard distributed system problems like ordering or exactly-once delivery. Quite the opposite in fact. While Streams do process change data in the order they happen, it's per shard (not globally), and you can't guarantee ordering when it comes to retries. And as soon as you drop an event on EventBridge, you lose any sort of ordering.

How does this guarantee events aren't skipped?

If you implemented this exactly as described, it wouldn't. It gives you the normal retry path you're already used to, but still has that 24-hour clock because of the stream data retention. The important piece that's missing is a scheduled reconciler. It's usually another Lambda function that queries a sparse GSI for anything still pending from more than five minutes ago, which is long enough that the stream handler has had its shot, and republishes it. So your full flow would look something like this:

Remember this

There is no clever workaround to the dual write problem. You can't order the calls in a way that mitigates the risk, you can't retry around it, and you can't/shouldn't move it into a stream handler and hope.

But you can stop having two writes. Well, two writes, but one transaction. 😅

Remember, what we're solving with the outbox pattern is a correctness problem. If someone asks "how can you guarantee the system is eventually consistent, even under backpressure?" this is what you reach for. It took me years to arrive here naturally and I've been accidentally building holes in my systems longer than I'd care to admit.

This pattern does make your architecture more complicated, but the trade-offs are often worth it, especially at scale. In exchange for 2x the WCUs, you get a solid way to store and act upon intent in your system. Anyway, take a look at how you're publishing events and see if there are any unexplained missing events in your system. If there are, you know what to do. 😉 That's what I did and you should be reliably getting your newsletters from now on.

Happy coding!

Send your events later

Allen Helton — Wed, 29 Jul 2026 20:26:23 +0000

I build a lot of automated workflows. Across the suite of apps that powers Ready, Set, Cloud, I have a dozen or so mature, long-running automations that do some work, wait for something, then do more work. A great example is my newsletter. I often write it on Fridays and publish it on Mondays. After I open a PR for an issue, my backend service does some initial work, but it waits until Monday to send the emails and start analytics tracking.

In the past, I've used the Step Functions Wait state for this because it's the exact use case it was designed for. But as part of a big refactor I did to separate Ready, Set, Cloud logic from the generic newsletter logic, I needed to find a simpler way to schedule work - ideally as part of my CI/CD pipeline.

I thought using the AWS SDK in a script to create a one-time EventBridge schedule that published an event at a designated time was my best bet. But that brings in some implications with IAM and passing roles to the scheduler in order to invoke a PutEvents call. It ended up being way too messy for a simple CI script.

But I had an epiphany that immediately made me think, "I should write about this" because it's very simple but not always obvious.

The surprisingly-simple-yet-not-obvious solution

I moved the EventBridge Scheduler event publish behind an EventBridge rule. Anything that can call PutEvents can now ask for a deferred event.

It's essentially a one-time loop with a customizable wait step in it. Here's an example payload of an event to get my static site to rebuild because future-dated content doesn't render automatically.

{
  "Source": "readysetcloud.app",
  "DetailType": "Schedule Event",
  "Detail": {
    "at": "2026-08-03T09:00:00",
    "timezone": "America/Chicago",
    "name": "newsletter-rebuild-227",
    "event": {
      "detailType": "Trigger Site Rebuild",
      "detail": { "issueNumber": 227 }
    }
  }
}

Publishing the above event will create a schedule that triggers at 9 a.m. CDT on August 3. The schedule will publish a Trigger Site Rebuild event back onto the bus, which is wired up to a rule that kicks off a build in Amplify. Nifty, right?

Just a little bit of logic

While I normally try to skip Lambda in this type of integration, I needed it for a couple of reasons. One, you can't set the EventBridge Scheduler as a target of a rule 🙃. And even if you could, I would have lost my ability to maintain schedules. If I needed to reschedule an event for a different time, I would have been out of luck.

So I wrote a Lambda function as the target that will upsert the schedule, validate the event (WHICH YOU SHOULD ALWAYS DO), and transform the provided time to the format the scheduler expects.

As a bonus, the Lambda function also recognizes when the scheduled date is in the past, and will publish the event immediately or throw it away, depending on a parameter optionally provided in the event payload.

So in this case, it made a lot of sense to add the function in the middle, which increases the complexity of the build, but the tradeoff is a richer experience.

Build your own platform

A few years ago, I made what I called my serverless toolbox, with some tools I could use across all of my apps that ran in the same AWS account. What I didn't know then was that I was essentially building a platform for all of my applications to run.

I've since abandoned the toolbox and have turned my attention fully toward building a platform my apps run on. It's the peak of DRY (Don't Repeat Yourself) and in this case, is incredibly useful. I've added this scheduling primitive to my platform repo so it can be used everywhere - my newsletter service, blog service, and all the other services I've built that power my website.

I only need to solve this problem once. Time and time again I've reimplemented things that worked in the past only to find myself with 4 versions of the same thing and at least one of them stale. So I make an intentional effort to consolidate the shared work and make the entry point to it as simple as possible (what's simpler than calling PutEvents?!).

Anyway, this is intentionally short and sweet. I was excited by the idea of delaying an event with nothing more than PutEvents, and wanted to share. For the full source code, you can check out the function here or view the entire PR to my platform app.

Happy coding!

PSA: That probably doesn't need to be SaaS

Allen Helton — Thu, 25 Jun 2026 21:35:21 +0000

I've published a weekly newsletter every Monday for the past 4 years that showcases content from the tech community. The sole purpose of all 221 issues has been to highlight creators who educate the community and teach you a thing or two every week.

Writing it for so long has tuned me into creators, content styles, trends, and outputs of the greater tech community. And lately it feels like thoughtful technical writing has become harder to find.

Which is odd, I haven't seen this kind of behavior from the community before. But it hit me the other day as I was scrolling through LinkedIn. Everyone is launching solo-SaaS products. Hyper-specialized web apps built to solve a niche problem, now wrapped in pricing pages and marketed as a service.

Developers are spending their time launching these micro-SaaS apps instead of writing about what they learn. We're seeing Product Hunt launches and marketing pages with colorful gradients where the blog posts used to be.

What happened?

AI. AI happened.

Specifically coding agents removed barriers preventing us from launching products in our free time. The time it would have taken to research, build, tinker, polish, rebuild, and polish some more has been replaced with a plain-text "Make it do this" and walking away from our laptops (or cell phones 🤯).

Don't get me wrong, I love how easy coding with AI agents is. But all-too-often I'm seeing developers be too trusting and building something they don't truly understand and moving on before lessons are absorbed.

It's all very cool and I still have a hard time believing this is real life. I've had times where I'm on my phone using a website I built, notice something I don't like, open the Claude app and describe the change I want, and 5 minutes later merge the PR from the GitHub app and it's live.

But it's having a ripple effect that might be hurting the tech community more than we realize.

Building like a SaaS isn't the same as running one

Throughout my career, I've been tempted to turn one of my projects into a fully-functioning SaaS. Every time I start down that path, I am quickly reminded that I romanticize running my own business.

Building like you're shipping a SaaS is one of the best things you can do for your career. Multi-tenancy as a design principle teaches you data isolation, auth, data modeling, and observability. All skills you need at your day job. You should absolutely build that way (please do!).

Taking it to production is a whole different ballgame. You need a business strategy, marketing, pricing, support SLAs, and about 40 other things that aren't engineering related. The stuff that we as developers conveniently forget when we tell ourselves "All I need is 100 users at $50 a month and I'm set." The stuff that takes ongoing commitment. Even on days you don't feel like supporting it, somebody else is depending on it.

My friend Elias Brange is a fantastic engineer. The kind of dev who builds with care and intentionality. He launched a game called Daily Wars on Product Hunt last week and posted this after:

Getting the code out the door was easy, thanks to AI. Getting someone to use it was the hard part.

The moment you decide others will use your game/site/service/thing, the design changes too. You can't hardcode your preferences. You add configuration pages, customizable timers, a place to dynamically store credentials. That 10/10 solution for your problem becomes maybe an 8/10 for everyone because it has to "just work" for use cases that aren't yours.

Then (maybe) users start showing up. Ten of them. Heck, let's say fifteen. They have questions you didn't anticipate. One of them gets locked out at 2 AM on a Sunday and emails you about it. Someone else hammers your API with thousands of requests per second because of a bug in their code while integrating. You're checking your dashboard between meetings at your day job.

And just like that your fun side project goes from a hobby to a job you didn't apply for.

Remember Saturday mornings spent tinkering, trying to figure out how to center a div without asking AI? Those days are gone.

Build for yourself

Instead of trying to launch your first, second, or Nth micro-SaaS, why don't you build for yourself? Make it something that serves an audience of one (you) but build it in a way where you still get meaningful practice in software engineering.

Open source the code. Keeping it open allows for a level of transparency that a fast-shipped SaaS app doesn't have. You're not signing up to maintain a project forever or triage issues from strangers. Treat it as a type of receipt. "Here's where the lesson came from. Clone it if you want, but I'm not your support engineer 😅."

Then write about what you learned. Honestly, this is the part you learn from the most. Writing a blog post or creating a video that teaches something you learned the hard way forces you to really understand what you encountered. And the best part? It costs nothing to maintain. It doesn't page you on a Sunday morning. And it lands you back in the seat you used to sit in, as a peer in the community instead of a vendor pitching to one.

Not everything needs to make you money

As a millennial, I grew up under the "monetize your passion" life advice and it's a conscious effort to not do it. It's terrible advice. I remember the first time I heard the phrase not everything you do needs to make you money and felt like a huge weight was lifted off my shoulders. Doing something for fun instead of doing it to earn beer money is liberating. With AI agents making the coding part of SaaS building trivial, it's harder than ever to simply build for the fun of it.

I suggest a new mantra for developers for 2026: not everything you build needs users.

The engineering you do for yourself still counts for something. The system you architect well still goes a long way. The hobby that produces nothing more than a blog post and an excited conversation is doing more for the community than the micro-SaaS that produces revenue from twelve strangers.

The next time you finish a side project, write the post before you write the pricing page. You might find that it was the output you were after all along.

Happy coding!

Your AI agents are a security nightmare

Allen Helton — Fri, 27 Mar 2026 15:04:48 +0000

I've noticed something about the tech industry over the past couple of months. Through the use of agentic code editors, developers are building small, custom solutions to problems they're having. Recently, there's been a surge in developers writing their own content platforms, abandoning bigger platforms like Hashnode and Medium. While I think this is genuinely amazing, we're creating a gap that we have to address sooner rather than later.

That gap, of course, is maintenance. Even though AI coding is genuinely good now, new development is only a fraction of the software lifecycle. What happens when you get your app to prod and it suddenly breaks?

Do you know the codebase like you did back when everything was hand coded? Would you even know where to begin troubleshooting if your API started returning a bunch of 500s?

Let's assume the answer is yes to both of those. Now let me ask the hard-hitting one: do you have time to troubleshoot and fix issues? This is a side project, not your day job. Many of us don't have the luxury of dropping the “real work” tasks for a few hours while we figure out what's going on.

I'm no exception. I've been building Good Roots Network, an app that connects local gardeners with people in their communities who need food. Growers list produce, gatherers like social workers and non-profits request it, and the system handles the coordination from availability through pickup. It's API-driven from the ground up. State machines, role-based access, real-time inventory transitions. Real production infrastructure running in my personal AWS account.

There's a lot of surface area to cover for a solo project. And when something goes wrong, I don't have the bandwidth to dig in and figure out how to get it back on its feet - which means I'm potentially blocking people from getting access to fresh produce for themselves or others.

So I built an agent to be on call.

Before I continue, thank you to Teleport for sponsoring this post. All opinions are my own.

Building an agent that runs locally

The oncall-agent is a local, persistent process that automatically responds to production incidents. It subscribes to a Momento topic for real-time notifications and when something fires, it gets to work on its own. No need to wait for me to react.

The agent has an internal state machine: receive the incident signal, authenticate against AWS, investigate, then post a summary to Slack. During investigation, the agent loop runs on Amazon Bedrock, using the AWS CLI to query CloudWatch logs and metrics, inspect Lambda functions and DynamoDB tables. It also has access to my source code and deployments via a GitHub app to diagnose root cause and open a PR with the recommended code change.

I run it locally (rather than as a deployed service) for a couple of reasons. First, I specifically want it to make code changes if they're needed, and pulling the source into a Lambda function in my account doesn't seem like a great use of resources. Second, and more importantly, it touches production infrastructure directly. I want full control over the process, and I want the blast radius of any security issue limited to my machine, not a cloud-hosted process that's running 24/7.

By the time I check Slack in the morning, the investigation and proposed solution is done and awaiting my approval.

Keeping the agent secure for a change

It's tempting to just open up the doors on your machine and give the agent everything it needs to be successful (and then some). But as we've seen throughout 2026, doing so is a security nightmare. I don't want to be another statistic who let an agent wreak havoc on my system because it was easy.

When it came time to give the agent AWS access, my first instinct was to put an access key in the .env file. Long-lived credentials. It's the familiar pattern, it takes 30 seconds, and it works. But the more I sat with that approach, the more it felt like I was solving the wrong problem.

My issue with that approach was that the agent didn't really exist as an identity in my system. It was just a privileged background process acting on my behalf with standing access.

So, I reconsidered the agent as a first-class principal. If it was going to investigate production issues autonomously, it needed to prove who it was at runtime, operate within a scoped session, and leave behind a traceable chain of actions.

Once I started thinking about the design this way, long-lived keys stopped making sense entirely. 👎

Identity is not the same thing as access

With this in mind, I needed something more than a vault. I needed a way for the agent to establish its identity at runtime.

That's where Teleport fits into the design. Instead of access tokens being defined in a config file, the agent starts every investigation by authenticating into a new session. From there, it can request scoped AWS access and perform specific actions with clear attribution.

What I can now see in CloudTrail is a stable GUID that uniquely identifies the agent as its own principal. I can filter to that identity and see everything the agent has ever touched.

If something unexpected shows up, I know immediately it was the agent, and not a human or another process. That's already a huge improvement over a shared access key where attribution is basically impossible.

Now, I have a legitimate session-bound identity operating on my system. Something that only has the permissions I gave it, but more importantly, is independently auditable.

When the agent runs tsh login, I set it up so I have to approve the MFA challenge myself. Fully autonomous credential rotation via Teleport Machine ID is available–and it's the natural next step–but I'm not there yet. I want a few more practice reps with the agent before I remove myself from the auth flow entirely. Trust has to be earned, and I'd rather manually approve an MFA prompt than wake up to an agent that went sideways with no human checkpoint anywhere in the chain.

Agent security posture is something I'm really honing in on. I'm changing my mental approach from "does this agent have the right key?" to "was this agent authorized to do this specific thing, in this specific context, at this specific moment?" It might feel like a subtle difference, but it's how we need to be thinking before we give agents the keys to our production systems. And a static credential sitting in an env file isn't the way to answer it.

This isn't a side project problem

The most common pattern I see these days is the same one I almost fell into: drop a long-lived token into the environment, grant it the permissions it needs, and ship it. Even if you do it as a “temporary stop gap” and swear you'll come back to fix it, you won't.

It always feels like it's fine until something unexpected happens. The problem with unexpected behavior in autonomous systems is that you need more audit resolution, not less. Imagine an agent made unusual calls to your production services. Was it a bug? A prompt injection? A legitimate investigation path you just didn't anticipate? With static credentials you can't tell. The trail goes cold at "this IAM principal made these calls."

Agentic AI is exposing some serious issues with the access model many of us are running. IAM was originally designed around humans doing episodic work. Even automated pipelines are episodic in that they run, finish, and stop. But that doesn't map cleanly to processes that persist indefinitely, reason autonomously, and act faster than the blink of an eye.

When I think about what agents need to operate safely, I land on a few things. They need a cryptographic identity of their own, meaning something that can be uniquely identified (much like an individual developer). They need access that's scoped and enforced at runtime–not granted once like an API key, and left open. They need some way to detect when they're behaving outside expected parameters, or when the operating context has been tampered with. And if they're coordinating with other agents or MCP servers, those connections need to be governed, too (you can't have a secured agent talking to an unsecured tool).

These are all things Teleport is building with their Agentic Identity Framework. My agent only scratches the surface of all this with its short-lived credentials and isolated identity. But starting there forces you to think about the agent as a principal that needs its own security model, which is the right mental shift we need heading into mid-2026.

This probably sounds a little familiar

We've all seen this play out before in different forms. When containerization became mainstream, teams that had good practices with VM security often skipped the basics on containers because it felt like a "smaller" problem. When Lambda came out, teams that carefully managed server permissions dropped service roles for wildcard policies because it was a new execution model and they were moving fast (hopefully I'm not the only one who did this 😬).

Agentic AI is the same thing happening again. A new execution model, a lot of excitement, and a temptation to skip the identity and access fundamentals–because it feels like a side project, a POC, or just something you're trying out.

The lesson is the same every time: the important components in production are still important even if the thing running is new, or small, or experimental. An agent with standing access and no attribution is a liability, whether it's running inside a Fortune 500 or on your laptop handling incidents for a community gardening app.

The oncall-agent repo is open source. The Teleport integration is in the state machine if you want to see how it fits together in practice. And if you're experimenting with agents at all, it's worth triple-checking how they establish identity and access production systems. Move a small workflow to session-bound credentials to help you make better architectural decisions and make your automations easier to trust.

Tools like Teleport make that change practical without completely reworking your stack. It's a small change in implementation, but a big change in how much autonomy you're willing to give your systems.

Happy coding!

When serving images from S3 stopped being good enough

Allen Helton — Wed, 07 Jan 2026 15:12:39 +0000

I published my first blog post 7 years ago. I wrote on Medium for about a year before I built Ready, Set, Cloud. For most of its life, the site hasn't had much of a facelift or performance updates. It's primarily served as a home for my writing, newsletter, and podcast.

I've been running this site for six years. I'm usually the kind of person who can't leave things alone for long, yet this has been largely unchanged for years. It worked. Until it didn't.

When I finally took a look at site performance, something was painfully obvious. Images. PageSpeed Insights showed that large, unoptimized images were dominating load time. Single file sizes served directly from S3, no format negotiation, and no caching were starting to add up.

That setup wasn't wrong when I built it. It was a perfectly reasonable tradeoff at the time. It was simple, low maintenance, and honestly, good enough. The site had a couple dozen readers, and small-scale simplicity won. But as the site grew, both in content and audience, performance expectations changed. Serving raw images straight from S3 no longer cut it. The site outgrew its initial build.

What I needed from image delivery

The knee-jerk reaction is to jump into S3 and manually optimize a bunch of images. That fixes a symptom, but it doesn't solve the underlying problem. I needed to step back and rethink what image delivery should look like for a content-heavy site serving tens of thousands of monthly visitors.

Manually resizing images, converting them to web-friendly formats, and deciding which size to upload per post was never going to scale. More importantly, I didn't want to change my workflow. I wanted to continue to write, publish, and move on without adding in layers of performance checking.

A good system should make that possible. It should optimize images automatically in the background. It should serve modern formats like WebP when the browser supports them. It should provide multiple image sizes so mobile devices aren't downloading desktop-sized assets. And it should be aggressively cacheable, because image delivery is a solved problem, and CDNs already do this exceptionally well.

Thinking about these requirements, I realized I wasn't really looking to build a new image optimization tool (like I would have done in the past). Instead, I was updating the site's image handling capabilities so these decisions were made once, centrally, in an industry-standard way.

Fitting a CDN into the workflow

I upload images to Ready, Set, Cloud using a small JavaScript script that lives in the repo. It takes a file prefix, scans a local folder for matching files, and uploads them directly to S3. There's no web UI and no automation that tries to interpret content. It's simple by design, and I wanted to keep it that way.

That simplicity turns out to be a feature. Because images already flow through S3, the system can react to uploads without changing the workflow at all.

Image uploads automatically trigger a Rust Lambda function (through EventBridge) that converts the original image to WebP and generates a handful of standard sizes. Those optimized versions are written back to S3 alongside the original.

This does a few important things. First, it makes image optimization deterministic - meaning every image goes through the same process every time. Second, it keeps work off the critical path. All of this happens asynchronously and doesn't interfere with the rest of the publishing workflow like cross-posting, writing analytics, or scheduling.

Finally, I added a CloudFront distribution in front of the S3 bucket. That keeps delivery fast and predictable by serving cached assets from points of presence around the world. Because it points at the existing bucket, there's no data migration involved. Image delivery improves without changing where anything lives.

Serving the right format without changing URLs

Once image processing was handled, delivery became the next concern. All optimized assets were already in S3, but I wasn't interested in changing 2,000+ image links across existing content just to take advantage of them.

The answer was to push that behavior into the CDN. By updating the distribution to look for WebP support in the Accept header, requests for images could be routed to the optimized versions without changing a single URL. Modern browsers already advertise their supported formats, so content negotiation becomes a routing concern rather than a frontend one.

A CloudFront function runs on the viewer request event and rewrites the request path whenever the browser advertises WebP support. It doesn't check whether the WebP file exists - that's resolved later when CloudFront fetches from the origin (the S3 bucket). The important part is that the rewrite happens consistently at the edge, and the resulting response is cached, so subsequent requests follow the same path.

Letting the browser choose the right size

Using WebP was only one part of the solution. Once multiple sizes of every image existed automatically, the next question was which one to serve. Mobile screens don't need desktop-sized images, and high-DPI displays can benefit from larger ones. Not to mention saving on load times for thumbnails on the home page.

Using srcset allows the browser to make that choice on its own. The markup stays the same, there's no runtime logic, and each client downloads only what it actually needs. The responsibility for choosing the right size moves to the client, where that decision can be made with full context.

<img
  src="/images/diagram.png"
  srcset="
    /images/diagram-480.webp 480w,
    /images/diagram-960.webp 960w,
    /images/diagram-1600.webp 1600w"
  sizes="(max-width: 768px) 100vw, 768px"
  alt="..."
/>

Ready, Set, Cloud is a static site generated by Hugo. In my case, adding srcset support meant overriding the render-image hook and adding a small amount of logic to create the additional attribute.

Performance improvements and an unexpected bonus

My primary goal with all this was to decrease load times so my site felt snappy without adding anything to my existing workflow. With the image optimization component plus the CDN for cached delivery (and the client-side srcset addition), I'm pleased to say page payloads dropped significantly. Pages render faster, Largest Contentful Paint (LCP) improved, and overall performance is more predictable.

On my homepage, the First Contentful Paint (FCP) dropped from 4.5 seconds to under a second. Load times are much more consistent across desktop and mobile as well, which has been an ongoing challenge.

The unexpected bonus was that these same changes also helped with search engine rankings. Faster pages, smaller downloads, and aggressively cached assets all feed into Core Web Vitals. So without targeting SEO explicitly, the site became easier to crawl, faster to index, and rank higher in search results simply by prioritizing user experience.

Making this easy to reuse

If you've built your own blog, you've probably run into these performance bottlenecks just like I have, and that's okay. Serving assets out of S3 is a valid solution and has worked well for me for six years as the site grew.

When my constraints changed, I wanted to extend the system without changing the deployment workflow. If you're interested in the same improvements I've described here, this setup is available in the Serverless Application Repository as a drop-in addition to an existing system. The full source is also available in GitHub if you want to dig into the details or adapt it further.

Above all else, this was about taking an entire class of decisions around image formats, sizes, and caching, and keeping them out of the day-to-day workflow. With everything in place, performance simply happens by default.

This was as fun as it was important for Ready, Set, Cloud. I wanted something easy to adopt, easy to remove, and something that quietly does the right thing as everything else moves around it.

Happy coding!

The Real Cost of Swapping Infrastructure

Allen Helton — Mon, 15 Dec 2025 20:43:40 +0000

I've gone through enough infrastructure evaluations as an architect to recognize the moment when the energy leaves the room. It's not when someone questions the performance numbers or the cost model. It's when someone pulls up the codebase and starts counting how many services need to change.

The infrastructure might be more reliable, easier to operate, or have better economics, but it doesn't matter if getting there means touching stable production code across dozens of services. The conversation shifts from "should we do this?" to "can we afford to do this?" and the answer is usually no.

That gap between "this is better" and "we can actually adopt this" is where many decisions stall or get turned down.

The real cost of infrastructure change

Architecture discussions tend to follow a familiar pattern. The whiteboard fills up with boxes and arrows, the tradeoffs look reasonable, everyone agrees that you'll come out the other end better. Then someone asks: how much code do we have to touch?

That question isn't about features or benchmarks. It's about risk. Architects evaluate the blast radius of change alongside performance and reliability. Every line of application code that needs to move, every client library that needs to be swapped, every behavior that needs to be re-learned increases the cost before you can even run a proof of concept.

For systems already in production, touching stable code introduces uncertainty. It stretches review cycles, kicks off regression testing, and makes rollback complicated. Good ideas often don't make it past this point because weaving them into existing applications costs too much.

This is especially relevant for infrastructure on the hot path. When caching misbehaves, it takes other systems down with it. Teams are rightfully cautious about changes here, even when the infrastructure side of the proposal is compelling.

What teams actually trust

Teams trust behavior they've observed in production, like how commands serialize, how errors surface, or how retries behave under load. That behavior has been exercised millions of times. It's hardened by real traffic, load testing, and years of incremental fixes. In practice, this behavior acts as a contract between the app code and infrastructure.

This is why client changes feel expensive even when two libraries look similar on the surface. Timeouts, connection handling, pipelining behavior, and edge cases around failures all shape how systems respond when stressed. At scale, subtle differences show up as tail latency spikes or incident tickets that are hard to explain.

For cache-heavy systems built on Redis or Valkey, this contract is often the wire protocol itself – RESP, the wire format the client already speaks. The application doesn't depend on “a cache,” it depends on this specific way of talking to one.

Changing what sits behind the contract

When you hold the contract constant and change what sits behind it, the risk potential drops dramatically.

Instead of rewriting cache layers or swapping SDKs across services, teams can point existing Redis or Valkey clients at Momento, authenticate, and issue the same commands they already use. The infrastructure changes. The operational model changes. The application code largely does not.

That distinction turns evaluation from a refactor into a configuration change. It lets teams observe real production behavior without committing to a rewrite upfront. More importantly, it makes rollbacks boring. Simply change an endpoint back and that's the extent of it.

This doesn't eliminate all risk. RESP compatibility has edges and limitations worth understanding. Not every Redis command is supported, but it shifts the evaluation risk from application code to infrastructure, where it's far easier to observe and reason about.

Lowering the cost of evaluation

I've noticed the infrastructure platforms that gain real adoption share a common trait: they meet teams where they already are. They respect existing contracts, existing mental models, and the realities of systems that have been running in production for years.

"Better" isn't enough if getting there requires destabilizing code no one wants to touch. The platforms that succeed make it easy to start small, observe real behavior, and back out safely when something doesn't line up. When evaluation feels reversible, teams engage honestly with the tradeoffs instead of inventing reasons to stay put.

RESP compatibility fits this philosophy. It doesn't ask teams to abandon the clients or patterns they rely on. It allows them to keep the contract they trust while changing the parts that benefit most from being managed: scaling, availability, and operational complexity.

In practice, that's often what separates interesting technology from technology that actually gets adopted.

Breakthroughs are just boring improvements that pile up

Allen Helton — Mon, 24 Nov 2025 21:04:05 +0000

We romanticize big engineering wins.

The performance chart that hockey-sticks up. The keynote slide with the impossible number. The "we hit a billion requests per second" humble brag.

The truth behind those moments is never one big breakthrough. It's dozens, possibly even hundreds, of small changes and enhancements. The kind that look ordinary when they land in a PR. The kind that your eyes casually skip over in release notes. The kind nobody celebrates… until they compound into something revolutionary.

I was recently reminded of this while listening to the Cache It podcast. Valkey project maintainer Harkrishn Patro described how his team scaled their system to handle a billion requests per second. He wasn't describing a feature launch, but rather the result of continuous refinement across every corner of the system.

Engineers, myself included, often hone in on small pieces of a system and lose sight of the forest for the trees. But with bold headlines and sensational marketing, we forget the forest is made from trees. As a result, major innovations overshadow the hard work that was done in the years and months leading up to "the big reveal."

So what happens if we take a stroll through that forest to see what the big announcement was really about?

Innovation removes friction

We tend to think innovation shows up wearing a cape. A disruptive new framework. A patented algorithm. A major architectural redesign. But most real innovation is far less glamorous – it's the process of shaving off tiny sources of drag until the entire system starts to glide. The "death by a thousand paper cuts" phenomenon is real.

Distributed systems make this painfully clear. The moment you push scale beyond "normal," you reveal dozens of inefficiencies that rarely show up in happy-path benchmarks. And this doesn't just refer to hyper scale! Problems change at every tier of scale. From POC to startup. From startup to regional. From regional to global. Suddenly, small issues aren't small anymore, they're multiplied across thousands of functions, nodes, and containers processing millions of requests per second.

That's exactly what the Valkey team discovered while pushing cluster sizes past limits that were never originally intended. Instead of rewriting the system from scratch, they focused on identifying and removing friction one piece at a time. For example, early pub/sub workloads broadcast messages to every node in the cluster, regardless of who needed the data. Reducing that via sharded messaging meant drastically less traffic flowing through the cluster bus, which freed up capacity for the work that mattered.

Or the size and frequency of internal coordination messages. By making those packets smaller and more efficient, Valkey stopped wasting cycles on communication overhead, freeing up capacity for serving actual requests.

Individually, neither of these changes would get a developer on stage at a conference. Nobody ships a "lighter gossip protocol!" feature release. Stack those improvements long enough, and the system starts performing at a level that once felt impossible.

Innovation isn't about adding more. It's about taking away.

When you remove enough drag, the system suddenly feels like it's leaping forward… even though it got there one subtle change at a time.

Reliability is a force multiplier

A fast system under perfect conditions is just a prototype. The moment you introduce chaos, like what happens regularly in production, performance either collapses or compounds. Reliability determines which way it goes.

Just like removing friction creates capacity, improving reliability creates opportunity. In distributed systems, fragility multiplies just as quickly as performance. One node degrades, then another, and suddenly a perfectly healthy cluster wastes capacity on recovery instead of serving requests. Failover becomes a performance bottleneck waiting to happen if handled poorly.

Back to the small wins with Valkey – when multiple primaries failed simultaneously (like during an Availability Zone outage), the system would struggle to elect new leaders because candidate requests flooded in all at once. Leadership battles caused downstream stalls that made the whole cluster feel unstable under pressure. The solution was introducing strategic delays to election behavior, allowing replicas to take over cleanly and confidently without overwhelming the system.

But that's the pattern – resilience opens up throughput. A cluster that can route around failure without spiking CPU or tail latency stays fast when users need it most. In Valkey's case, the additional throughput in unfavorable situations kept cycles free to handle more requests per second. Another blip on the release notes that continued to add up to the breakthrough.

Reliability amplifies every other improvement in the system. It creates breathing room. It enables aggressive scaling. It gives teams the confidence to grow beyond today's limits. When recovery becomes routine and uneventful, progress accelerates.

Scaling is an operational behavior

Contrary to what you might think, scaling isn't a hardware problem. It's a behavior problem. If changing scale feels risky, disruptive, or expensive, teams simply don't do it – even when they need to. They take the safe route: over-provisioning up front and praying they don't get surprised later.

There's nothing innovative about survival mode.

Real scalability emerges when the cost of adjusting capacity falls so low that it becomes a non-event (that's one of the reasons people like serverless so much). The faster and safer adding nodes, redistributing data, or shifting workload patterns feel, the more frequently teams perform them. And the more frequently teams perform them, the closer infrastructure can track real-world demand without waste, stress, and cost.

Once again, the Valkey team learned this firsthand. Since Redis 1.0, data migration has been a scary operation. Moving slots between nodes meant risking performance hits, stalled operations, or complicated (and error prone) manual intervention. So people avoided it, and clusters either stayed small or stayed oversized. Innovation would stall because operating it at its potential felt like walking a tightrope.

Atomic slot migration changed the behavior. Now, moving data around a cluster is predictable, controlled, and boring – exactly what operators need when managing production systems. This feature opened up capabilities that already existed but weren't safe to reach for.

When scaling becomes effortless, teams stop treating it like a last resort and start doing it as a reflex. Progress begins to compound. Capacity and confidence reinforce one another. And suddenly, what once felt like a milestone becomes just another Tuesday.

Breakthroughs happen gradually… then suddenly

From the outside, breakthroughs look like a leap. One day the headline appears. The graph turns vertical. Everyone wonders how a system could possibly jump from "pretty good" to "borderline unbelievable" in such a short time.

But from the inside, it looks like months, sometimes years, of sanding down rough edges. Fixing the tiny things that don't feel worthy of a celebration. Optimizing behavior that only shows up in profiling. Eliminating friction one pull request at a time.

That's exactly what happened with Valkey. A billion RPS was never the goal, but it was the receipt of countless invisible improvements. The milestone only looks magical because we didn't see the work it took to make it boring.

And this isn't unique to distributed systems or caching engines. It's universal. Every major engineering accomplishment is built from the moments where someone looked at a sharp corner and decided it didn't have to stay sharp.

Progress compounds. Confidence compounds. Capability compounds. Breakthroughs happen gradually… then suddenly.

So if you're in the middle of the grind, whether you're shipping incremental changes, cleaning up edge cases, or improving reliability and operability in small ways, take a breath of solace. You're stacking. You're creating the conditions where something extraordinary is going to look effortless.

The day is coming when the big announcement arrives, the system holds steady, and people assume it was easy. But you'll know better. You'll know it was earned one small improvement at a time.

Happy coding!

Cache Rebalancing Was Broken. Here's How They Fixed It.

Allen Helton — Tue, 18 Nov 2025 22:16:54 +0000

Few things make SREs more nervous than rebalancing a cache cluster.

You know the feeling. You add a node, trigger a rebalance, and suddenly latency graphs start jumping. It's a familiar risk of the job, especially when your cache sits between your users and your database. A small configuration mistake here can suddenly unleash a storm of GET requests on your primary data store.

I admit, I never really understood the concept of a slot or why it was needed. But after listening to a recent episode of the Cache It podcast on the new atomic slot migration feature in Valkey 9.0, I finally decided to dig in. The deeper I went (and the more times I replayed the episode), the more it clicked.

My learning adventure led me to ask, and finally answer, four important questions regarding slots and what happens to them when a cluster resizes. Let's dive in.

What are slots?

Most caching systems rely on consistent hashing to decide where data lives. It keeps keys evenly balanced across nodes while allowing clusters to grow or shrink with minimal reshuffling.

Valkey uses a fixed hash-slot model, specifically 16,384 slots, that together represent the entire keyspace. While 16,384 slots might seem random, it's actually 2¹⁴, a power of two chosen because it balances just the right amount of precision and efficiency. With that many slots, data can be distributed across large clusters without adding unnecessary overhead to routing or metadata.

Every key hashes to one of those slots, and each node in a cluster owns a subset of them. Rather than mapping every key directly to a node, Valkey maps slots to nodes. Since keys are deterministically hashed to slots, this makes scaling predictable. When you add or remove a node, Valkey only has to move the slots it owns, not millions of individual keys.

When the Valkey client library hashes a key, it already knows which node owns the corresponding slot. If the topology changes (aka slots being assigned to a different node due to a scaling event), Valkey issues a quick redirect so the client can retry against the correct node.

What was broken about the old migration model?

The old migration model moved one key at a time between nodes, triggering a flurry of redirects and topology changes. Clients were constantly told, "Sorry, this key moved, try over there."

It was essentially a brute-force way of moving slots from one node to another. It worked, but it wasn't elegant – and it definitely wasn't fast.

Each key transfer required multiple round trips, and every slot migration forced clients to refresh cluster topology. Large values could even block threads during serialization.

When you're working with millions of keys across your cluster, that adds up to a resource-intensive process that can take minutes to complete, all while the cluster remains live and serving traffic.

The result was instability and slower tail latencies during migration. Which means it was something you'd postpone unless you absolutely had to run it.

How does atomic slot migration fix it?

Valkey 9.0 redesigned the slot migration process using the same principles that power replication. Instead of moving keys one by one, atomic slot migration runs in three distinct phases:

Snapshot – The source node forks a background process and captures a point-in-time snapshot of the slots being migrated while continuing to serve live traffic.

Streaming – Any writes that happen during the snapshot are captured in a buffer and streamed incrementally to the target node.

Finalization – Once all data is synchronized, Valkey briefly pauses new writes, sends a final marker, and performs a single, atomic handover.

This three-phase approach eliminates the multiple round trips, client redirects, and fragile user experience that used to come with slot migration. Because the process runs in the background and atomically switches ownership when complete, there is no risky in-between state. Once it finishes, your slots are already assigned to the target nodes without interruption or confusion.

Why is this better?

For operators, this is a big deal.

Fewer round trips, fewer topology changes, and no split-brain state during migration. You can rebalance entire clusters without disturbing workloads or waking up the on-call engineer (hooray!).

Both models still exist today, but atomic slot migration represents a new standard for reliability. It shows how thoughtful engineering can make the hardest operational tasks feel invisible.

As Khawaja said in the latest Cache It podcast episode with Jacob Murphy, "This moves us closer to a world where scaling a cache never means taking it offline."

And that's a world every SRE wants to live in.

Happy coding!