DEV Community: kuberdenis

Multirun: A Multi-Agent Run Framework

kuberdenis — Sat, 24 Oct 2026 07:09:35 +0000

Ask a coding agent to do the same non-trivial thing twice, and you get two different implementations, both fine and both defensible. Denis had been noticing this for months and doing absolutely nothing about it, because with a single run there is nothing you can do. One run is an anecdote. You read the diff, you decide you like it or you don't, and the actual question never comes up.

The question is what it would have done the other nine times.

You can't see that at n=1. And n=2 doesn't really help either, because two runs is a comparison, and what he wanted was a distribution of where the model lands reliably and where it is quietly flipping a coin on his behalf.

So he spent a day building a framework that runs one task ten times in parallel, keeps every run isolated from the others, and then shows what the ten did differently. It is called multirun, and it lives here:

[a2wio/multirun]

Note: this post is a direct continuation of Denis's "Preview Environments" and "Agentic Workflows: Branched Environments" posts. Same core trick, pointed somewhere new. If the words "database branching" mean nothing to you yet, the second one explains them properly, so it doesn't get repeated here.

Introduction

He built it out of curiosity. What came out the other end were two uses worth recommending to somebody else, so those go up front, and the machinery comes afterwards.

The first one is evals. Denis had never written one. Not properly, not against his own prompts. Every time the subject came up it sounded like infrastructure (a harness, a judge model, a golden dataset, some score to chase) and it went onto the list of things he would do when he had a spare week. Well… the spare week never arrived.

Running the same task ten times and reading where the ten answers agree is an eval. A crude one, but a real one, and there is no judge model or benchmark anywhere in this post. The ten runs in here converged completely on the schema and then split exactly 5/5 on two separate decisions, and those splits are the result. They name the sentences his task file was missing. He did not have to guess at them, and nothing had to be built to find them beyond a loop that runs the task more than once.

The second one he has been calling results branching, which is a straight lift from database branching, and he is keeping it. It is for the days when you are about to build the thing you will live with for years: the tenancy model, the auth boundary, the schema that will have three years of rows in it before you get another shot at it. The normal habit is to ask the agent, read what comes back, and either take it or argue with it. That is one sample out of a distribution you never get to see.

So don't take the first answer. Fan the same task out ten ways, let them run without seeing each other, and read the spread before you commit to one. The fan-out in this post took fifty minutes of wall time and about thirty-two dollars at API rates, which is not nothing, but it is a great deal less than what a wrong ON DELETE costs you eighteen months later.

Neither of those needs much cleverness. They need ten runs that cannot touch each other, and that is the hard part.

The isolation is the whole design problem, and it splits into two halves.

Ten agents editing one repository is nonsense, and that half is easy: git has had worktrees for years, every run gets its own, done. Ten agents sharing one database is the harder half. It was hard for a long time, and it stopped being hard the moment the database learned how to branch. That is the reason this project became buildable in a day instead of a quarter.

TL;DR

multirun takes a repository at a specific sha and a task, and fans out N isolated runs of Claude Code, capturing everything each one did.
Every run gets a git worktree and its own Postgres branch, both off the same seeded parent. Diffing a run's branch back against that parent shows what the agent did to the data, not only the files.
First real fan-out: ten runs, same task, same model, max_parallel: 3, on his k3s cluster. 10/10 finished, 20/20 checks passed, 50 minutes wall, 930 turns, zero leaked branches.
All ten reached the same schema shape. Then they split 5/5 on whether deleting a team deletes its data, 5/5 on the join table's primary key, and 8/2 on whether the new column is allowed to be NULL.
Convergence on structure, coin flips on judgment. There was no way to see that without running the thing ten times.
Two uses fell out of it. It is a cheap eval, since where the ten answers split is where your prompt was ambiguous. And it is a way to branch a decision you will live with for years, instead of taking the agent's first answer for it.

What a run is

The whole framework fits on one line:

run = (source, task, variant) -> artifacts

Source is a repository at a specific sha, plus an optional database seed. Never vendored, always pinned. If the ten runs don't start from a byte-identical tree, comparing them means nothing.

Task is a markdown prompt and its checks, versioned as files, so that an old task can be re-run against a newer source later.

Variant is whatever differs per run: the model, a prompt variation, or nothing at all. That last case is the one he cares about, because when nothing differs, the only variable left is the model itself.

Everything else in the system (the tables, the dashboard, the comparison writeups) is a view over artifacts.

Why branching beats copying

Ten runs need ten databases, and they all need to begin life identical.

The obvious approach is to copy one database ten times, and the obvious approach is bad. A copy costs you the full size of your data every single time, it takes minutes, and afterwards you own ten real databases that you have to remember to delete. At 30 MB that is merely annoying. At anything resembling production data, you will not do it twice.

Neon separates compute from storage, so a branch is a new logical pointer into the same storage, and only the pages that actually diverge ever get written. Nothing is duplicated up front. He went through the mechanics at length in Agentic Workflows: Branched Environments, so here it is only the number that matters. A branch taken off this bench's parent while this post was being written was ready in 1.38 seconds.

That parent holds about 30 MB. He has not tried this at 300 GB, so take it as reasoning and not as a measurement.. but the whole point of copy-on-write is that the number shouldn't move much, because nothing is being copied.

The diff

Provisioning speed is nice, but the diff is why he built it this way.

Every run branches off the same parent, so when a run finishes its branch gets diffed back against that parent, and that gives two things: the schema delta (what DDL it emitted) and the data delta (rows added, changed, deleted, per table).

A transcript tells you what the agent said it did. The branch diff is what it did. Ten of them are directly comparable, because all ten are measured against the same origin.

The tradeoff is that branches are real objects with a real bill attached. His plan includes ten of them and charges per branch per month after that, and a leaked branch is exactly the sort of thing nobody notices for a quarter. So teardown is a first-class state with its own row in the results database, rather than a finally block somewhere in the code, and every run passes through it whether it succeeded, failed, or timed out:

PENDING -> PROVISIONING -> SPAWNED -> RUNNING -> DIFFING
        -> TEARDOWN -> DONE | FAILED

Notice that DIFFING comes before TEARDOWN. The diff needs the branch alive, and the branch needs to die. Getting those two in the wrong order is how you end up with ten orphans and no data.

One config file describes an entire fan-out. global: holds the defaults and runs: is a list of per-run overrides. For this fan-out every single run is identical, which is the point.

Putting it in the cluster

Locally, all of this is a loop. On his k3s cluster it becomes a controller.

core is a plain Deployment, no CRD and no operator boilerplate. He did not want to write a reconciler for this, and does not think it needs one. It watches a ConfigMap inbox that multirun publish drops a fan-out config into, merges global + per-run config for each run, renders that into a per-run ConfigMap, and spawns exactly one Kubernetes Job. Run state lives in a Postgres results database rather than in etcd, because he wants run history queryable in SQL months from now, not kubectl get output that ages out of existence.

The runner itself is deliberately stupid. It reads /config/run.yaml, does the run, writes artifacts to a shared PVC. It never talks to the Kubernetes API, never talks to the Neon API, and never imports orchestrator code. Config in, artifacts out is the entire contract. That is what keeps a failed pod debuggable, since you can hold a whole run in your head from one YAML file and one directory.

The agent runs through Anthropic's Agent SDK on his Claude subscription, rather than the raw API. Two things follow from that, and both bit him. Subscription credentials inside a pod rotate, so a stale Secret fails auth with nothing obvious in the logs. And ten simultaneous agents on one subscription hit rate limits, which is why max_parallel defaults to 3. Ten branches does not mean ten at once.

Where he cut a corner

This was running on the cluster, and working, before it was under ArgoCD.

The manifests existed. The registration PR was open. And he had kubectl apply'd them "as a bridge" to unblock the fan-out. Well… the bridge became the deploy. Everything on that cluster is GitOps-managed, and this one thing was a hand-placed object pretending to be one, which is arguably worse than not deploying it at all, because it looks completely fine right up until the moment someone syncs.

So it got adopted properly: an Application, the manifests in git, secrets as SealedSecrets. The adoption rolled nothing. There was zero drift between what he had applied by hand and what git said, so the fan-out that was running at the time never even noticed.

The lesson he took from it is smaller than "use GitOps", and more useful. He had verified that the core pod is running, when the thing he was claiming was the app is deployed. Those are two different sentences, and he checked the easy one.

After adoption. The bridge he'd applied by hand and the manifests in git were byte-identical, so nothing rolled.

Ten agents, one task

The task he picked is the most boring migration in software, and he picked it for exactly that reason. Take an app that was built single-user, and add teams to it. Every SaaS does this eventually. There is no clever answer waiting at the end.

The prompt is a short markdown file, and the important thing about it is what it refuses to specify:

This app was built single-user: every row that matters hangs off one user id, and nothing in the schema knows what a team is. Add teams.

A user can create a team, belong to teams, and act within a team context.

Data that was per-user and should now be shareable within a team is reachable by teammates; data that should stay personal stays personal. Deciding which is which is part of the task.

The rows that already exist keep working: an existing user signs in and loses nothing.

Access control still holds afterwards — a member of team A can not reach team B's data through any path you added.

How you model it — tenant column, join table, something else — is your call. Make the schema change as a real migration, wire the app code to it, and say at the end what you chose and what it costs.

The source is a Next.js + Drizzle + Neon template of his, pinned at one sha. The seeded parent database holds 3 users, 3 profiles and 4 assets, deliberately tiny, so that he could read the data diff with his own eyes. Two checks, both dumb: the app still builds, and something with "team" in the name exists in the schema. The grading happens in the diffs, not in the checks.

Ten runs, all sonnet. 10/10 done, 20/20 checks passed, 50 minutes end to end at three at a time. Between 77 and 117 turns each, 462k output tokens and 63 million cache reads across the ten, 143 files touched, and every single one of the ten branches released afterwards.

▶ Effort, run by run: per-run turns, output tokens, cost, wall time and files touched, an interactive widget, runs on the original post.

Per-run effort out of the trace data. Flip the measure and the ten bars stay put.

Where they agreed

Every single run built the same thing: a teams table, a team_members join table, a team_id column on assets.

None of them did tenant-column-only, and none did anything exotic. All ten decided that assets are the shareable thing and profiles stay personal, without ever being told which was which, and all ten kept every pre-existing row alive. Row-level security never came up once. Access control stayed in application code, ten times out of ten.

He expected them to scatter. They converged hard.

Where they didn't

Does deleting a team delete its data? Five runs said SET NULL, so the assets survive and fall back to personal. Five said CASCADE, so the assets die along with the team. That is a user-visible product decision that nobody asked the agent to make, and it came out exactly 5/5.
What is the join table's primary key? Five went composite (team_id, user_id). Five went surrogate id with a unique constraint on the pair.
Is the new column allowed to be NULL? Eight runs made assets.team_id nullable. NULL means personal, existing rows need no touching, and the migration stays pure DDL. Two runs refused the null entirely: NOT NULL, a personal flag on teams, and a data migration that mints a personal team for every existing user and moves their assets into it. Same sentence, two different philosophies of tenancy.

There is smaller scatter underneath all that, too. Six used a Postgres enum for the member role, three used text with a CHECK, one left it as bare text with a default. Six gave teams a unique slug, four didn't. Six added a current-team pointer to profiles, and then split between two different names for it.

Ten runs is not enough to pin any of these ratios down. A 5/5 at n=10 is something a true 70/30 would produce often enough that you can't rule it out, so the thing to take from this is that there is a split there at all, on a decision the task file assumed had one answer, rather than the exact 50/50. Getting the real ratio means more runs, which the bench can do since it's just a config value, but he hasn't run them.

▶ Where the ten runs landed: seven schema decisions across ten identical agent runs, an interactive widget, runs on the original post.

All seven decisions, sorted by how many of the ten agreed. This is the real comparison data out of the run diffs. Hover a segment for the run numbers.

Run 06's database is the only one of the ten with actual rows sitting in teams and team_members, one team and two members. No migration inserted those, which means that agent tested its own feature through the running app and then left the test data behind. The only reason anyone knows is that the data diff caught it. Nothing in the transcript would have sent him looking.

The failure

It is the more useful half.

His first attempt at this fan-out failed 10 out of 10, in under ten minutes. The runner image had no openssh-client, so cloning the private template over SSH died in the init container, every time.

The smoke test never caught it, because smoke clones a public repository over HTTPS, a completely different code path from the one every real run takes. The fix was one line in a Dockerfile.

Teardown held, though. All ten of those branches were released anyway, which is the only reason he could shrug at it and re-run.

Watching it happen

Ten agents working in parallel inside pods you cannot see into is an unpleasant feeling. So the last piece is a read-only dashboard: instances, runs side by side, per-run detail with the phase timings, the check results, the branch diff, and the artifacts.

The plan had been to change the runner for the live view, streaming each turn to a file on the shared PVC as it happens. Then he went to implement it and found the runner had been doing exactly that since day one. Every agent message, appended to trace.jsonl, flushed per message, sitting on the PVC right next to the artifacts.

So the dashboard tails that file over a websocket, which makes live and replay the same thing. A run that finished four hours ago plays back through the identical code path as one happening right now, and no pod logs are involved anywhere, so it still works long after the pod is gone.

Every fan-out this bench has run, including the one that failed 10/10.

The ten runs of add-teams. Notice how tight the effort is. Whatever the variance is, it is not effort variance.

One run: phase timings, both checks, and what it did to the data.

About the cost readout. The SDK reports what each run would have cost at API rates: $2.49 to $4.39 per run, $32.25 for all ten. None of it was billed, since this runs on his subscription. But it is the number he uses to think about what a fan-out is worth, and at roughly thirty dollars he can afford to ask what it would have done the other nine times.

Closing

Variance is a property of the model and the prompt. That is what he is keeping from this.

Ten identical runs did not produce ten random results. They produced total agreement on the schema, and two exact 5/5 splits on decisions that nobody asked them to make. Both halves of that are actionable. The agreement tells him the model has a strong prior about how tenancy gets modelled in Postgres, and he can stop reviewing that part quite so hard. The splits tell him precisely which sentences his task file was missing.

And once you can measure a split, you can try to close it: write the prompt more precisely, run ten more, see whether 5/5 turns into 9/1. That is an eval. It took a loop and a config file, there is no judge model in it anywhere, and until this week he had no way to run one against his own prompts at all.

The other half is the one he would actually go and use on Monday. Pick the next change you would genuinely regret getting wrong: the tenancy model, the auth boundary, the schema that will outlive the code sitting on top of it. Don't take the agent's first answer for it. Run it ten times and read the spread. Where all ten agree you can move quickly, because that is the model's prior and it is usually the dull correct thing. Where they split, there is a decision sitting there that was always yours to make, and the agent has been making it for you.

He has been calling that results branching. It is the same mechanism as the eval, pointed at one decision instead of at the whole prompt.

He keeps coming back to what he wrote in the branched environments post, that the pre-AI techniques are what make the agentic ones actually work. This is the same thing again, one layer up. Worktrees, database branching, GitOps, teardown as a state machine.. none of that is new, and none of it is about AI. It is all boring infrastructure practice, and it is the only reason ten agents can run at once without stepping on each other.

He started this wanting to know whether he could trust the model. He still doesn't really have an answer to that. What he has instead is a list of the places where he hadn't decided something and it decided for him. That has turned out to be more useful.

Thank you for reading :)

After-post notes

_N1: The bench, the fan-out and the numbers in this post are all real. The config that produced them is in the repo, and you can run the same fan-out yourself.

N2: If this post inspired you to think, feel free to email Denis and start a discussion! He loves talking technology, and even so if you are, too, actively living in "the arena"._

You can find everything about Denis on denislavgavrilov.com/about

Thank you for reading and have a good one!

.mr-signoff{display:flex;gap:.7rem;align-items:flex-start;margin:2.25rem 0 1rem}
.mr-signoff>img{width:40px;height:40px;min-width:40px;border-radius:9999px;margin:.1rem 0 0;display:block}
.mr-signoff-name{font-size:.8em;line-height:1.3;margin:0 0 .4rem}
.mr-signoff-name span{color:var(--color-muted)}
.dark .mr-signoff-name span{color:var(--color-muted-dark)}
.mr-signoff-msg{display:inline-block;border:1px solid var(--color-rule);border-radius:.9rem;padding:.4rem .8rem;font-size:.95em;line-height:1.45}
.dark .mr-signoff-msg{border-color:var(--color-rule-dark)}

Darling · Denis's assistant
we built this together

diffium-db: A live diff of what agents do to your database

kuberdenis — Fri, 23 Oct 2026 07:12:55 +0000

The following post is a recap of my Claude Code sessions on building diffium-db. Although the content is AI-generated, I go through the entire post to see if any differences occur - no, everything is through my own process of developing diffium-db. If you'd like to discuss anything in relation to this project directly with me, you can do so on dennis@a2w.io or on x.com/@kuberdenis.

Note 2: this project is a direct continuation of another project of mine that I use to watch diffs - github.com/kubeden/diffium.

I let agents run migrations. That is how the work goes now. You describe what you want, something writes the SQL, you read it, you approve it, and a few seconds later the schema is different. The ten minutes after that are the problem.

git diff shows you the migration file. It does not show you what the migration did. And it shows you nothing at all about the row an agent updated on the way past, because updating it seemed reasonable at the time.

There are good schema diff tools, but they mostly work the same way. You run them afterwards, against two databases, and they hand you a delta. Nothing sits open next to you while the work is happening.

So I built one. diffium-db is a terminal UI, and it is meant to stay open while you work.

Resources

kubeden/diffium-db, the repo (MIT)
kubeden/diffium, the file-diff TUI this one is modelled on
OpenTUI, the terminal framework it is built with

What it does

You point it at a Postgres, take a baseline, and leave the watcher open. From that moment everything that changed is listed on the left, the change itself is on the right, and it re-reads the database once a second.

Here it is catching a migration. Two columns and an index appeared on demo.users while I was looking at it:

The left pane is everything that changed since the baseline. The right pane is the object you have selected, before and after.

It watches tables (columns, defaults, identity, constraints, indexes, triggers), views, materialized views, enums and functions. It also watches rows. That turned out to be the interesting half, and I will come back to it.

There are four commands. watch opens the TUI, snapshot takes a baseline, baselines lists the ones you have, and diff prints the changes and exits. That last one takes --exit-code, so it returns 1 when anything changed, which is enough to fail a CI job or stop an agent's loop before it does the next thing.

Setting it up

You need Bun 1.3 or newer and a Postgres to watch. There is exactly one dependency past that, @opentui/core, so the install is quick.

git clone https://github.com/kubeden/diffium-db && cd diffium-db
bun install
export DATABASE_URL='postgresql://...'

Then take a baseline and open the watcher:

bun run src/index.ts snapshot
bun run src/index.ts watch

That is the whole setup. The repo ships a demo so you can see it move without waiting for an agent to do something. examples/demo/01-baseline.sql builds a small demo schema with users and projects in it, and examples/demo/02-agent-change.sql is what an agent does when you tell it "add billing plans and clean up".

Two terminals. In the first:

psql "$DATABASE_URL" -f examples/demo/01-baseline.sql
bun run src/index.ts snapshot --schema demo
bun run src/index.ts watch --schema demo

In the second, while the watcher is open:

psql "$DATABASE_URL" -f examples/demo/02-agent-change.sql

Five changes land on the screen: a new enum, two new columns and a new index on demo.users, an index gone from demo.projects, one row inserted and one row deleted. j and k walk them, s switches between side by side and inline, e writes the whole diff to a file.

Where to point it

The demo is a demo. For real work, do not point it at production. The point is to let the agent run and read what it did afterwards, and you want that happening somewhere you can throw away.

So I give the agent a branch. On Neon that is two commands, plus the project id, because I have more than one project and the CLI refuses to guess:

neonctl branches create --project-id <project> --name agent-run
neonctl connection-string agent-run --project-id <project>

Point DATABASE_URL at what the second one prints, take your baseline there, and let the agent work on the branch. The branch is copy-on-write off its parent, so it comes up holding the same schema and the same rows as the database you care about. The one I made while writing this took 1.2 seconds. An idle branch suspends its own compute too, so the ones you forget to delete are not sitting there running.

Then you read the diff. If the migration is what you wanted, run it against the real database. If the agent did something you did not expect, you found that out on a copy, and the copy gets deleted.

That is the loop, and it is how this post was made. The demo and every screenshot in it ran on a branch of the demo project. The demo database itself still holds the state I left it in.

Any Postgres works and nothing in the tool is Neon-specific. Branching is what makes this cheap enough to do every time instead of only when you remember.

What a row diff can prove

Structure is the easy half. Rows are where it gets uncomfortable.

To tell you that a row changed, you have to remember what it was. Storing every row is not an option for anything real, so diffium-db stores a fingerprint instead: the primary key, plus md5(t::text) of the whole record, plus a short preview. Cheap to compute in the database, cheap to keep, and it separates an insert from an update from a delete exactly. If the key is new, it is an insert. If the key is gone, it is a delete. If the key is the same and the hash moved, someone edited that row.

That works right up until the table gains a column. Then every hash in the table changes at once, and a naive tool would tell you that every row in it was edited. Nobody edited them. The shape moved underneath them.

diffium-db does not do that.

~3? in the left pane, and the note at the top of the diff. It detected the column change, marked the table, and told me which half of its own answer to trust.

The count is written ~3? and not ~3, and the first line of the diff says why. Two columns arrived, so every row in the table reads as edited, and the update number is the one part of that answer the tool cannot stand behind.

Inserts and deletes are still exact, because those come from the primary key and the key does not care what the shape is. Only the update count is guesswork, and printing a guess plain, next to numbers it can prove, would make the whole screen worth less.

Tables without a primary key, and tables over --row-limit (5000 by default), get a row count and no claims at all. Same principle. I would rather it say less.

Structure is text

The structural diff rests on one decision, and it explains a lot of the behaviour.

Every object renders to one canonical block of text with a deterministic line order. A table becomes its columns, then its constraints, then its indexes, then its triggers, always in that order. The structural diff is then just a line diff of those blocks.

That one choice pays for a lot. Adding a column is one added line instead of a table that vaguely "changed", and side by side, the +2 -0 counts and horizontal scrolling through a long default all come free.

The same change inline instead of side by side, so nothing is cut off at the pane edge. s toggles it, w toggles wrapping, and both are remembered between runs.

There is a cost. diffium-db tells you what the database looks like now, not the DDL statement that would get you there. For a watcher that is the right side of the trade. If you want generated migrations, you want a different program.

Where the baseline lives

A baseline is a stored value, not a live connection, so you can take one now and diff against it tomorrow, keep several named ones, or commit one next to the migration that produced it.

By default they go in .diffium-db/snapshots/ as JSON. Pass --store neon --store-url <url> and they go into a diffium_db schema in Postgres instead. Use that when CI and your laptop need to agree on what "before" means. The watcher never looks at its own schema, so the store URL and the watched URL can be the same database.

With the branch above, that means the baseline can sit on the branch itself, one schema over from the thing it is describing. The branch is then the entire experiment, and deleting it takes the record with it.

Future work

What v1 does not do, roughly in the order I want to fix it.

No ORM awareness. v1 tells you a column appeared. It cannot tell you which migration file or which model definition put it there. That mapping is an ORM's business and it is the next thing I want. src/orms/ is where it goes, documented and deliberately empty.

Postgres only. The catalog reading is isolated in src/pg/, and nothing above that boundary knows what a catalog is, so a second dialect has an obvious place to go. It does not exist yet.

No sequences. Every bigserial makes one and they move on every insert. Pure noise until someone asks.

Polling, not replication. It re-reads once a second. On a large database logical replication would be much cheaper, and that is the obvious thing to do if polling ever starts to hurt.

One thing is not on that list and never will be. diffium-db does not write to the database it is watching. The only thing it writes is its own store, and only if you point it at one.

Closing

It is about 2,200 lines of source and 900 of tests. 64 of those tests run with no database at all, and that includes the terminal ones, because OpenTUI will render into memory and let you assert on the screen it drew. So a change to the layout breaks a test instead of quietly looking wrong in a screenshot I forgot to retake.

What I keep coming back to is the row diff. It would have been easier to print a number for every table and let people assume it meant something. The version that says "I cannot prove this one" is a bit uglier to look at. I will take it. A thing that sits next to an agent and reports on what it did has to be more careful than the agent is.

It is on GitHub under MIT. If you try it and it tells you something wrong, open an issue.

Thank you for reading.

Databricks for AI Engineering

kuberdenis — Thu, 22 Oct 2026 07:00:06 +0000

TL;DR. Lakebase is Neon's Postgres inside Databricks, so an agent gets its own branch of production in seconds. Governed tables sync into it and read back in under 10ms, and every write the agent makes to a change-feed table lands in a Delta table in Unity Catalog about 15 seconds later. Branches are one-way and nothing merges back. You rehearse on the branch, decide from the rows it changed, and replay the approved change as a migration. I ran the branching half on Neon myself: ten agents, ten branches, all ten deleted.

Almost every agent running inside a company today is read-only. It searches, it summarises, it answers questions about data somebody else already wrote down, and it changes nothing.

Nobody sat in a meeting and decided this. It is what you get when the first serious question about a pilot is "what happens when it's wrong", and the honest answer is that nobody knows yet. So the write path gets taken away.

Databricks has an agent framework, and so does everybody. What caught my attention this year sits lower down: the database underneath the agent picked up two properties that used to be your job to build. You can hand an agent a copy of production for the length of a task, and the rows it touches land in a governed table without the agent knowing anything about it.

I want to walk through what that is, where it stops, and what it means for the write path.

What is a database branch?

I have written about this twice before, in Preview Environments and Agentic Workflows: Branched Environments, so I will keep it short.

A branch is a copy-on-write clone of a database. The storage engine keeps every change as a new version instead of overwriting the old one, so a new branch copies no data at all. It points at the same versions the parent points at, and starts recording its own changes from that moment.

Two things follow.

Branch creation takes seconds, and stays at seconds no matter how big the database is. A branch of a 2TB database is not slower than a branch of a 10GB one. There is nothing to copy.

And a branch costs storage proportional to what it changes. A branch that modifies 50MB inside a 500GB database costs you roughly 50MB, not 500GB. So "give every test run its own database with production-shaped data in it" stops being a budget conversation.

If you have ever maintained a shared staging database you know what this is for. It is the database where somebody else's half-finished migration is why your test failed.

What Databricks shipped

Databricks bought Neon in May 2025 for about a billion dollars. Neon was a serverless Postgres company whose whole architecture was the branching I just described. Ali Ghodsi said at the time that roughly 80% of the databases being created on Neon were created by agents, not by people.

That technology is now Lakebase. It went generally available on AWS on January 22, 2026, with the announcement blog in February, and on Azure in March. Postgres 17, pgvector in the box, 16TB of storage per branch (8TB at GA), scale-to-zero that comes back in hundreds of milliseconds on the next query, branching and point-in-time recovery.

Two integrations are the part worth your attention.

Synced tables pull a Unity Catalog table into Postgres so an application can read governed lakehouse data with sub-10ms latency and full ACID transactions. Three modes: snapshot for a one-time copy, triggered for scheduled refreshes, continuous for streaming with a 15-second floor on the interval. They are read-only by strong recommendation, only additive schema changes survive a triggered or continuous sync, and each synced table holds up to 16 connections against an instance limit of 1,000. Plan capacity around those last two.

Change Data Feed goes the other direction, and it is the one I did not expect. You turn it on per table, and from then on every insert, update and delete on that table is read off the Postgres write-ahead log and written as a row into a Unity Catalog managed Delta table, batched and flushed about every 15 seconds. The destination is named lb_<table_name>_history and carries five columns you did not write: _pg_change_type, _pg_lsn, _pg_xid, _timestamp and _sort_by.

It sits behind a preview flag, it needs REPLICA IDENTITY FULL on the source table, it does not handle partitioned tables, and it records rows and not DDL. Read the limits before you plan around it.

▶ Figure 1. The round trip: a governed table into Postgres, every agent write back out (interactive), an interactive widget, runs on the original post.

Figure 1. The two integrations, end to end. Select either arrow for its limits.

What it wrote becomes a table

The values of _pg_change_type are insert, delete, update_preimage and update_postimage. So for every update your agent makes you get the row as it was and the row as it became, two separate rows in a Delta table, ordered by transaction ID and log sequence number.

▶ Figure 2. What one write leaves behind: the rows an insert, update or delete lands in the change feed (interactive), an interactive widget, runs on the original post.

Figure 2. Rows written to the change feed per statement type. Select a statement.

Think about what that replaces. If you want to know what an agent did to your data today, you have three options. You can read the agent's own trace, and trust a language model's account of its own behaviour. You can hand-roll audit tables and triggers, at the price of every schema change becoming two schema changes. Or you can turn on full statement logging and go spelunking in a text file at 2am.

The Change Data Feed version is a SQL query. Every write, before and after, in a governed table sitting in the same catalog as everything else your company governs, under the same permissions.

It is plumbing, so nobody demos it. In a regulated company "we cannot see what it did" is the objection that kills the project, and a change feed answers the data half of it in SQL.

You cannot merge the branch back

Once you have seen branch-per-agent-run, the next thought is obvious. Let the agent work on its branch, review the result, merge the good ones. Git for your database. I assumed it existed and went looking for the API.

It does not exist. From the Databricks community, the accepted answer to exactly this question:

Currently, Lakebase does not have a built-in "merge" or "promote" operation that pushes child branch changes up to the parent.

Neon's own position, from before the acquisition, is that this is deliberate. They do not support automatic merging of diverged branches because a database merge is unsafe and ambiguous. Two branches both insert a row with id 4… Git calls that a conflict and makes a human read it. A database cannot reliably tell you it happened.

So the recommended path is the boring one. You capture what you did as migration scripts and apply the same scripts to the parent with Flyway or Liquibase or Alembic. Schema, not data. Migration tools are built for DDL, and getting an approved data change back to the parent is a separate problem you solve through the application or a script written for that one job.

Databricks does document a schema diff now. Open a child branch, click Schema diff in the Parent branch section, and you get a side-by-side DDL comparison against the parent. It compares schemas, it does not merge them, and you still write the migration yourself.

Which means a branch is not a feature branch. Nothing you accumulate inside one gets folded back in, and the only thing that comes out of it is a decision about what to do next.

Ten agents, ten branches, all thrown away

Let me make that concrete, because I have run it, on Neon and not on Lakebase.

I built a small framework called multirun that takes one task and runs it N times in parallel. Each run gets its own git worktree and its own database branch off a shared seeded parent. Same prompt, same starting schema, ten agents who cannot see each other.

From the run I wrote up: ten agents, 50 minutes and 10 seconds of wall clock, 930 turns between them, 143 files touched. A branch off the ~30MB parent was ready in 1.38 seconds.

All ten designed the same thing at the top level, a teams table with a join table and a foreign key on assets. Then they split. Five chose SET NULL when a team is deleted and five chose CASCADE. Five used a composite primary key on the membership table and five used a surrogate. Eight left the team reference nullable and two made it required with a backfill.

Ten runs cannot pin a ratio. What they establish is that the split exists at all, on a question the task file assumed had one answer.

▶ Figure 3. Ten branches, nothing merged: where ten identical agent runs landed on three schema questions (interactive), an interactive widget, runs on the original post.

Figure 3. Where ten identical runs landed on three schema questions. n = 10 per question.

The product of the whole exercise was the diff. I compared each branch against the parent to see what the agent had done to the data, wrote up the comparison, and deleted all ten branches. Not one was merged anywhere. It never occurred to me to try, because what I wanted out of it was a decision about SET NULL versus CASCADE.

I only noticed I had been working that way after I went looking for the merge API and found there wasn't one.

What I think this changes

If you are trying to get an agent past read-only inside a company, three steps fall out of all this.

Rehearse. The agent gets a branch, provisioned by API for the length of the task, with real production-shaped data in it. It writes whatever it wants, and nothing it writes through that connection reaches the parent database. Everything else it can touch, an API call, an email, a file in object storage, is still yours to sandbox separately.

Decide. You diff the branch against the parent, or you query the change feed. Either way you are looking at rows, not at the agent's summary of itself. A human approves, or a deterministic check approves, or both.

Replay. The approved change gets applied to production as its own deliberate act, by migration or through the normal application write path with the usual permissions on it. Then the branch goes away.

None of that makes an agent safe by itself. A branch isolates database state and nothing else, so the credential you hand it still has to be the smallest one that will do the job. And the change feed is a governed record of which rows moved, not a full account of what the agent did. It will not tell you who approved it, what the agent read, or which prompt set it off.

What you do get out of the box is the place to rehearse and the row-level record of the rehearsal, and those are the two things I expected to have to build myself. The rest is ordinary engineering with no model in it anywhere.

I have not run this on Lakebase. I have run the branching half, on Neon, on my own hardware, for about thirty-two dollars of list-price tokens, and it worked well enough that I stopped thinking of database branches as a testing trick. The rest is a workspace I have not spun up.

So if your agent is stuck at read-only, go and look at where it would write, and at how you would show somebody afterwards what it wrote.

Thank you for reading.

After-post notes

N1: multirun is open source at github.com/a2wio/multirun, and the write-up linked above has the full numbers.

Experiments in Software Factories #1: Production app in 40 minutes

kuberdenis — Wed, 21 Oct 2026 07:00:04 +0000

pastmonday.com is live. It has auth, a weekly calendar, tiered billing with Stripe, referral credits, file uploads, and an AI chat that knows your calendar. A small but complete SaaS, the kind of thing that used to be a three-month project.

Two things about it are unusual, and they're the reason for this post.

First: I didn't hand-write it. A software factory did.. four reusable Claude Code skills that turn a one-page brief into a shipped product, stage by stage. I talked about this on YouTube recently, and the skills themselves are public on GitHub.

Second: look at the vendor list. Postgres is Neon. Auth is Neon. The data API is Neon. Object storage is Neon. The LLM calls go through Neon. Stripe handles money and my own cluster serves the containers, and that's the entire list. The backend is one vendor, and until a few weeks ago that sentence was impossible to say, because half of these services didn't exist.

Neon announced they're building the full backend: Postgres, Auth, and the Data API are GA, and Object Storage, Functions, and an AI Gateway just entered private preview. I got early access, pointed the factory at all of it, and this post is the build report.

Note: this post is a transcription of my video on software factories, passed through AI and edited for reading. If you'd rather watch than read, here it is:

▶ Watch: Software Factories are SCARY but possible (and easy to do)

TL;DR

A software factory, four reusable Claude Code skills reading one PROJECT.md brief, shipped pastmonday.com from empty repo to production in about 40 minutes.
The entire backend is Neon: Postgres, Auth, and Data API (GA), plus Object Storage and AI Gateway (private preview, early access).
Money and limits are server-authoritative: atomic single-statement quota consumes, idempotent Stripe webhooks, AI budgets checked before every call.
The AI Gateway's official SDK provider wasn't installable yet, so the agent verified the OpenAI-compatible endpoint live and shipped anyway.. the API-first thesis from my last Neon post, proven again.
Neon now ships agent skills for their new services. The factory and the platform are converging.

Resources

The software factory

The core idea is one separation: the skill is the HOW, the brief is the WHAT.

Everything project-specific lives in a single PROJECT.md in the repo: name, one-liner, the plans table, the palette and fonts, the entities, which services are enabled. Everything reusable lives in the skills. When I want a different product, I write a different brief. The factory doesn't change.

There are four stages, each a separate skill, each reading that same brief:

Foundation (set-project-structure) builds the skeleton: Next.js with the directory contracts, the Drizzle schema, env keys with comments on where to get each value, design tokens, and placeholder clients for every enabled service. It deliberately builds less: seams, not features.
Landing (set-landing-page) is the taste stage. It studies the brief's reference sites, writes real copy before layout, and works against an explicit banned-patterns list: no gradient hero, no "Supercharge your workflow", no fake testimonials.
Backend (set-app-backend) is the correctness stage. Money and limits. Every quota, credit, token budget, and dollar is enforced on the server, atomically, idempotently. More on this below, because this is where Neon's new services plug in.
Dashboard (set-dashboard) builds the app people live in: the calendar, the modals, the streaming AI chat, consuming only the contracts stage 3 exposed.

Each stage stops at an approval gate before building, verifies current package versions instead of trusting its training data, and must leave the repo shippable when it ends. No half-built sections, no "we'll polish later".

Click through the stages here.. this is the actual pipeline that built pastmonday:

▶ The software factory pipeline (interactive), an interactive widget, runs on the original post.

I've written the first three stages up in detail before, if you want the deep dives: the concept, the brief, and the landing page.

What matters for this post: a factory like this is only as good as the surfaces it builds on. Stage 3 is where the factory has to wire auth, storage, data access, and LLM calls, and that's usually where you accumulate vendors. One for identity, one for buckets, one for model routing, plus the database. Four dashboards, four bills, four sets of credentials to leak.

This time, all four seams landed on the same platform.

One backend

Here's what pastmonday actually runs on, and the state of each piece:

Service	Role in pastmonday	Status
Postgres	entries, profiles, billing, referrals, asset metadata	GA
Neon Auth	signup, login, sessions, the identity table	GA
Data API	PostgREST-style HTTP reads	GA
Object Storage	user file uploads, S3-compatible	private preview
AI Gateway	all LLM calls, metered per request	private preview

A few receipts from the codebase, because "it works" is cheap to say.

Auth ships its own schema. Identity lives in neon_auth.user, and the factory's rule is: never mirror it. App tables hold foreign keys straight into it, so there's no sync job, no webhook that copies users around, no second source of truth:

// Neon Auth's identity schema, referenced, never mirrored
export const neonAuth = pgSchema("neon_auth");
export const authUser = neonAuth.table("user", { id: text("id").primaryKey() });

// profiles FK straight into it
userId: text("user_id").notNull()
  .references(() => authUser.id, { onDelete: "cascade" }),

Storage speaks S3, so there's no proprietary SDK to learn. The client is the plain AWS SDK, and it auto-reads the endpoint from AWS_ENDPOINT_URL_S3. Uploads are presigned PUTs with keys scoped per user, and since a presigned PUT can't enforce a size limit, the confirm route does a HeadObject and deletes anything oversized. The interesting part: Neon's storage branches with your database. Every branch gets its own isolated storage state. Your files and your rows stop drifting apart across environments.

AI Gateway is one API for models from Anthropic, OpenAI, Google and others, with per-request logging and rate limits, running on the same gateway infrastructure Databricks says handles 125 trillion tokens a month. For the app it means one token in the environment instead of a drawer full of provider keys, and the model is a config value:

const provider = createOpenAICompatible({
  name: "neon-ai-gateway",
  baseURL: `${NEON_AI_GATEWAY_BASE_URL}/ai-gateway/mlflow/v1`,
  apiKey: process.env.NEON_AI_GATEWAY_TOKEN,
});
export const chatModel = provider(process.env.NEON_AI_GATEWAY_MODEL);

Poke around the real architecture here, every box is clickable:

▶ pastmonday's backend, all Neon (interactive), an interactive widget, runs on the original post.

Building on features that shipped last week

In the last Neon post I made one argument: Neon is API-first, so an agent can reach a primitive the moment it exists, before anyone builds it a button. I proved it by building a feature Neon hadn't shipped yet.

This build pushed that idea further than I expected, twice.

First, the AI Gateway is so new that its official AI SDK provider package wasn't even installable yet when the factory ran stage 3. A human would have stopped there. The agent didn't: it probed the gateway, found the OpenAI-compatible endpoint at /ai-gateway/mlflow/v1, verified it live with a real request, and wired the app through the AI SDK's generic OpenAI-compatible provider instead. The code carries a comment saying "swap to the official provider once it installs". That's what API-first buys you: the endpoint existed, so the missing package didn't matter.

Second, the convergence I didn't see coming: a week after I published my factory skills, Neon shipped agent skills of their own.. instruction files that teach coding agents how to integrate Functions, Storage, and the AI Gateway, installable with one command:

npx skills add neondatabase/agent-skills

Read that again from where I'm standing. I write skills that teach an agent how to build products. Neon now writes skills that teach the same agent how to consume their platform. The factory and the platform are meeting in the middle, and neither side needed the other's permission. This is what it looks like when a vendor treats agents as the primary developer.

The part that has to be exactly right

A factory that ships beautiful broken apps is a liability, so stage 3 is paranoid by design. pastmonday charges money and sells limits: Free gets 1 entry and a 5,000-token AI budget, Pro gets 10 entries and 20,000 tokens a cycle, Max is unlimited. Limits only mean something if they can't be raced.

One constraint made this sharper than usual: the factory wired Drizzle through Neon's HTTP driver, which doesn't do interactive transactions. No BEGIN, hold the lock, decide, COMMIT. So every money-critical operation had to become a single atomic statement. This is the actual entry creation, quota check and insert in one round trip:

WITH consumed AS (
  UPDATE billing
     SET used_this_cycle = CASE WHEN used_this_cycle < monthly_quota
                                THEN used_this_cycle + 1 ELSE used_this_cycle END,
         extra_credits   = CASE WHEN used_this_cycle >= monthly_quota AND extra_credits > 0
                                THEN extra_credits - 1 ELSE extra_credits END
   WHERE profile_id = ${profileId}
     AND (used_this_cycle < monthly_quota OR extra_credits > 0)
  RETURNING id
)
INSERT INTO calendar_entries (user_id, title, ...)
SELECT ${userId}, ${title}, ...
WHERE EXISTS (SELECT 1 FROM consumed)

Quota first, then credits, else nothing gets inserted and the caller gets a 402. Two concurrent requests can't both take the last unit, not because the code is careful, but because the statement can't be split.

The same paranoia runs through the rest: the AI budget is checked server-side before every gateway call and metered from real usage on finish, Stripe webhooks dedupe on event id in a processed_stripe_events ledger so a replayed event can't grant credits twice, and quota-exhausted returns a typed 402 the UI turns into an upgrade prompt instead of an error toast.

None of this is Neon-specific, but it's the standard the factory holds, and the new services slotted into it without a fight. That's the actual test of a backend platform: not the demo, the enforcement paths.

Where this goes: branches that fork the whole stack

My favorite Neon primitive has always been branching. I've built preview environments where every pull request gets its own database branch, created and destroyed by the agent through the API.

The new services extend that idea past the database. Storage state branches with your data today, and Neon manages Functions through the same CLI and API as everything else. The direction is obvious and they're saying it out loud: a branch should fork the backend, not just the schema.

For a software factory, that's the missing piece. Today my factory pipeline validates each stage against the production-shaped environment it builds. The version I want is: every factory run, every agent session, every experiment gets a full isolated copy of the backend.. rows, files, functions.. for the lifetime of a task, then vanishes. Database branching already proved this workflow for schema changes. Extending it to the whole backend turns "spin up a staging environment" from an afternoon of Terraform into a flag on a CLI call.

That's the same story as the spending-caps feature from my last post, which is still marked "coming soon" by the way, and I'm still holding them to it: the primitives ship first, through the API, and the people building on top get to run ahead of the roadmap.

Closing

The pitch for this stack isn't "Neon does everything". It's narrower and more honest: a full working SaaS.. identity, data, files, AI, money.. now fits in one brief, four skills, one backend vendor, and one Stripe account. pastmonday went from empty repo to production on my cluster in about 40 minutes, and most of that time I was making taste decisions at the approval gates, not doing integration work.

If you want to go deeper: the video covers the factory concept, the skills are on GitHub to steal, the series walks each stage, and pastmonday.com is the living proof.. go create an entry, that's a race-safe atomic quota consume you're clicking.

And if you're at Neon: ship the branch-forks-everything future. I have a factory waiting for it.

Neon ships to the API first, so build the feature yourself

kuberdenis — Tue, 20 Oct 2026 07:07:08 +0000

Everyone is talking about how agents can build anything now. And they mostly can. But an agent is only ever as good as the surface you point it at, and most platforms give it a terrible one: a dashboard full of buttons it can’t press, a “coming soon” page, a feature locked behind a UI that no agent will ever click through.

Neon is different, and that difference is the whole reason I keep reaching for it when I’m building with Claude Code.

Neon is API-first. In my experience, new capabilities show up in their API long before they’re a nice button somewhere, and a lot of things never become a button at all. The console you click around in is just one client of that API. Which means: if a primitive exists, your agent can reach it, read the docs for it, and compose whatever it needs out of it, before anyone ships you the polished version.

This post is me proving that to myself, out loud. I had an itch Neon doesn’t officially scratch, so instead of waiting, I pointed Claude Code at the API and built the missing feature. Twice in this post there’s a little interactive thing you can play with.. they’re not screenshots, they’re running the actual logic in your browser.

What “API-first” actually means for an agent

When I say a platform is good for agentic coding, I mean something very specific: the thing my agent wants to do is expressible as an API call, and the docs to do it are readable.

That’s it. That’s the whole bar, and it’s shocking how many tools fail it. An agent can’t “open settings and toggle the thing.” It can’t watch a launch video. It reads docs and it makes requests. So the best platform for an agent is the one whose entire surface is an API, with reference docs that read like a contract.

Neon is exactly that. Projects, branches, compute endpoints, autoscaling bounds, consumption metrics, quotas, billing.. all of it sits behind api-docs.neon.tech as plain REST. I leaned on this hard in Agentic Workflows: Branched Environments, where the entire preview-environment trick is just the agent calling the branch API on every pull request. No clicking. No waiting for a feature. Just primitives, composed.

So when I hit a wall recently, my instinct wasn’t “I wish Neon had this.” It was “the primitive is probably already in the API, let me go build the wall-climbing tool myself.”

How serverless compute actually bills you

Before the wall, I need you to understand why Neon is even billed the way it is, because it’s the thing that makes the rest of this post matter.

Most databases you’ve rented are billed flat. You pick a box (some vCPUs, some RAM, an RDS instance, a droplet, whatever), and you pay for that box every hour of every day whether you touch it or not. You provision for your peak, because you have to survive your worst Tuesday, and then you pay for that peak 24/7 even though your worst Tuesday happens for about twenty minutes a month. The other 729 hours, you’re paying for an idle machine to sit there and be ready. It’s a hotel room you booked for a year so you’d have somewhere to sleep three nights.

Serverless compute flips that. Neon doesn’t sell you a box, it sells you what you actually consume. Compute is metered in CU-hours (a CU, a “compute unit”, is about 1 vCPU and 4 GB of RAM), it autoscales up when load arrives and back down when it leaves, and crucially it can scale to zero when nothing is connected. Storage and egress are metered separately, on their own dials. You’re billed for the area under the curve, not for the height of the tallest spike.

Here’s the part people miss: done right, this is cheaper than flat, often dramatically. Think about what your databases actually do. A staging environment that gets used three hours a workday. A side project nobody visits at 4am. A preview branch that lives for the length of a pull request and then dies. A little agent experiment that runs in bursts. Under flat billing you pay for all of those as if they were busy 24/7. Under serverless they cost roughly what you use, and the idle time trends toward free. The same workload that needs an always-on box sized for peak might bill a fraction of that when only the active hours count.

But (and you can feel the “but” coming) the exact thing that makes it cheap makes it dangerous. Flat billing has a ceiling built in for free: the box can only do so much, and you already paid for it, so the worst case is “it gets slow.” Usage billing has no ceiling. A runaway query, a retry loop, an agent that decides to reindex everything, a traffic spike, a bit of egress someone’s scraping.. none of that hits a wall. It just scales up, serves the load, and bills you for it. The curve that trends toward free when you’re idle trends toward terrifying when you’re not.

So to safely capture the “cheaper than flat” win, you want two things: a floor (scale to zero, so idle is free) and a ceiling (a hard cap, so a bad night can’t bankrupt you). Neon gives you the floor for free. The ceiling is where my wall is.

The itch: a “Coming soon” I didn’t want to wait for

I run a lot of autonomous-ish things on Neon. If you’ve read the Clopus posts you know I’m comfortable handing a Claude Code instance the keys and letting it run. And given everything I just said about unbounded usage billing, you can see exactly what keeps me up at night: it isn’t the agent doing something weird, it’s the bill. An autoscaling Postgres that an agent (or a traffic spike, or a runaway loop) can push to 8 CU while I’m asleep is a genuinely scary object to own.

So I went looking for the ceiling: a hard spending limit. And Neon does have a spending limit. You set a monthly dollar number, and when you cross 80% and then 100% of it, your org admins get an email. Genuinely useful.. but it’s the soft version. It never actually stops anything. Your database keeps happily burning money while you read the email.

Here’s the part I want to be fair about, because it’s the whole mood of this post: the hard version is clearly on the way. Open the spending-limit dialog and there’s a “Suspend projects” toggle sitting right there.. wearing a “Coming soon” badge. They’re building exactly the feature I want.

I just didn’t want to wait for the changelog.

That’s the honest framing here. This isn’t “Neon is missing something, boo.” The feature is coming. But I wanted it today, running tonight, before my next autonomous experiment, and an alert is not a cap. I wanted a number that, when crossed, turns the compute off. And because Neon is API-first, “coming soon” for the dashboard doesn’t mean “coming soon” for me. So I went and looked at what the API already exposes.

The levers the API hands you

This is the part that made me grin. The console gives you exactly one spend control, and it’s the toothless one. The API gives you the levers to build a real one. Click through these, it’s the actual difference that made the tool possible:

▶ Neon API levers (interactive), an interactive widget, runs on the original post.

Read that left-to-right and the story tells itself. The console: an alert. The API: a way to read true spend, a way to set a per-project cap that suspends compute, and a way to flip an endpoint off the instant you decide to. The feature didn’t exist, but every ingredient for it did. That gap, between “the product doesn’t have it” and “the API absolutely lets you build it,” is the gap agentic coding lives in.

So I didn’t file a feature request. I opened Claude Code.

Building the thing

The opening prompt was basically just me describing the itch and telling it where to read:

Neon’s native spending limit is alert-only, it emails you but never stops compute. I want a small tool that reads a project’s real usage from the Neon API, converts it to dollars, and actually suspends compute when I cross a monthly budget I set. Read the API reference first (api-docs.neon.tech), figure out which endpoints expose usage and which ones can stop a project, then build it.

It read the docs, found the consumption endpoints and the quota field, and had a working first pass fast. Convert usage to dollars, set a compute_time_seconds quota equal to the dollars-of-compute I had left, let Neon’s own metering pull the trigger. Clean.

Then I actually stress-tested it, and learned the lesson the hard way.

The per-project usage field you get from GET /projects/{id} (the obvious, no-org-id-required one) lags, and can freeze badly. In live testing it stuck at one value while real usage was around 70x higher. And because the project quota is gated by that same frozen meter, a quota I’d set 5x under what I was actually spending simply never tripped. The “cap” did nothing. I was over budget and the safety net just sat there.

That’s the kind of thing you only find by running it for real, and it’s exactly where API-first pays off a second time. The fix wasn’t to wait for Neon to make the field accurate. It was to route around it, because the API already had better roads:

The per-project usage field is freezing, it’s reading ~70x under real usage, so the quota never suspends. Switch the usage source to the consumption_history/v2 API (it needs an org_id but it’s invoice-accurate), and stop trusting the quota as the trigger.. disable the endpoint directly the moment accurate spend crosses the budget.

Two API calls I hadn’t even needed in v1 turned a broken backstop into a real cap:

consumption_history/v2 for truth: billing-grade usage that matches the invoice, not the field that lies.
PATCH endpoint disabled=true for the stop: meter-independent, immediate, no waiting for Neon’s metering to catch up.

npx neon-budget check     # read-only: what am I spending vs budget?
npx neon-budget enforce   # apply the cap (quota + endpoint disable)
npx neon-budget watch -i 15m   # run it on a loop

The whole thing is a few hundred lines, it’s on GitHub, and I want to be clear about why it exists: not because I’m smart, but because Neon’s surface let an agent and me build, in an afternoon, a capability the product doesn’t ship. That’s leverage.

Play with it

Here’s the budget logic, running live in your browser. Drag the sliders: set your plan, your average compute, how many hours it’s actually active (this is the serverless dial from earlier.. push it below 730 and watch what you save versus an always-on box), storage, egress. Then pick a monthly budget and a policy. It computes the real Neon cost and shows you exactly what neon-budget would do: just alert, throttle the autoscaling ceiling down in tiers, or hard-stop and disable the endpoint.

▶ Neon Budget Simulator (interactive), an interactive widget, runs on the original post.

This isn’t a mockup, it’s the same cost math and the same decision function that runs in the tool, ported straight into the page. Push the compute up, watch the hard-stop trip. Drop egress under 500 GB and watch that line go to zero (it’s included). The dollars to compute_time_seconds quota it prints is the literal value the tool PATCHes onto your project.

Why this is the real reason Neon wins for agents

Step back from spending caps for a second, because the tool is just the example.

The reason Neon is so good for agentic coding isn’t any single feature. It’s that the feature you want is almost always already expressible, because they build API-first. When the primitive lands in the API early, the gap between “Neon doesn’t do that yet” and “I have it running” is one good prompt and an afternoon, not a roadmap.

That changes how you work. You stop treating the product’s feature list as your ceiling. The branch API gave me preview environments before I “had” preview environments. The consumption and endpoint APIs gave me a hard spending cap that, strictly speaking, the product still doesn’t offer. None of that required Neon to ship me anything new. It required them to have been honest enough to expose the platform as an API, and an agent willing to read the docs.

That’s the bet I’m making with agentic coding in general: the winners will be the platforms that are complete at the API layer, because that’s the only layer an agent can actually build on. Neon is the most complete one I’ve found. So I keep building things it doesn’t have yet, and so far, it keeps already having the pieces.

Closing

Don’t wait for the button. If the primitive is in the API, it’s already yours: point your agent at the docs and go build the feature. With Neon, more often than not, the hard part is already done for you, sitting quietly in the API reference while everyone else waits for the changelog.

Thank you for reading!

Agentic Workflows: Branched Environments

kuberdenis — Sun, 18 Oct 2026 07:02:28 +0000

Agentic workflows are becoming more and more valid, especially in enterprise. While for personal development, you will mostly get away with tmux+, in a professional environments, high speed and lack of policy will get you in trouble.

For this practical exploration session, I onboard my project onto Neon and re-create my "Preview Environments” implementation to turn into a truly separated (branched) environment for every new Pull Request (which in itself is generated entirely by an agent). All in Git.

The repositories for this project are here:

Complete example: [kubeden/txtwrite]
Codex & Claude Code skills for your agents to “one-shot” it for you:
1. Codex: [kubeden/skills/codex]
2. Claude Code: [kubeden/skills/claude-code]

Note: since this implementation relies heavily on Neon’s (or Databricks’ Lakebase) architecture, the “plug-n-play” repo (2.a., 2.b.) requires your database to be either a Neon DB or a Lakebase DB.

If you enjoy videos, here is the entire content here, but on Youtube!

▶ Watch: Interactive widget

Why care & what does this solve?

I’ve been thinking a lot about agentic workflows lately. I am a very heavy AI user, and with a great ton of curiosity in the space. Often I can’t help but notice how similar terminal agents and DevOps practices are… it is almost like they were made to be used together. DevOps heavily relies on automation and what better direction to go than to put AI in the very foundation of automation…?

Convenient Interface

One of the great benefits of this system, is that it has a single convenient interface. Be it Git, or Slack, it is a single interface. You can open issues from your phone, get instant notifications in your email on failures, and get both a UI and a CLI for you OR your agent to track every step of what’s happening.

Tracking & Collaboration

With Git specifically, this workflow benefits from various tracking & collaboration possibilities. Multiple people can collaborate on issues & PRs. Reviewing the results. Seeing what works and what doesn’t.. and so on, and so on. Plus, you can always see closed PRs, and if your team follows standards practices i.e. merges small, incremental PRs, you get a very good “disaster recovery system”-like system through Git’s history.

Another great benefit from Github (or any Git provider)’s set of functionalities, is that you can set policies to disallow all kinds of negative actions!

Sandboxed environment

I was thinking a lot about this… you can configure a single VPS and install your agent there, but I am a little concerned how the said "agent dev vm” will not be ephemeral… which, combined with the recent high number of CVEs, is in my opinion, certain trouble. It would be very convenient, though…. So anyway, this particular implementation that I am writing about here, is ephemeral meaning after every CI finishes, the agent machines are destroyed so you don’t have to worry about managing long-lasting environments.

Free… and flexible

Well… it is free. I see people attempting this with external services, and many startups seem to be providing similar experience to their end users, but for I have always believed that for good engineering, you (or your team) need to own your resources. Self-management gives you the most flexibility, and with flexibility, you are only bottlenecked by your imagination (and laziness).

Prerequisite Knowledge

In order to make this article understandable for everyone, I am going to go over the main concepts utilized further.

What is a…?

Preview Environment

Preview environments are such environments, that are an exact replica of your production (or staging) environment, that are being spawned on-the-go. Preview environments are very useful for rapid development, as one can wire all kinds of test flows to trigger as the environment gets created, as well as get any person to start using/testing it immediately. There is no DevOps/Infra person required for this.

Agentic Workflow

There are mixed opinion in the community about what exactly an “agentic workflow” is — is it a split-paned tmux session with multiple terminal agents on display? Is it a Linear-Slack-ClaudeCode integration? Maybe a custom written agent with agents SDK that reacts to a pushover notification? Well… in my opinion, all of these are agentic workflows. I think of the word “workflow” as a set of actions that are combined into a system. And “Agentic” workflows, are a set of actions an AI agent goes through/executes, based on a trigger (and possibly… criteria).

Database Branching

Now… when I first started playing around with the concept of preview environments & AI, I thought I knew what database branching was. Well, I was wrong. You see, I thought that to “branch” a database was just a fancy way to say “make a copy of it.” Turns out… it is a lot more neat! Database branching, when “true” (as in, true branching), becomes practical when the compute and storage layers are separated. This way, at the point of separation (when a new branch is created), new writes are stored as deltas at the storage layer, which allows only the diverged pages to be stored separately and not the entire database. As such, your database is not being fully cloned. Instead, a new logical space of its state is created, where you can safely make changes, and later on have the ability to apply the desired changes back to your real database (through migrations, schema diffs, restore/reset workflows, or application-level data movement, depending on what changed)… essentially materializing only the diverging parts of that logical space.

This technique enables some otherwise hard-to-achieve features:

Setting a TTL on a test branch i.e. automatic cleanup
Fast restore in the case of an accidental delete action / similar
Quick iteration (branching) without interfering with production’s performance
Branching off a development branch without having to deal with the syncing of a separate development instance

The first three paragraphs of this post do a very good job in explaining it: Database Branching in Postgres: Git-Style Workflows with Databricks Lakebase

Architecture

The following is a high-level diagram for how the different system component all depend on and interact with each other. For the deeper, workflows-based diagrams, read further.

High-level diagram

The implementation has 3 core parts:

1/ Github: The agentic workflow “engine”

2/ Application infra (K8s+Neon): The place where my app lives (Frontend-only)

3/ Database infra (Neon): The place where my app becomes dynamic (Backend)

How it works

The foundation is standard: there is a Github repository, which contains a React codebase that holds the frontend for a web-based markdown editor. It uses the /src structure, and uses deno for its runtime environment.

Further, for backend, I use Neon. Mainly for the managed Auth, as well as (obviously) for a database.

Upon a desired change, I create an Issue and tag my agent <@kubeden-agent>. This triggers the , which:

Verifies trigger policies
Collect full issue context for the agent
Installs all required software in the runner
Creates (or reuses) the separate GIT branch
Creates a separate Neon preview branch
Boots the agent (Codex) and feeds it the context
Agent implements the code changes
Runs configured checks & migrations against the Neon preview branch

At the point where the agent finishes its work, the workflow:

Commits changes & pushes to the branch
Opens /or updates/ a Pull Request
Updates the status reply in the Issue
Dispatches the preview workflow

Workflow I: Issue to PR

At this point, there is a Pull Request opened, which triggers the next major CI , which goes like this:

Checks out the PR commit
Creates or reuses the Neon preview branch
Runs migrations/checks
Builds a new container image (with the branch-specific Neon URLs for auth, and data) & pushes it to the registry
Updates & pushes the ArgoCD manifests (in the infra repository)
ArgoCD creates a new application through the AplicationSet
Polls the new ApplicationSet app status
Updates the status reply with the preview URL

Workflow: Preview Deployment

Some details worth mentioning are that Github runs a webhook against my ArgoCD upon the push in my infra repository so that the preview application creation can be immediate.

And in the case where I add a reply to my PR i.e. when I am not satisfied with the initial attempt, it runs the Preview CI again, with the only change that it reuses the already live resources, instead of spawning new ones.

And here is the entire diagram.

Complete Issue→PR Agentic Flow

Conclusion

This is the second time I am doing a preview environment-like Github integration. I believe preview environments, as a technique, are the perfect fit to combine with agentic workflows. The reason why I returned to this implementation is that I was left triggered by my previous “workaround” with Supabase, and the table separation felt a little wrong. It was at this point I talked to a friend, with whom I discussed database branching and after a little knowledge transfer, he showed me Neon (and the blog post I linked to in the beginning of this post). I was mainly excited about the technique they use to separate compute from storage, which is exactly what makes true branching possible. Not only, but having the “managed backend” functionality I enjoy so much, with a Git-like iteration, makes me think we can really get into some complex implementations like the recently-viral “product factories”.

I think using AI is obviously required in our current times, but I also think many engineers skip a crucial step in their thinking phase-they forget to combine their pre-AI knowledge with all the new skills and concepts they are passively gaining & being attached to. It’s like many of us separate what worked pre-AI, and what works now, post-AI. Techniques like Preview Environments, GitOps, and major DevOps practices, are honestly only making agentic workflows more powerful.

So… if you are reading this post as a pre-AI engineer, maybe don’t get that excited about how amazing the agents are, and remember who you are-how deep you can think. And when you do, combine your deep thinking & extensive experience WITH the AI agents capabilities. I am sure the results are going to be amazing.

Thank you for reading.

After-post notes

N1: If this post inspired you to think, feel free to email me and start a discussion! I love talking technology, and even so if you are, too, actively living in “the arena”.

You can find everything about me on denislavgavrilov.com/about

Thank you for reading and have a good one!

How to do a zero-downtime schema change with Neon

kuberdenis — Sat, 17 Oct 2026 07:00:18 +0000

The following post is a recap of my Claude Code sessions doing a schema change the careful way. Although the content is AI-generated, I go through the entire post myself and every number in it comes from a run I watched happen. If you'd like to discuss anything in relation to this directly with me, you can do so on dennis@a2w.io or on x.com/@kuberdenis.

Splitting a column is the smallest interesting schema change there is. users.name becomes users.first_name and users.last_name, because someone finally wants to write "Hi Ada" in an email.

The one-line version of that change is a rename, a backfill and a deploy, run in whatever order you get to them. It works on your laptop. On a table with two hundred thousand rows and traffic on it, the same three steps give you a stretch of 500s in the middle, and the length of that stretch is however long the slowest of the three took.

The pattern that fixes it is old and has a name: expand and contract. Add the new shape, run both shapes at once for a while, then remove the old one. What is new is that you no longer have to be brave about it. A Neon branch is a full copy of the database in about two seconds, so every step can be run somewhere disposable first, read, and only then pointed at the real thing.

I built the whole thing to see whether "zero downtime" survives being measured. It does. 319,568 requests went through the app while the migration ran, and none of them failed.

Resources

kubeden/diffium-db, the tool used to read what each rehearsal did
Neon branching, the copy-on-write branches this leans on

What is running

A small orders API in Bun, two instances of it on two ports, and a proxy in front that only routes to a port whose health check is passing. Behind them, Postgres 18 on Neon with app.users and app.orders, seeded with 200,000 users and 100,000 orders.

The HTTP contract never changes. A user is posted and read back as { name, email } on the first request and on the last one. Only the storage underneath moves. That is the whole trick, and it is why the app can be deployed four times during the change without anything outside noticing.

Two instances behind a proxy is the part people skip in blog posts, and it is the reason the pattern exists. During a rolling deploy, old code and new code are talking to the same database at the same time. Every rule below follows from that one sentence.

The load generator holds 24 clients open and each of them does the same four requests in a loop: create a user, read it back, place an order, list the orders. It counts failures, and it also compares the name that comes out with the name that went in. Counting 200s is not enough. A migration that loses half a name returns 200 all the way down.

The instrument, before anything moves. wrong is the column that matters.

The order of operations

Six steps, and the order is the whole content of the pattern.

Expand. Add first_name and last_name, nullable. Add a trigger that keeps them in step with name.
Deploy the dual write. The app now fills both shapes. It still reads name.
Backfill. Fill the new columns for every row that predates step 1.
Constrain. Say the new columns are always present, without locking the table to prove it.
Flip the reads. The app answers from first_name and last_name.
Contract. Deploy an app that never mentions name, then drop the column.

Each of steps 1, 3, 4 and 6 touches the database. Each of them gets rehearsed on a branch first.

Rehearsing on a branch

Every database phase goes through one script first. It cuts a branch off production, takes a diffium-db baseline on it, applies the phase, prints what changed, and deletes the branch. Without the timing and the tidying, that is four commands:

neonctl branches create --project-id "$NEON_PROJECT_ID" --name "$BRANCH"
URI="$(neonctl connection-string "$BRANCH" --project-id "$NEON_PROJECT_ID")"

DATABASE_URL="$URI" diffium-db snapshot --schema app
psql "$URI" -v ON_ERROR_STOP=1 -f "sql/$PHASE"
DATABASE_URL="$URI" diffium-db diff --schema app

The branch came up in 2.4 seconds holding the same schema and the same 200,000 rows, because it is copy-on-write off its parent rather than a restore. Applying the expand phase to it took 0.7 seconds.

The expand phase, run on a copy. Two columns, a trigger, and a helper function. Nothing else.

Reading the diff is the point. psql told me the phase succeeded. It did not tell me what it did, and "what it did" is where a migration written by an agent surprises you. I got a list of three changes and no fourth one, on a database I was about to throw away.

1. Expand

Two nullable columns with no default, so Postgres writes catalogue rows and never touches the table itself. Table size does not enter into it.

begin;
set local lock_timeout = '2s';

alter table app.users add column first_name text;
alter table app.users add column last_name  text;

lock_timeout is the line people leave out. An ALTER TABLE that cannot get its lock waits, and while it waits it blocks every reader queued behind it. A migration that fails in two seconds is a migration you rerun. A migration that waits is an outage with a different name.

Then the trigger, which is the load-bearing part of the whole post:

create trigger users_name_sync
    before insert or update on app.users
    for each row execute function app.users_name_sync();

Whichever shape a writer filled in, the trigger derives the other one. Application code doing the same thing is not enough, because during the deploy in step 2 there is an instance still running that has never heard of first_name, and psql and agents are writers too. The sync belongs where every writer goes through it.

On production it was over inside a second. Here is the traffic across that moment.

failed 0 on both sides of the DDL.

And the table afterwards: both columns exist, and 215,400 rows have nothing in them.

200,000 of those are the seed. The other 15,400 arrived from live traffic between the start of the run and the expand.

2. Deploy the dual write, and what that deploy costs

The app now writes name, first_name and last_name on every insert. It still reads name, so nothing about its answers changes. This deploy is invisible from outside, and it is the last moment where rolling back is free.

The deploy script replaces one instance, waits for it to answer, then replaces the other. My first version of it dropped 25 requests every time it ran, and the schema change had nothing to do with it.

The instance was exiting the moment it was told to. Requests already inside it died with it. The fix is three steps in a fixed order, and it is worth writing out because it is the actual mechanism behind the phrase "zero downtime":

draining = true;              // 1. start failing the health check
await Bun.sleep(600);         //    and let the proxy notice
server.stop(false);           // 2. close the listener
while (server.pendingRequests > 0) await Bun.sleep(10);   // 3. wait

That took it from 25 failures to 6. The remaining 6 were the new instance passing its health check while its connection pool was still cold, then queueing two hundred requests behind a TLS handshake until the runtime's 10 second request timeout killed them. Warming the pool before opening the port took it to 0.

I am including this because both bugs are in the deploy, not in the migration, and that is the general case. Expand and contract is a set of rules for making the schema safe to deploy through. It cannot save a deploy that drops connections on its own.

3. Backfill

Rows written since the expand already have both shapes. Everything older is still name-only, and that is 215,400 rows.

Not in one statement. A single UPDATE over all of them holds row locks for its entire life, gives you nothing to watch, and cannot be stopped halfway. Batches of 2,000, committing as they go, with a partial index so that finding the next batch stays cheap once the easy rows are gone:

create index concurrently if not exists users_backfill_idx
    on app.users (id) where first_name is null;

CONCURRENTLY keeps writes running while the index builds, and it cannot run inside a transaction, which is why this file is deliberately not one.

215,400 rows in 15.6 seconds, in batches of 2,000. Nothing failed while it ran.

Nothing left to fill.

Before any of this, run the check that says whether the split is reversible at all:

select count(*) from app.users
 where btrim(name) is distinct from btrim(<the split, put back together>);

Zero means every name survives the round trip. Anything else means you have decided what happens to those rows, whether or not you know it yet. Mine was zero, which is a property of names I generated, not a property of names.

4 and 5. Constrain, then flip the reads

NOT VALID first, then validate. The first statement writes a catalogue row and takes the lock for that long. Every new and updated row is checked from that moment. The second statement reads the whole table under a weaker lock, and writes keep going while it runs.

alter table app.users
    add constraint users_first_name_present check (first_name is not null) not valid;

alter table app.users validate constraint users_first_name_present;

Then the app is deployed reading first_name and last_name, still writing both. This is the step that is safe only because step 3 finished. Flip the reads with rows still unbackfilled and half your users are suddenly called " ".

6. Contract

The rehearsal for this one is the one worth reading, because dropping a column is the step with no undo.

The column, the trigger, the function and the NOT NULL constraint that came with the column, all leaving together. On a copy.

The order is not negotiable. Deploy an app that never mentions name, confirm it is on every instance, and only then drop the column. Contract while one instance still references it and that instance starts returning 500s until someone restarts it.

begin;
set local lock_timeout = '2s';
drop trigger users_name_sync on app.users;
drop function app.users_name_sync();
alter table app.users drop column name;
commit;

0.81 seconds.

What the traffic saw

319,568 requests over 10 minutes 49 seconds. 0 failed, 0 with a wrong name, p50 41ms, p99 208ms. The migration is in there: two DDL phases, a 215,400 row backfill, a constraint validation and four rolling deploys, and none of it shows up as a change in the numbers.

The end state is 279,892 users, every one of them with a first and last name, and no name column.

What breaks it

Take out the trigger and everything above still passes its tests, right up until a rolling deploy has both versions live. Here is the whole failure in six lines:

create table control.users (id bigint generated always as identity primary key,
                            name text, first_name text, last_name text);

-- old instance, still running, writes the only column it knows about
insert into control.users (name) values ('Ada Lovelace');

-- new instance, just deployed, reads the new columns
select coalesce(first_name,'') || ' ' || coalesce(last_name,'') as api_returns
  from control.users;

 api_returns
-------------

(1 row)

The value is one space. Status 200. Correct content type. Ada's name is gone. This is the failure the whole pattern is arranged around, and it is invisible to anything that only counts error rates, which is why the load generator in this demo compares the name it wrote against the name it read.

Building it yourself

The demo is three TypeScript files, six shell scripts and seven SQL files, and everything in it that matters is quoted somewhere above.

The API is one file. APP_VERSION picks its storage layer, so the four deploys are four values of an environment variable instead of four commits:

v1  name is the only column there is
v2  writes both shapes, still reads name
v3  reads first_name / last_name, still writes both
v4  never mentions name. only now can the column go

In front of it, a proxy routes to whichever of the two instances passes its health check. Beside it, the load generator from the top of the post. The rest is a script per job. Start the two instances and the proxy. Roll a version out in the drain order above. Rehearse a phase on a branch. Print the columns and the null count once a second.

The SQL is the schema, the seed, the losslessness check, and one file per database phase.

Set the database up, and run the check before anything else. It has to come back 0.

psql "$DATABASE_URL" -f sql/00-schema.sql -f sql/00-seed.sql
psql "$DATABASE_URL" -f sql/check.sql

Then three panes. Traffic in the first, the database in the second, the steps in the third:

./bin/up.sh v1
bun src/load.ts            # pane 1, leave it running
./bin/watch-db.sh          # pane 2

./bin/rehearse.sh 01-expand.sql
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f sql/01-expand.sql
./bin/deploy.sh v2

./bin/rehearse.sh 02-backfill.sql
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f sql/02-backfill.sql
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f sql/03-constrain.sql

./bin/deploy.sh v3
./bin/deploy.sh v4

./bin/rehearse.sh 04-contract.sql
psql "$DATABASE_URL" -v ON_ERROR_STOP=1 -f sql/04-contract.sql

Three environment variables and that is the whole configuration. DATABASE_URL for whichever branch you are treating as production, NEON_PROJECT_ID so the rehearsal knows where to cut, and a diffium-db checkout so it can read what came back.

Any Postgres works for the app itself. Branching is what makes rehearsing every phase cheap enough that you do it every time instead of only when the change looks scary.

Why I like Neon

kuberdenis — Wed, 14 Oct 2026 08:09:17 +0000

I've been using Supabase for two years only until recently when I moved all my new projects to Neon. I figured I'd take the time and note down my thoughts on why this shift happened.

An agents-first company

When testing out Neon, I noticed that a few features were missing, only to find these features were available through the API. Which immediately made me a good impression as nowadays with the amount of intelligence LLMs have, and how good terminal agents are, I strongly rely on personal software - if I need something, I spawn it in a few hours. Which means I need a company that understands this and lets me interact with what I am paying for, from their API.

A quick example I can give, is: [kubeden/neon-budget] - a cli to enforce usage limits that I deployed on my k3s cluster.

Not only do they do this, but their documentation is very obviously made with the thought that "AIs are going to use this" in mind. See their MCP: [neon-mcp-server]

Speed of development

I am a fan of Backend-As-A-Service companies, mostly because of reliability. If I decide to self-manage a storage container, or a database, don't get me wrong - I will succeed. What I am mostly concerned about is my app failing / my cluster over-filling, and me having to deal with backups (which I won't have) and restore. So I am ok paying a small monthly fee to take this trouble away from me.

With Neon, when I started using them, they were still only DB-As-A-Service. I thought, "ehh... it would've been perfect if they had {feature xyz}" - and shortly after, about a month of me thinking this, they release storage, ai gateway, and functions. You can see it in their Discord, they are building at a very high velocity, which is great.

(see the changelog from 4 days ago [neon.com/changelog])

Branching

I think this is obvious.. if you're reading this, you are probably considering using Neon, so you already know about their branching functionality - creation time, separation of environment, etc. etc.

So I won't write much about this. I wrote another big blog post about branching that you can read here: Denislav Gavrilov: Branched Environments

What email APIs cost in 2026: 7 workloads priced on 6 providers

kuberdenis — Wed, 07 Oct 2026 16:34:23 +0000

Every email API prices differently. Some sell monthly tiers, some bill per email, and some charge for newsletters separately. So I priced the same seven workloads on Amazon SES, Mailgun, Postmark, Resend, SendGrid and SMTPfast.

Prices come from the official pricing pages on October 6, 2026, in USD. Sources are at the bottom.

TL;DR

Monthly cost, cheapest first.

Side project (1,000 emails): Mailgun $0 · Resend $0 · SMTPfast $0 · SES ~$0.11 · Postmark $15 · SendGrid $19.95
Early SaaS (20,000 emails): SES ~$2 · SMTPfast $19 · SendGrid $19.95 · Resend $20 · Postmark $29.50 · Mailgun $33
Growing SaaS (100,000 emails): SES ~$11 · SendGrid $34.95 · Resend $35 · SMTPfast $49 · Mailgun $75 · Postmark $115
High volume (500,000 emails): SES ~$53 · SMTPfast $349 · Resend $350 · SendGrid ~$387 · Mailgun $400 · Postmark $455
1M emails: SES ~$106 · Resend $650 · Mailgun $700 · SendGrid $733 · Postmark $775 · SMTPfast $849
Newsletter (5,000 contacts, 4 sends): SES ~$2* · SMTPfast $19 · SendGrid $25 · Postmark $29.50* · Mailgun $33* · Resend $40
Launch spike (20k plan, 60k month): SES ~$6 · Resend $29 · SMTPfast $29 · SendGrid ~$33 · Postmark ~$82 · Mailgun $105

*Sending only, with no contact lists or broadcast tools included.

Once you're past the free tiers, Amazon SES is the cheapest at every volume, by 3–10×. But it's raw infrastructure. Webhooks, logs and templates are yours to build. Among full email APIs, the cheapest option changes with volume: SMTPfast at 20k and 500k, SendGrid and Resend at 100k, and Resend at 1M.

How each one charges

Amazon SES: $0.10 per 1,000 emails plus $0.12/GB of data. Since July 2026, new accounts default to a $0.16 plan, and you can switch back to $0.10 at any time.
Mailgun: monthly tiers from $15 (10k emails). Extra emails cost $0.40–$1.80 per 1,000 depending on the tier.
Postmark: monthly tiers from $15 (10k emails). Extra emails cost $1.15–$1.80 per 1,000.
Resend: $20 (50k) or $35 (100k), then Scale from $90. Extra emails cost $0.46–$0.90 per 1,000 once you turn overage on. Newsletters are billed separately, per contact.
SendGrid: Essentials from $19.95 (50k), Pro from $89.95 (100k, with a dedicated IP). Extra emails cost $0.47–$1.33 per 1,000. Newsletters are a separate product.
SMTPfast: $9 (10k), $19 (50k) or $49 (200k), with newsletters included. Extra emails come from prepaid credits at $1 per 1,000. It runs on Amazon SES.

Free tiers

Amazon SES: no free tier for new accounts since July 2026, only AWS credits ($100–200). New accounts start in a sandbox that sends 200 emails a day to verified addresses until AWS approves you.
Mailgun: 100 emails a day, 1 domain, 1 day of logs. No card, but without one you can only send to 5 verified addresses.
Postmark: 100 emails a month, forever. No card. Postmark approves your account manually before you can email other domains.
Resend: 3,000 emails a month (100 a day), 3 domains, 30-day logs, inbound included. No card.
SendGrid: no free plan since 2025, only a 60-day trial at 100 emails a day.
SMTPfast: 3,000 emails a month, 1 domain, 1,000 contacts, 2 broadcasts. No card. Sending access is reviewed before you can send.

The scenarios

SES numbers assume about 50 KB per email and the $0.10 rate. Open "Show the math" under each one for the full calculation.

1. Side project

1,000 emails a month: magic links, password resets and receipts.

Free tiers decide this one. Mailgun, Resend and SMTPfast cover it for $0. Postmark's free 100 a month doesn't, and SendGrid's trial ends after 60 days.

Show the math

Mailgun: $0. Free plan, as long as no day goes over 100 emails.
Resend: $0. Free plan (3,000 a month, 100 a day).
SMTPfast: $0. Free plan (3,000 a month).
SES: ~$0.11. 1,000 × $0.10/1k + 0.05 GB × $0.12.
Postmark: $15. Basic 10k. The free plan only covers 100.
SendGrid: $19.95. Essentials 50K once the trial ends.

2. Early SaaS

20,000 transactional emails a month.

SMTPfast, SendGrid and Resend all land around $20 with room to grow to 50k. Plans that start at 10k pay overage for the second 10k.

Show the math

SES: ~$2. 20k × $0.10/1k + 1 GB × $0.12.
SMTPfast: $19. Growth (50k), with unlimited domains and open/click tracking.
SendGrid: $19.95. Essentials 50K.
Resend: $20. Pro (50k).
Postmark: $29.50. Pro 10k $16.50 + 10k × $1.30/1k.
Mailgun: $33. Basic $15 + 10k × $1.80/1k.

3. Growing SaaS

100,000 emails a month.

SendGrid and Resend both have a 100k tier around $35. SMTPfast's next tier jumps to 200k for $49.

Show the math

SES: ~$11. 100k × $0.10/1k + 5 GB × $0.12.
SendGrid: $34.95. Essentials 100K. Shared IPs only.
Resend: $35. Pro 100k.
SMTPfast: $49. Scale (200k), with a 99.9% SLA. Growth plus 50k of credits would be $69.
Mailgun: $75. Foundation 100k, with a dedicated IP included.
Postmark: $115. Basic 125k.

4. High volume

500,000 and 1M emails a month.

At 500k, SMTPfast and Resend tie at about $350. At 1M, SMTPfast becomes the most expensive here, because every email past 200k costs $1 per 1,000 in credits.

Show the math
500,000 emails:

SES: ~$53. 500k × $0.10/1k + 25 GB × $0.12.
SMTPfast: $349. Scale $49 + 300k × $1/1k in credits.
Resend: $350. Scale 500k.
SendGrid: ~$387. Essentials 100K $34.95 + 400k × $0.88/1k. Pro 700K, with a dedicated IP, is $499.
Mailgun: $400. Scale 500k.
Postmark: $455. Basic 700k.

1,000,000 emails:

SES: ~$106. 1M × $0.10/1k + 50 GB × $0.12.
Resend: $650. Scale 1M.
Mailgun: $700. Scale 1M.
SendGrid: $733. Pro 700K $499 + 300k × $0.78/1k.
Postmark: $775. Basic 1.5M.
SMTPfast: $849. Scale $49 + 800k × $1/1k in credits.

5. Newsletter

5,000 contacts, 4 sends a month (20,000 emails).

Only SMTPfast, SendGrid and Resend include contact lists and broadcasts. SMTPfast is the cheapest of the three because it doesn't bill newsletters as a separate product.

Show the math

SES: ~$2. Sending only.
SMTPfast: $19. Growth: 25k contacts, 50k emails, 50 broadcasts. Starter covers 5,000 contacts but only 10k emails.
SendGrid: $25. Marketing Campaigns Basic 10K (10k contacts, 30k emails). No automation on Basic.
Postmark: $29.50. Sending only, on a broadcast stream. Same price as transactional.
Mailgun: $33. Sending only. Mailgun has no broadcast product.
Resend: $40. Marketing Pro, 5,000 contacts with unlimited sends. Billed separately from transactional.

6. Launch spike

You're on the plan from scenario 2, and a launch month sends 60,000 emails.

Plans that include 50k absorb most of the spike. Plans that start at 10k pay $1.30–$1.80 for every extra 1,000.

Show the math

SES: ~$6. 60k × $0.10/1k + 3 GB × $0.12.
Resend: $29. Pro $20 + 10k × $0.90/1k. Overage has to be switched on, otherwise sends fail after 50k.
SMTPfast: $29. Growth $19 + 10k credits ($10). Credits are prepaid, otherwise sends are rejected after 50k.
SendGrid: ~$33. Essentials 50K $19.95 + 10k × $1.33/1k.
Postmark: ~$82. Pro 10k $16.50 + 50k × $1.30/1k.
Mailgun: $105. Basic $15 + 50k × $1.80/1k.

Hidden costs

Amazon SES: webhooks need SNS or EventBridge, per-message logs need the paid deliverability dashboard, and a dedicated IP is $24.95/month.
Mailgun: overage can't be turned off. The $15 plan has 1 domain and 1 day of logs; 30-day logs need Scale ($90+).
Postmark: overage can't be capped. The Basic plan has no inbound email, and DMARC monitoring is $14 per domain per month.
Resend: every To, CC and BCC address counts as an email. Newsletters are billed per contact on top of transactional.
SendGrid: Essentials has no dedicated IP option and only 3 days of activity history.
SMTPfast: no dedicated IPs on any plan. Extra emails cost $1 per 1,000, about 4× its effective Scale rate, and sends are rejected once the allowance and credits run out.

Which one should you pick?

For most startups: SMTPfast

$0 to start: 3,000 emails a month, no credit card.
$19 covers a lot: 50k emails, unlimited domains and open/click tracking (scenario 2).
Newsletters included: contacts, segments and broadcasts in the same plan (scenario 5).
No surprise bills: extra emails come from prepaid credits, so you never get an overage invoice.
Easy to switch: the API is Resend-compatible, so moving is a base-URL and API-key change.

Revisit the choice past about 300k emails a month or when you need a dedicated IP. At 1M, Resend or SES directly costs less.

Everything else

Amazon SES: the lowest price at any volume, if you're happy to build webhooks, logs and templates yourself.
Mailgun: many domains, inbound routing, and a dedicated IP from $75/month.
Postmark: 45-day logs and the same support on every plan.
Resend: the cheapest full API at 1M emails, and the 100k tier at $35.
SendGrid: 100k for $34.95, plus a marketing suite with automation on its Advanced plans.

Sources

Prices checked October 6, 2026.

Amazon SES: pricing, pricing plans, sandbox
Mailgun: pricing
Postmark: pricing, billing FAQ
Resend: pricing, quotas and limits
SendGrid: Email API pricing, Marketing Campaigns pricing, free plan change
SMTPfast: pricing, prepaid credits

Neon vs Supabase vs PlanetScale vs AWS: what Postgres costs in 2026

kuberdenis — Tue, 06 Oct 2026 15:09:07 +0000

Every database pricing page uses different units, so I priced the same six workloads on AWS (RDS and Aurora Serverless v2), Neon, PlanetScale and Supabase, with Convex on the side.

Prices come from the official pricing pages on October 6, 2026 (AWS us-east-1). Sources are at the bottom.

TL;DR

Monthly cost, cheapest first. "Aurora" means Aurora Serverless v2.

Side project (0.8 GB, ~4 h/day): Neon $0 · PlanetScale $5 · Aurora ~$7 · RDS ~$18 · Supabase $25
SaaS busy in business hours (10 GB, 4 GB RAM): Neon ~$27 · PlanetScale $28 · RDS ~$53 · Aurora ~$54 · Supabase $75
Mostly quiet with daily spikes to 16 GB RAM: Neon ~$50 · PlanetScale ~$97 · Aurora ~$99 · Supabase $225
A database for every PR (40 PRs a month): Neon ~$3 · PlanetScale ~$20 · Supabase ~$39
50 mostly idle per-customer databases: Neon ~$17 · Aurora ~$31 · PlanetScale $250 · Supabase $515 · RDS ~$700
Steady 24/7 load (4 GB RAM, 50 GB): PlanetScale $33 · RDS ~$57 · Supabase ~$82 · Neon ~$95 · Aurora ~$180

Usage-based databases (Neon, Aurora) win when the database has idle time. Fixed instances (PlanetScale, RDS) win when it's busy 24/7.

How each one charges

AWS RDS: fixed instance per hour, from $0.016/hr. Always on.
Aurora Serverless v2: $0.12 per ACU-hour (1 ACU ≈ 2 GB RAM). Scales to zero and wakes up in about 15 seconds.
Convex: per function call, GB stored and GB transferred. Not Postgres.
Neon: $0.106 per CU-hour (1 CU ≈ 4 GB RAM). Scales to zero and wakes up in a few hundred milliseconds.
PlanetScale: fixed clusters from $5/month. Always on.
Supabase: $25/month for Pro, plus about $10/month per project (the first one is covered). Always on.

Free tiers

AWS: $100–200 in credits, and the Free plan ends after 6 months. Card required.
Convex: 0.5 GB and 1M function calls a month. No card.
Neon: 1 GB per project (up from 0.5 GB on October 2), 100 projects, auth and storage included. No card.
PlanetScale: no free tier. Plans start at $5/month.
Supabase: 500 MB, 2 projects, auth and storage included. Pauses after 7 days without activity. No card.

The scenarios

Setups are matched by RAM, and a month is 730 hours. Open "Show the math" under each one for the full calculation.

1. Side project

0.8 GB of data, used about 4 hours a day.

Free tiers decide this one. It fits in Neon's 1 GB free plan but not in Supabase's 500 MB.

Show the math

Neon: $0. 120 h × 0.25 CU = 30 CU-hours, inside the free 100.
PlanetScale: $5. PS-5 single node.
Aurora: ~$7. 120 h × 0.5 ACU × $0.12, plus $0.10 storage.
RDS: ~$18. db.t4g.micro $11.68 + 20 GB storage $2.30 + public IPv4 $3.65.
Supabase: $25. Over the 500 MB free limit, so Pro. The $10 credit covers the instance.

2. SaaS busy in business hours

10 GB of data and 4 GB RAM, busy 10 hours a day on weekdays.

Neon and a PlanetScale single node tie at $27–28. Everything else costs 2–3× more.

Show the math

Neon: ~$27. 220 h × 1 CU × $0.106 + 10 GB × $0.35. The Launch plan has no SLA. The Scale plan, with a 99.95% SLA, makes it ~$52.
PlanetScale: $28. PS-40 single node, no SLA. The 3-node version is $83.
RDS: ~$53. db.t4g.medium $47.45 + storage $2.30 + IPv4 $3.65.
Aurora: ~$54. 220 h × 2 ACU × $0.12 + $1 storage, pausing outside business hours.
Supabase: $75. Pro $25 + Medium (4 GB) $60 − $10 credit.

3. Mostly quiet with daily spikes to 16 GB RAM

20 GB of data. Light load for 22 hours a day, then a 2-hour spike that needs 16 GB RAM.

Fixed instances have to be sized for the spike all month. Autoscaling databases pay for it only while it lasts.

Show the math

Neon: ~$50. (0.25 CU × 22 h + 4 CU × 2 h) × 30 days = 405 CU-hours × $0.106, plus 20 GB × $0.35.
PlanetScale: ~$97. PS-160 single node (16 GB) $96 + $1.25 extra storage.
Aurora: ~$99. (0.5 × 22 + 8 × 2) × 30 = 810 ACU-hours × $0.12, plus $2 storage.
Supabase: $225. XL (16 GB) $210 + Pro $25 − $10 credit. No autoscaling.
RDS: not priced. It has the same problem as PlanetScale and Supabase.

4. A database for every PR

40 pull requests a month, each with its own database for 3 days.

Neon branches cost almost nothing while idle. Supabase and PlanetScale branches bill every hour they exist, and AWS has no built-in branching.

Show the math

Neon: ~$3. 40 × 3 active hours × 0.25 CU × $0.106. Branches share storage with their parent.
PlanetScale: ~$20. $5/month per dev branch, prorated: 40 × 72/730 × $5.
Supabase: ~$39. A Micro instance per branch: 40 × 72 h × $0.01344.
AWS: no built-in branching. You can script snapshot restores or Aurora clones yourself.

5. 50 mostly idle per-customer databases

50 databases of 200 MB each, each active about 10 hours a month. If you put every customer in one shared database, see scenario 2 instead.

Idle databases cost only storage on Neon and Aurora. On fixed instances you pay for 50 servers.

Show the math

Neon: ~$17. 50 × 10 h × 0.25 CU × $0.106 + 10 GB × $0.35.
Aurora: ~$31. 50 × 10 h × 0.5 ACU × $0.12 + $1 storage.
PlanetScale: $250. 50 × $5.
Supabase: $515. 50 × $10 + Pro $25 − $10 credit.
RDS: ~$700. 50 × (db.t4g.micro $11.68 + storage $2.30).

6. Steady 24/7 load

50 GB of data and 4 GB RAM, busy around the clock.

Fixed instances win. If your database is busy more than about 61% of the month, a fixed RDS instance beats Neon on compute.

Show the math

PlanetScale: $33. PS-40 single node $28 + 40 GB × $0.125. High availability costs $98, because storage is billed on all 3 nodes.
RDS: ~$57. db.t4g.medium $47.45 + 50 GB × $0.115 + IPv4 $3.65.
Supabase: ~$82. Pro $25 + Medium $60 − $10 credit + 52 GB extra disk × $0.125.
Neon: ~$95. 730 h × 1 CU × $0.106 + 50 GB × $0.35.
Aurora: ~$180. 730 h × 2 ACU × $0.12 + $5 storage.

Where Convex fits

Convex is a reactive document database you query with TypeScript functions instead of SQL. It bills per function call ($2.20 per million after the free 1M), so a small app can cost pennies. Pro is $25 per developer per month.

Hidden costs

AWS: $3.65/month per public IPv4 address. RDS Proxy on Aurora costs at least ~$88/month and blocks scale-to-zero.
Convex: hitting the team spending limit disables every project in the team.
Neon: no hard spending cap, only email alerts. Persistent connections stop compute from scaling to zero.
PlanetScale: high-availability storage is billed on all 3 nodes.
Supabase: every project and branch pays for compute 24/7, and the spend cap doesn't cover compute.

Which one should you pick?

For most startups: Neon

$0 to start: 1 GB per project, 100 projects, no credit card.
You pay for usage: ~$27/month for a business-hours SaaS (scenario 2).
Cheap previews: ~$3/month for a database per PR (scenario 4).
A full backend: auth, storage, functions and a Data API that branch with the database.

Revisit the choice once your database is busy around the clock (scenario 6).

Everything else

AWS RDS: steady 24/7 load, or you need to stay inside AWS.
Aurora Serverless v2: inside your AWS VPC with idle time, if 15-second cold starts are fine.
Convex: a reactive TypeScript app that doesn't need SQL.
PlanetScale: the cheapest fixed instance for steady load.
Supabase: auth, storage, functions and realtime on one platform at a fixed monthly price.

Sources

Prices checked October 6, 2026.

AWS: RDS for PostgreSQL, Aurora, Free Tier
Convex: pricing
Neon: pricing, plans, backend launch
PlanetScale: pricing, Postgres pricing
Supabase: pricing, compute billing

gctx: git auth manager

kuberdenis — Tue, 27 Jan 2026 13:42:39 +0000

I created a kubectx-inspired git auth manager

kubeden/gctx:

sets a global git identity
rewrites origin to an ssh alias
allows deterministic auth for multiple accounts

Try it at: github.com/kubeden/gctx

How to build a good container platform

kuberdenis — Mon, 24 Nov 2025 21:01:34 +0000

How to build a good container platform

Description

Platform Engineering is the act of designing, provisioning, and maintaining a platform.

You provision resources.
You configure resources.
And you use your platform.

It sounds easy but it's not.
Most engineering teams fail.

Blocks

In order to build a good platform, you need to build logically.

Base (Cloud / On-Prem / Hybrid Cloud / Multi-Cloud)
Application Engine [(k8s / vms / serverless) in our case k8s]
Platform Components (monitoring, application, recovery, automation)
Application Components (ci/cd, configuration repo)
Subtleties (secrets management & rotation, states, processes)

Organizations hire teams, often to do one thing. Which is wrong.
How one team does something, is often not how the others will.

Expertise is often different, which leads to "know-how drifts" and technical debt.

Standartization

This is maybe the most important foundation block for a platform to thrive.

Standartization is almost always ignored:

name suffixes & directory structures
utility separation
unified deployment methods
documentation & knowledge transfers

These are the "atoms" in a good platform.

Name Suffixes & Variability

Name suffixes are your map points.
Directory structures are your streets.

Every organization should have strictly defined name suffixes and directory structures that should be applied across:

iac repositories
application management repositories
cloud resources
secret namings
service accounts
and so on.. and so on..

A good name suffix system includes 3 to 7 matching variables.

Example:

customer
region
environment
iteration
type

Continuing the example, this would translate to:

e.g. directory structure: ${customer}/${region}/${environment}/${type}-[${iteration}?]
e.g. terraform resource: ${customer}-${region}-${environment}-${iteration}-${type}

Imagine this at scale.
It is easy to navigate it now.
It is easy to understand.

It is.. nice to work with! :)

Utility Separation

Many organizations don't separate.

They create a repository called devops and they put everything inside:

terraform
bash scripts
yaml
helm
kustomize
ansible
and god know what else

Instead, you should be grouping logically:

infrastructure provisioning: manage the lifecycle of your resources
infrastructure configuration: manage the configuration on your "live" resources
application deployment: application configuration
application code: well... your application code

Here is an example of well-separated & logically-grouped repositories:

infrastructure: terraform, bash (as few as possible)
k8s-cluster-configuration: yaml, kustomize, helm, jsonnet
[application]-k8s-configuration: standartized releases / deployments / libs

This way, you achieve the following:

a single place to provision / depracate infra
a single place to provision / depracate platform components
a single place to provision / depracate applications

Platform Resources: Provisioning

Regardless of the technology you choose, you should handle provisioning of resources as a standalone practice.

Here is an exapmle of a good Terraform set-up:

Repository: infrastructure
Directory structure:

modules/
base/
.ci(or .github, or whatever the standard is)

Where in /modules, you treat it as "libs", a place where you manage the versioning of your modules (community or personal).
Example continuation of modules can looks like this:

modules/
- azure/
- azurerm/
  - key-vault/
  - community/
  - personal/
  - acr/
  - ...
- /azuread
  - ...
- /aws
- /control-tower
  - community/
  - personal/
- /rds
  - ...
- /s3
  - ...

And in base, you call these modules:

base/
- [customer]/
- [ region ]/
  - [ environment]/
  - common/
    - locals.tf
    - provider.tf
    - key-vault.tf
    - acr.tf
    - ...

You never click in the cloud provider.
But you also don't configure here.
You do not "hack" yamls.
You do not apply hard-to-figure-out scripts.

You only spawn and delete. :)

Platform Components: Configuration

In order to have a good platform, you need to be flexible, clear, and rock-solid in terms of its capabilities.

One of the most hardened platforms I've ever built (and I built it 3 separate times) is as follows:

Management:

ArgoCD: Management, IAM, GitOps.

Routing:

ingress-nginx: routing, one lb, reserve ips in your subnet, manage routing programmatically
cert-manager: automated ssl
external-dns: automated dns

Secrets Management:

external-secrets: vaults, stores, you name it
sealed-secrets: plain-text secrets in your repo (for stuff like the initial argocd secret)

Monitoring:

grafana-operator: well.. grafana :) graphs and stats
prometheus: metrics
blackbox-exporter: alerts
thanos (if multi-cluster): consolidation of metrics
eck: elk :)

Backups:

velero: backups

Scaling & State:

reloader: re-apply configs
keda: scale applications
karpenter (if on AWS, Azure have it by default): scale nodes

All of this should be managed through a single ArgoCD application called "cluster-apps" or similar.

Tools like Jsonnet and Kustomize are your best friends here.

(read about the app-of-apps pattern too)
((no i do not like applicationsets))

Application Configuration

Your application configuration repositories should be standartized.

A well-thought-of application configuration repository is no different than the infrastructure, or the platform repositories.

You should again think in the following direction:

releases: packages
libs: building blocks for /releases
deployments: calling your packages

You see, the same concept applies over and over again -- manageability, readibility, and standartization.

I usually go for Jsonnet, but I wanted something simpler, so I went for Kustomize, only to find myself in a very thight situation, and ultimately decided upon with Helm.

Jsonnet is awesome, but it often requires a lot of special expertise and in case your main engineer leaves, it is possible you're going to have problems finding adequate peresonel.

CI/CD

Since we are mostly relying on GitOps for our platform, the CI/CDs worth mentioning are:

infrastructure:
- terraform plan/apply: one pipeline per resource type (e.g. ${customer}/${region}/${environment}/${logical-group}/${resource-type}-${iteration}/pipeline.yml
- pipeline-of-pipelines: a global pipeline to manage the update / management of the child pipelines
application:
- build & test: well, build and test
- deploy: since we're relying on GitOps, our deploy is a single git line change :)
k8s-cluster-configuration:
- a pipeline to bootstrap the master argocd (optional, since you are allowed to do this through Terraform the first time :))

Notable mentions include:

kubent: kube no trouble
opa policy-bound pipelines: hard, but very rewarding if done correctly

Preview Environments

Preview environments are environments that are automatically spun up.

In our case, we do this through Pull Requests.

On every pull request against a branch, a label kicks in, and ArgoCD (through ApplicationSets... angry) creates a new application and automatically syncs it. After that, through the use of ASO (Azure Service Operator) we provision a database.

Upon a pull request close, the application is automatically deleted, and with it, the resource managed through ASO, too.

Two other possible ways to achieve this are:

cnpg operator: it connects to a live database, dumps the database, and provisions it. Authentication happens through external secrets.
PVC + Snapshot

With preview (or dynamic as I like to call them) environments, developers have instant access to a staging environment, automatically provisioned for them, with their pull request. Very powerful.

Subtleties

Here are important subtleties many engineering teams, even though good, often fail to deliver on:

ArgoCD configuration: should happen programmatically, this is a common chicken-egg problem, which I personally solve by having the entire ArgoCD helm locally, pointing to it from inside the ArgoCD, which I provision through CI/CD manually the first time. This applies to the master CI/CD. In a child-instances configuration, all child ArgoCDs are managed through the master. Through GIT. Then the rest, like projects, groups, roles, policies... all should happen in your k8s repository.
Cluster Apps: I use the app-of-apps pattern, which I achieve with either Kustomize or Jsonnet. Kustomize is often easier to maintain. I do not use ApplicationSets because it is harder to apply my name suffix & variability philosophies there.
State: Always remote
IAM: Always in either Terraform or the k8s (for ArgoCD) repos. Always add to groups. Do not apply one-time assignments.
Secrets: I always strive to use OIDC. Expired secrets suck and OIDC is an incredible technology that solves this.

Another important thing is expertise. I personally have been shooting educational videos of how our entire platform works, every couple of months. I have also made the following docs:

required-tools.md: kubectx/kubens, az cli, tf cli, etc. etc.
onboarding.md: here I explain the concept of variability, the variables for our organization, and link to a short video where I demo the concept by finding a resource in both Azure and in the Terraform repository, and make a small application configuration through the k8s repo, and an application repo.

Those have proven to be very valuable as we onboarded 10+ people through them.

So.. the conclusion is, "find someone who is willing to teach".

Closing

Thank you for reading and I hope you learned something, got inspired, or fired up. Either way, I enjoyed writing this.

Denis

out 🫡