DEV Community: Ivan Dolgov

A release plan is a return plan

Ivan Dolgov — Fri, 23 Oct 2026 18:05:15 +0000

Every team I have worked with promotes changes through environments, and every team has shipped something that passed all of them and still went wrong in production. That is not a failure of discipline. It is what environments are: a sample of reality, taken under conditions you chose.

00 / What staging actually proves

Staging proves that a change does not fail in the ways you thought to look for. It is a real and useful thing to prove, and it is smaller than it feels, because staging is missing the parts of production that cause the interesting failures:

the data volume, and the one tenant whose data is shaped unlike everyone else's;
concurrency — real users doing things at the same time, with caches already warm;
the integrations, in the state they are actually in today;
the clients, in the versions people are actually running.

So the question that decides how a release goes is not "did it pass staging?" but "if it is wrong in production, what do we do in the next ten minutes?" On a live product that question has to have an answer before the deploy, because afterwards you will be answering it under time pressure, in
front of customers, at the hour you chose to release.

01 / Three classes of change, and only one is cheap to undo

Reversible by redeploying. Code only, no schema change, no data written in a new shape. Going back is deploying the previous artefact — which means keeping it, and having gone back at least once on purpose, so the path is known rather than theoretical.

Reversible with work. A schema or data change made additively: add the new field, write both, backfill, read the new one, and only later stop writing the old. Each step is individually reversible, which is the whole point of doing it in that order. Write down which step is the point
of no return, and treat crossing it as a separate decision with its own date.

Not reversible. A destructive migration, an email that went out, a call made to someone else's API, a mobile build already on people's phones. For these, "roll back" is not a plan and never was. What works instead is making the change invisible until it is verified: behind a flag, in a shadow table, in a job that writes somewhere nobody reads yet.

Most incidents I have seen in this area come from a change in the second class being treated as if it were in the first.

02 / Blast radius, before the change rather than during it

On a packaging-industry SaaS I worked on, the frontend, the backend and the mobile app lived in separate repositories, each with its own deployment pipeline and its own environments. That is a good structure: each part ships on its own schedule. It also hides something, which I learned while
verifying rather than while planning — they all meet at one data layer, so a change to the shared schema reaches further than the repository you have open.

What that means in practice: before a change, answer "what else reads this?" by looking, not by remembering. A field you are removing from a GraphQL schema is being selected by clients you did not deploy. The web client can be fixed in an afternoon. The mobile one cannot, because the version in someone's pocket is the version they have until they update, and the slowest updater sets how long your backend keeps serving the old shape.

The architectural rule that falls out of this is boring and worth following: the shared layer changes additively, and the removal happens one release later than you want it to.

03 / Night releases buy a window, not safety

If your customers are concentrated in one timezone, releasing outside their working hours is a cheap and real advantage, and it is not available to everyone. But it is worth being precise about what it buys: not a lower chance of being wrong, only time to notice and act before anyone is
affected.

That time is only useful if someone is using it. Which means deciding, before the release, what you will look at in the first fifteen minutes — the one query, the one dashboard, the one screen that tells you the migration did what it was supposed to — and who is making the call.

04 / The paragraph that does the work

Not a process document. Five lines per change, written while you still have the whole picture in your head:

What changes.
What else touches it.
How we check, in production, that it worked.
How we go back.
When we stop trying to fix forward and go back instead.

The fifth line is the one that matters and the one usually missing. At two in the morning, with a customer's data in an unknown state, nobody decides well how long to keep trying. Deciding it in advance, when it costs nothing, is the whole trick.

One concrete version of line 3, from a Node.js 16 to 20 migration on that same product: the check was not "the tests pass". It was a defined workflow — set up a tenant, then do the ordinary things a customer does — run by hand against the deployed system, because the dependency and runtime problems that migration produced were not the kind that unit tests notice.

05 / What I would settle before the next one

Which class the change is in, in writing, and where its point of no return is.
What the previous artefact is and whether anyone has ever deployed it on purpose.
The oldest client version still in use, and whether the change is safe for it.
The one check that proves the change worked, and the one that proves it did not.
Who is awake, and what they are allowed to decide alone.

The longer version of the engagement these come from — what was diagnosed, what was migrated and how it shipped — is written up as a case study.

When per-user identity pricing stops working

Ivan Dolgov — Fri, 23 Oct 2026 18:05:09 +0000

Identity is the component that is cheapest to adopt and most expensive to leave. The decision is usually made in an afternoon, early, by whoever is setting up the first environment — which is exactly the moment when the per-user option costs nothing, because there are no users.

00 / Two different shapes of cost

A managed identity provider bills per monthly active user. A self-hosted one bills per instance, per database, and per engineer. One of those lines slopes with your success. The other is nearly flat, and then steps.

For most B2B products the sloping line never gets steep: a few thousand seats is a rounding error next to the cost of running an auth server yourself, and the decision is not close. For a consumer product planned to be large, the lines cross. The only questions are where, and whether being wrong is survivable.

01 / Price it on the day, not from memory

Rates change, and so do the models. What is worth knowing in advance is the shape of the traps. Amazon Cognito's pricing page, read on 2026-10-23:

three feature tiers, and a new user pool defaults to the middle one — Essentials, at $0.015 per MAU above the free tier;
the 10,000 MAU monthly free tier applies on Lite and Essentials. The Plus tier has no free tier at all;
federated SAML/OIDC users have a separate allowance of 50 MAU per month, whatever tier the pool is on;
user pools created on or before 22 November 2024 keep an older, larger allowance of 50,000 MAU — so the price depends not only on which tier you are on but on when the resource was created.

Auth0 prices its B2C plans by monthly active users too, with a free tier in the tens of thousands of MAU at the time of writing. The number that matters for a product planned at scale is the one
not on the page: the top plan is quoted, not listed. If the plan is to grow past the published tiers, the cost of the managed option above that point is not something you can look up — it is something you will negotiate, later, from a position where moving is expensive.

That asymmetry, not the per-MAU rate, is the thing to notice.

02 / What is actually in the self-hosted column

An estimate that sets a per-MAU rate against "an instance and a database" is not an estimate. It is an argument with the inconvenient half left out. The honest column contains:

Availability. Sign-in is on the critical path of everything, including sessions that expire mid-use. An identity server without a failover plan is a single point of failure introduced into a system that may not otherwise have one.
Upgrades on somebody else's schedule. A CVE in an auth server is not a backlog item.
Key rotation, session invalidation, and a tested procedure for "revoke everything, now".
Backups of a store holding password hashes — restored at least once, by someone who wrote down how long it took.
Account recovery and email deliverability, which is the unglamorous half of consumer identity and the half users actually meet.
Abuse: bot signups, credential stuffing, rate limits per account and per address.
The engineer who knows how it works — and the second one, for when the first is away.

Priced at a loaded hourly rate, that column is rarely small. It is also mostly fixed: the same work serves ten thousand users and ten million. That is precisely why the two lines cross at all.

03 / The lock-in is not the protocol

"We can move later, it is all OIDC" is half true. The protocol is portable. The user store is not.

Subject identifiers change. Anything that stored sub as the user's identity has to be remapped, including rows in your own database.
Password hashes may not be exportable. Where a provider will not hand them over, migration means every user resets their password. That is a product event, not a deployment.
MFA enrolments, social-account links and refresh tokens do not transfer.
Anything that trusts the issuer is pinned to it: gateway authorizers, mobile builds already in the field, partner integrations.

So the question is not only which option is cheaper but which direction is cheaper to be wrong in.
Leaving a self-hosted provider is a migration you can plan, because you hold the hashes. Leaving a managed one may not be. That asymmetry belongs in the decision and is usually left out of it.

04 / Sometimes the argument is not money

Where sign-in belongs is an architecture question before it is a pricing question. If one product signs users in, identity can live inside it. If several products are meant to share one account, identity is a platform component: it belongs outside any single application, on its own hostname,
with its own lifecycle. Moving it there later means changing every redirect URI and every token issuer already shipped.

On the consumer AI chat product behind this case study, that is where it ran: Keycloak on its own domain, intended as the single sign-in entry point for a planned family of AI-interactive applications sharing one database. Cognito and Auth0 both price by user, the product was planned for a large user base, and the per-user line was the one that would have grown with it. During that engagement the wider platform remained design intent rather than a delivered system — which is part of the lesson too. The platform argument is a bet on a roadmap, and roadmaps move.

05 / What to settle before committing

Three user-count scenarios — launch, plan, and the number that would count as success — priced on both sides, on the day, not from a blog post.
Engineer-hours in the self-hosted column, at a loaded rate, including the on-call half.
The export path for users, hashes and MFA enrolments, written down before launch and confirmed with the provider rather than assumed.
What happens while identity is down, and whether anyone has tried it.
Who upgrades the thing, how often, and in what window.

Either answer is defensible. "It is cheaper" without the second column, and "we can migrate later" without the export path, are not.

Why your DynamoDB app got slow as data grew

Ivan Dolgov — Fri, 23 Oct 2026 17:59:25 +0000

The complaint usually arrives in the same shape. The product was fine. Nobody changed the architecture. And now a list screen takes nine seconds, the mobile app times out on export, and it is getting worse every month — for everyone, not for one customer on one screen.

Nothing broke. That is the part worth sitting with, because it decides where you look.

00 / Growth does not break a data model. It reveals one.

A data model encodes the assumptions of the weeks it was written in: how many records there would be, how many tenants, which screens would read what. Those assumptions are rarely written down, and they are never wrong on day one — there is not enough data for them to be wrong yet.

What changes is the volume, and volume turns a constant-time mistake into a linear one. A full table read costs nothing at a thousand items. At a hundred thousand it is the product.

So the question is not "what regressed?" Nothing regressed. The question is "which assumption just stopped holding, and where is it encoded?"

01 / Read the evidence, not the documentation

On the engagement where I did this at length — a production SaaS with web and mobile clients — the documentation of the data model was out of date, and the team did not share an accurate picture of how it actually behaved. That is normal. Documentation describes the model somebody intended; the
code and the running system describe the one you have.

Three sources, in this order:

The code. Every read path. Not what the ORM or the resolver is named — what it actually issues.
The queries. Profiled, with real parameters, against real volumes. A query that is fast against your seed data tells you nothing.
CloudWatch. Request behaviour over time, so you can tell a slow operation from a slow pattern.

Where the code was hard to follow, I drew the data flow out as a diagram before changing anything. That is not ceremony. If you cannot draw it, you do not know what a change will touch, and on a live product that is the whole risk.

02 / The five patterns

These are the ones I have found most often together. Each is individually survivable. They compound.

Scan where a Query against a key would do

Scan reads the entire table and then filters. Query goes to a key and reads what it needs. The filter in a Scan runs after the read, so a filtered Scan that returns three items has still read everything you are paying for, in time and in capacity.

How to confirm it: search the code for Scan and, for each one, write down which key you would query by instead. If you cannot name one, you have found a key design problem, not a query problem.

Missing global secondary indexes

A GSI exists so that a common access pattern has a path that is not the primary key. Without it, the only ways to answer "all orders for this customer, newest first" are a Scan or a read of more items than you need.

How to confirm it: list the questions your screens actually ask. Each one should map to a key or an index. The ones that map to neither are your list.

Keys that do not match how the data is read

This is the one that is actually a redesign, and it is why the first two keep coming back. A partition key chosen because it was the obvious identifier, rather than because it is what reads group by, forces every reader to work around it forever.

How to confirm it: for each access pattern, ask whether the items you want are adjacent. If the answer is "they are scattered and we filter", the key is wrong.

N+1 in resolvers

In AppSync — and in any GraphQL layer — a list field resolves once, and then a nested field resolves once per item. One request for a list of 50 becomes 51 requests. The code looks innocent, because
each resolver is correct in isolation.

How to confirm it: count the requests for one list screen, not the time. The number should not grow with the size of the list.

Full datasets to the client, no pagination

The screen shows twenty rows; the request returns everything the table has. This is the easiest to find and the easiest to fix, which is exactly why it survives for years: it never looks like the problem, because it is not slow until it is.

How to confirm it: look at response sizes, not durations.

03 / Changing it on a product that is live

Diagnosis is the smaller half. The data belongs to customers who are using it right now.

What I would do again:

Ship in milestones, each one verified step by step on staging while the team reviews the code in parallel. Not one large change at the end.
Release outside usage hours when the customers are concentrated in one timezone. This is a cheap advantage and it is not available to everyone.
Update the data-model documentation when you are done. You just spent weeks reconstructing it. Leaving it unwritten guarantees the next person repeats that cost.

What I would do differently:

Plan the way back for every change, before making it. Promoting through dev, staging and production reduces risk; it does not guarantee that a production release succeeds. A data migration is exactly where that gap hurts, and "we will roll forward" is not a plan.
Map the blast radius first. With separately deployed clients sharing one data layer, a change to the shared schema reaches further than the repository you are editing. I learned that while verifying, not while planning.

04 / Three things I would not do

A full single-table redesign as the first move. It is often the right end state and almost never the right first step: it is the largest possible change to the thing you least understand, made while the symptom is still unexplained.

Caching in front of a Scan. This makes the graph look better and leaves the cost exactly where it was, now with a staleness problem on top. Cache after the access pattern is right, not instead.

Chasing a number before defining the measurement. Decide what you are measuring, on which operation, at what data volume — before you touch anything. Otherwise you cannot tell improvement from noise, and you will not be able to answer the only question your client will ask.

05 / What "better" looked like

I am not going to give you a percentage. No measurement of the before and after on that engagement is approved for publication, and a number I cannot source is worth less than the method.

What is defensible: after the work, search timeouts decreased and customer complaints decreased substantially. The product kept running throughout.

If you want the longer version of this specific engagement — the constraints, what was migrated, and how it shipped — it is written up as a case study.

Choosing a stack by proof of concept

Ivan Dolgov — Fri, 23 Oct 2026 17:59:24 +0000

A technology choice defended with "it is what we know" is sometimes right and never checkable. A proof of concept makes it checkable — but only when it is run as a decision procedure, and not as an early version of the product.

00 / A proof of concept is a question with a deadline

Three things separate it from a prototype: a question whose answer is allowed to be no, a criterion agreed before anyone writes code, and a date on which it ends. Drop any one of them and the PoC quietly becomes the first version of the system — the demo ships, nobody re-reads the question, and the stack has been chosen by sunk cost.

The question has to be able to fail. "Can we build this in X?" cannot: the answer is always yes, given enough time. "What does X cost us on the part we are actually worried about, in two days?" can.

01 / Make the comparisons comparable

Same task, same timebox, same reviewer, and the criteria with their weights written down before the first commit. The task should be the thing you are afraid of — the query that worries you, the concurrency, the integration whose documentation looks thin — and not an endpoint every candidate
handles identically.

Three ways this goes wrong, all of them common:

each candidate is built by the person who already prefers it;
the candidates get different amounts of time, usually in favour of whoever started first;
the criteria are written after the results, and fit the winner remarkably well.

02 / Criteria that survive the meeting

On the consumer product behind this case study, the backend language was chosen after several proofs of concept rather than by preference, against four criteria: scalability, maintainability, speed, and how well the codebase would work with AI-assisted development. Go came out of that balance, and other parts of the stack were decided the
same way.

Three of those four are ordinary, and each is worth turning into a question with an answer rather than an adjective:

Scalability — what happens at ten and a hundred times the load you expect, and what does it cost there? Measure it; do not reason from the vendor's page.
Maintainability — how long does it take someone who did not write the code to make a small change safely? That is a number, and the PoC is the cheapest time to take it, while the codebase is still small enough for a newcomer to read in an hour.
Speed — of the system and of the team. A runtime that is fast and that nobody on the team writes fluently is slow.

03 / The criterion almost nobody tests for

The fourth criterion is the one worth being explicit about, because teams now take it on faith in one direction or the other: how well a codebase in this stack will hold up when much of it is written with an assistant.

It is testable, and the thing to measure is the review, not the output. Give the same feature to the assistant in each candidate and look at:

how much of what came back was correct, and how much was plausible and wrong;
what the compiler or the type system rejected before a human had to — a stack where wrong code fails to build is cheaper than one where it runs and fails later;
whether the standard library covered what the assistant would otherwise have pulled a dependency for, because dependencies chosen by autocomplete are a maintenance decision made by nobody;
whether formatting is settled by a tool, so that diffs are about logic instead of style;
how long the result took to review, which is the number that actually scales with the team.

A large public corpus helps. It is not the whole story: the wider a language's surface, the more there is to be confidently wrong about, and the more a reviewer has to hold in their head to catch it.

04 / Letting the answer be no

A PoC can also answer "do we keep what is already here?". On that engagement an earlier codebase existed, written by one developer more than a year earlier, and for the scope being started it offered little foundation. That is a legitimate result, and it needs the same written question and the same deadline as any other — otherwise the discussion drifts towards the previous author instead of the code.

When a question has been answered, the PoC should be deleted, or promoted deliberately with a plan for the parts that were never finished. Kept "just as a reference", it gets copied.

05 / The half a PoC does not cover

The language decision on that project anticipated AI-assisted development. The team's workflow for it was not standardised from the start, and output often did not meet shared expectations and had to be reworked, which lowered quality overall. It is the thing I would do differently, and it is
not a tooling problem.

A technology evaluation answers which tool. It does not answer how we work: what a task looks like before it is handed to an assistant, what review covers that it did not before, what gets written by hand, and what nobody merges without reading twice. That second question deserves the
same deliberate treatment as the first, and usually gets none — because it has no artefact to show at the end of the week.

Choose with evidence, then write down how the chosen thing is used. The first half is the easy one.