DEV Community: vivek Itp

Error messages are part of the platform

vivek Itp — Wed, 09 Sep 2026 18:47:15 +0000

Nobody reads a platform's documentation on a good day. On a good day the pipeline is green, the deploy went out, and the developer never thinks about the platform at all. The platform only gets read when something breaks, and what gets read is the error message.

That makes the error message the most-read text a platform team ever writes. It is also, almost always, the text nobody on the platform team wrote on purpose. It is whatever the underlying tool printed, passed through untouched, with a stack trace attached.

I spent a long time treating errors as something that happened to the platform rather than something the platform produced. This post is what changed my mind, and what I do differently now.

The gap between what it said and what it meant

Here are the kinds of messages developers actually see, next to what the platform team knows they mean. The gap between the two columns is where support requests come from.

What the developer saw	What it actually meant
`ImagePullBackOff`	The image tag you deployed was never pushed. Your build job probably failed.
`exit code 137`	The runner ran out of memory. Not your code.
`Error: forbidden: User cannot list resource "pods"`	You are in the wrong namespace, or you were never granted access to this one.
`KeyError: 'DATABASE_URL'`	The secret for this environment was never created.
`x509: certificate signed by unknown authority`	The internal CA is not installed in your base image.
`Error: no matches for kind "Ingress" in version "extensions/v1beta1"`	The template you copied is three years old.

Every row in the right column is something the platform team could have printed instead. None of them did, because the message came from a tool underneath the platform and nobody caught it on the way up.

1. An error has three jobs

A useful error message answers three questions, in this order:

What broke? In one plain sentence, from the developer's point of view.
Why? The most likely cause, stated as a fact if it is known and as a guess if it is not.
What do I do now? A command, a link, or a name. Something the developer can act on in the next minute.

If a message does only the first job, the developer knows they are stuck. If it does all three, the developer is usually unstuck before they think of asking anyone.

What this looked like for me

The first error I rewrote was the image pull failure. The raw version was ImagePullBackOff with a pod name. Developers would see it, check that their code compiled, re-run the deploy, see it again, and then ask in the channel.

The rewritten version, printed by the deploy step, read something like: "Deploy failed: image tag abc1234 does not exist in the registry. This usually means the build job for this commit failed or has not finished. Check the build job here: link."

The first line was the same information. The second and third lines were the difference between a support request and a self-fix. Questions about that error mostly stopped after the change.

2. Write for the person who did not build the system

Most error messages are written, implicitly, for the person who wrote the code that raised them. They use internal names, assume knowledge of the architecture, and skip context that the author never needed because they already had it in their head.

A developer on a product team does not have that context. They do not know that "reconciler" means the deploy controller, that "upstream" means the service behind the proxy, or that error code 4012 is the one about quotas.

The test I use: would a competent engineer who joined last week and has never read our platform code understand this message? If not, it is not finished.

The message with an internal name in it

We had a validation step that rejected service configs with a message naming the internal component that did the checking. Something like "rejected by policy-gate: rule R7 failed." Everyone on the platform team knew what R7 was. Nobody else did, and there was nothing in the message to search for.

The fix was to make the rule print its own explanation: "Service config rejected: memory must be at most 4Gi for services without a capacity exception. Yours is 8Gi. To request an exception, see: link." Same rule, same check. The developer just got to read the reason instead of the rule number.

3. Put the fix in the message

The most valuable thing an error can contain is the next command to run. Not a description of the fix. The fix.

A lint failure should print the command that auto-fixes it.
A missing secret should print the command or link that creates it.
A permissions error should name the group that grants the permission and where to request it.
A deprecated config field should print the new field name and the one-line change.

This is also the cheapest place to put documentation, because it is the only documentation that is guaranteed to be read at the moment it is needed.

# Before
Error: validation failed for field 'healthcheck'

# After
Error: 'healthcheck' must be a path starting with '/', got 'healthz'.
Fix:   change it to '/healthz' in service.yaml.
Docs:  https://internal/docs/service-config#healthcheck

The link that replaced a wiki page

For a while the most-linked page in our support channel was a wiki article explaining how to request access to a new environment. Someone would hit a permissions error, ask, and get the link. Dozens of times.

We put the link in the error message. The wiki page did not change. The number of people who needed to be told about it dropped close to zero, because the tool told them at the exact moment they needed it.

4. Lead with the plain sentence, keep the raw detail

The instinct when wrapping an error is to hide the original. That is a mistake. The original error is often the only thing that helps when the plain-language guess is wrong, and someone on the platform team will eventually need it.

The structure that works: plain sentence first, cause and action next, raw detail last and clearly labelled.

The developer reads the top. The platform engineer, if it gets that far, reads the bottom. Nobody has to scroll through a stack trace to find out that a tag was missing.

The wrapper that hid too much

My first attempt at friendlier errors over-corrected. The deploy tool caught every failure and printed a short summary, and dropped the original output entirely. It was clean until the summary was wrong. Then the developer had a confident sentence pointing at the wrong cause and no way to see what had actually happened.

We put the raw output back, below a separator, under a heading that said it was for debugging. The summaries stayed. The dead ends went away.

5. Treat the most common errors as bugs

Every platform has a small set of errors that account for most of the confusion. You can find them without any tooling: read a month of the support channel and count. The same five or six messages will come up over and over.

Each of those is a bug in the platform, even if the underlying tool is behaving correctly. The bug is that the platform let a confusing message reach a developer. Put them on the backlog, in priority order by how often they appear, and fix them the way you would fix any other bug.

The list on the wall

At one point I kept a short list of the errors that generated the most repeat questions, ordered by count. It had maybe eight entries. We worked down it during quiet weeks, one rewritten message at a time.

It was some of the highest-return work the platform team did that year, and almost none of it involved changing what the platform actually did. It only changed what it said.

6. Test error messages like features

If a message matters, it deserves a test. Not a test that the error is raised, which most codebases already have, but a test that the message says what it should: names the field, includes the fix, links to the right place.

This sounds excessive until the first time someone refactors the validation code and the helpful message quietly reverts to a generic one. Nobody notices, because errors are not on the happy path, until the support channel fills up again.

Takeaways

The error message is the platform's real interface. It is read more than any docs page, at the exact moment it matters.
Three jobs: what broke, why, what to do next. A message that does one is not finished.
Write for someone who joined last week and has never seen the platform code.
Put the fix in the message. A command or a link beats a description every time.
Plain sentence on top, raw error on the bottom. Never hide the original.
Count the repeat errors and fix them as bugs. It is cheap, and it compounds.

What I still have not worked out is errors from tools I do not own. When a managed cloud service or a third-party CLI returns something cryptic, the platform can wrap it, but the wrapping is a guess, and the guess goes stale every time the vendor changes their wording.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/09/error-messages-are-part-of-the-platform/.

Self-service on a slide vs self-service in practice

vivek Itp — Wed, 09 Sep 2026 18:42:05 +0000

Every internal platform I have seen described in a slide deck was self-service. Developers would "spin up a new service in minutes" with "no tickets" and "no waiting on the platform team". I have never seen one of those slides that was accurate on the day it was presented.

That is not because the people writing the slides were lying. It is because self-service is a property of the whole path a developer walks, and the slide only describes the part the platform team built. The parts they did not build, the access request, the DNS entry, the secret that has to be created by someone else, the approval that lives in a different tool, are invisible from the platform side and completely visible from the developer's.

This post is about measuring that gap honestly, and about what actually closes it.

The only definition that matters

Self-service means a developer with a normal amount of access can go from "I need a new service" to "it is running in production and I can see it" without asking a human for anything.

Not "without asking the platform team". Without asking anyone. The moment there is a ticket, a Slack message, or a form that someone else has to act on, the path is not self-service, no matter how good the tooling is on either side of that gap.

That definition is strict on purpose. A path that is self-service except for one step is not ninety percent self-service. It is blocked at that step, and every developer will wait there.

What the path usually looks like

Here is the shape of the path I have seen most often, regardless of company or tooling. The platform team owns the middle. The friction lives at the edges.

Step	Who does it	Self-service?
Create the repository from a template	Developer	Yes
Get the pipeline running	Developer, using the shared template	Yes
Get a namespace or environment	Platform team, via ticket	No
Get a database or queue	Another team, via a different ticket	No
Get secrets into the environment	Security or platform, via request	No
Get a DNS entry and certificate	Network team, via email	No
Get production deploy permission	Manager approval, via form	No
First production deploy	Developer	Yes

The three "yes" rows are what the slide describes. The five "no" rows are what the developer remembers.

What this looked like for me

The first time I actually walked the path myself, start to finish, as if I were a developer on a product team, it took the better part of a week. Not because any single step was slow. Because each step was a request to a different person, and each person had a queue.

The platform tooling in the middle was genuinely good. Repository template, pipeline, deploy, all worked first time. I still spent most of the week waiting. Nobody on the platform team had ever counted the waiting, because none of it happened in our tools.

1. Count tickets, not features

The most useful number a platform team can track is not how many features the platform has. It is how many times a developer has to ask another human for something on the way to a first production deploy.

I call it the ticket count, even though half of those requests are never actual tickets. A Slack message asking for a namespace is a ticket. An email to the network team is a ticket. An approval form is a ticket. If a human has to act before the developer can continue, it counts.

Then track the second number: how long each of those requests waits. Not how long it takes to fulfil, which is usually minutes. How long it waits in someone's queue before anyone looks at it, which is usually days.

The number that changed the conversation

When I first put the ticket count in front of the people who owned the platform roadmap, it was somewhere around six human touch points for a new service, with a median wait of a day or two each. That was the first time the roadmap conversation shifted from "what feature should we add" to "which of these requests can we make disappear".

It also surfaced something uncomfortable: the platform team was not the bottleneck. We were the fastest queue in the chain. The slow ones belonged to teams that had never been told they were part of the developer experience at all.

2. Most tickets exist because of a default nobody set

When you look at why each request exists, a pattern shows up. Most of them are not there for safety. They are there because nobody decided what the default should be, so a human decides it every time.

A namespace request exists because nobody decided that every repository gets a namespace automatically.
A resource quota request exists because nobody decided what a reasonable starting quota is.
A database request exists because nobody decided that a service can provision its own small database within limits.
A production access request exists because nobody decided that the team that owns a service can deploy it.

Each of these is a policy question that a human answers by hand, dozens of times, usually the same way. Writing the answer down once and letting the tooling apply it is the whole job.

The request that was always approved

One of the requests in our chain was for a new namespace with a default quota. When I looked at the history, every single one had been approved, almost always with the same quota. The approval step existed because at some point someone had worried about cluster capacity, and the worry had turned into a permanent human gate.

We replaced it with a rule: every repository created from the template gets a namespace and a standard quota at creation time, with a documented way to ask for more. The approval disappeared, the ticket disappeared, and the cluster did not run out of capacity. The worry had been real. The gate had never been the right answer to it.

3. Pre-approve the safe path, and only the safe path

The objection to removing gates is always the same: what stops someone doing something dangerous? The answer is not a human. It is a narrow path that is safe by construction, with the gates kept only for leaving that path.

In practice that means:

The template produces a service that is safe to deploy as-is: no public exposure, conservative resources, no elevated permissions.
Everything on that path is pre-approved. Nobody reviews it, because the review already happened when the template was written.
Anything off the path, a public endpoint, a privileged container, a larger quota, still needs a human. But that human is now reviewing exceptions, not routine.

This is the part that makes security and platform teams comfortable with removing gates. They are not removing review. They are moving it from every request to the template, where it happens once and applies everywhere.

Where the pre-approval broke down

The thing I got wrong the first time was making the safe path too narrow. The template covered a stateless HTTP service and nothing else. Anything with a queue consumer, a scheduled job, or a database needed the exception route, which meant most real services went through the exception route, which meant we had rebuilt the ticket queue with extra steps.

The fix was widening the path until it covered what most teams actually built, and treating each new exception request as a signal that the path was still too narrow. The exception route is supposed to be rare. If it is not, the template is wrong, not the developers.

4. Own the whole path, or at least measure it

The hardest part is that most of the tickets belong to teams the platform team does not control. Networking, security, database operations, a change board. Telling those teams to remove their gates does not work. Showing them where they sit in the developer's week sometimes does.

What I have seen work:

Publish the path as a single diagram, with every human touch point marked and the median wait for each. Put it somewhere visible.
Ask each owning team one question: what would need to be true for this step to be automatic? Usually the answer is a policy nobody has written down.
Offer to build the automation for them. Most teams are not against self-service. They do not have time to build it.

Each green node was, at one point, a ticket to a different team. None of them stopped being someone's responsibility. They stopped needing a human in the loop for the routine case.

The team that was never asked

The slowest step on our path was a DNS and certificate request that went to a networking team by email. From the platform side it looked like an immovable external dependency. When I finally talked to that team, they had no idea their queue was on the critical path for every new service. They had automation for internal zones already. It just was not connected to anything developers could reach.

Connecting it took a couple of weeks and one conversation. The step went from a multi-day wait to a couple of minutes, and the platform team did not build most of it.

5. Measure it the way a developer feels it

If the goal is honest self-service, the metric has to be something a developer would recognise. The ones I keep coming back to:

Time from repository creation to first production deploy. Wall clock, not effort. This is the number that captures the waiting.
Human touch points on that path. The ticket count from above. Target: zero for the standard path.
Exception rate. What fraction of new services need the off-path route. If it is high, the path is too narrow.
Support requests per new service. How many times the developer had to ask a question that the tooling should have answered.

None of these need a new tool. They need someone to walk the path occasionally, as a developer would, and write down what happened.

Takeaways

Self-service means no human in the loop, not just no platform engineer in the loop. One gate blocks the whole path.
Count the tickets, including the ones that are not tickets. Slack messages, emails, and approval forms all count.
Most gates are undecided defaults. Decide the default once, encode it, and the gate disappears.
Pre-approve the safe path and keep review for exceptions. Then widen the path until exceptions are rare.
The slow steps usually belong to someone else. Show them the path. Offer to build the automation.

What I still do not have a good answer for is the last step on most paths: the production approval that exists for compliance reasons rather than technical ones. I have made it faster and made it clearer, but I have not yet made it go away, and I am not sure it should.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/07/self-service-on-a-slide-vs-in-practice/.

Why this blog exists

vivek Itp — Wed, 09 Sep 2026 18:42:01 +0000

After more than seven years in DevOps and platform engineering, I have a backlog of small lessons that never made it into a document. This blog is where they go.

The problem with "small" problems

Most of the work I have done was not about big architecture decisions. It was about the things in between: a pipeline that takes twelve minutes when it should take four, a Kubernetes error that means nothing to the developer reading it, a deploy process that everyone has quietly learned to work around.

None of these are outages. Nobody writes a postmortem for them. But they are what a developer's day actually feels like, and they decide whether a team ships confidently or nervously.

What I plan to write about

CI/CD – what makes a pipeline something developers trust, and what quietly breaks that trust.
Kubernetes – the parts that are hard to explain to people who just want to deploy their service.
Platform engineering – what "self-service" really requires, and where internal platforms fall short of the slide deck.
Developer experience in general – the friction that is too small to name and too expensive to ignore.

What to expect

Short posts, written from experience, with the failures left in. I will update posts when I learn something new, and the last-updated date will show that.

If you disagree with something, or have seen the same problem solved better, I would genuinely like to hear about it. The GitHub repository for this site accepts issues.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/05/why-this-blog-exists/.

Kubernetes for people who did not ask for Kubernetes

vivek Itp — Wed, 09 Sep 2026 18:33:24 +0000

Nobody on an application team wakes up wanting to learn Kubernetes. They want to ship a service, see it running, and fix it when it breaks. Kubernetes was a decision the platform team made, usually for good reasons, and then handed to people who never asked for it.

I have been on the side that made that decision. I still think it was right. But I have also watched what it costs a developer the first time they see CrashLoopBackOff in a terminal and have no idea what it means, who to ask, or whether it is their fault. That moment is where a lot of developer experience quietly goes to die.

This post is about closing the gap between what developers need from a cluster and what the cluster asks of them in return.

What developers asked for, and what they got

They asked for	What the cluster asked of them
Deploy my service	Write a Deployment, a Service, an Ingress, maybe a ConfigMap and a Secret
See my logs	Learn `kubectl`, find the right namespace, find the right pod, pick the right container
Roll back	Know that rollouts have history, know the command, have permission to run it
Know if it is healthy	Understand readiness versus liveness probes and what happens when each fails
Give it enough memory	Understand requests versus limits, and what `OOMKilled` means at 2 AM

None of the right-hand column is unreasonable for a platform engineer. All of it is unreasonable to expect of someone whose job is the left-hand column.

1. Start from the three things developers actually do

Strip away the tooling and a developer interacts with the runtime in three ways: deploy a version, look at what it is doing, and go back to the previous version when something is wrong. Everything else is either rare or someone else's job.

That means the first question about any Kubernetes platform is not "which ingress controller" or "which service mesh". It is: how many steps does it take a developer to do each of those three things, and how many of those steps require knowing a Kubernetes concept?

What this looked like for me

When I first mapped the deploy path for a typical service, I counted the Kubernetes concepts a developer had to touch to get from a merged pull request to a running pod they could see. It was somewhere around ten: namespace, deployment, service, ingress, image pull secret, config map, resource requests, probes, labels that had to match exactly, and the kubectl context to even look at any of it.

Most developers had learned exactly enough to copy the previous service's YAML and change the name. Which worked, until it did not, and then nobody could explain why.

2. Every concept you expose becomes a support ticket

Kubernetes leaks. Not in a technical sense, but in the sense that every abstraction it offers eventually shows up in a developer's terminal as an error they cannot decode.

Some of these leak more than others. In my experience the ones that generate the most confusion, in rough order:

Resource limits and OOMKilled. The pod dies, restarts, dies again. The application logs show nothing because the process was killed from outside.
Probes. A service that is fine but slow to start gets restarted forever because its liveness probe fires too early.
Labels and selectors. A Service pointing at nothing because one label has a typo. No error. Just no traffic.
Namespaces and contexts. "My pod is not there." It is there. They are looking in the wrong namespace.
Image pull errors. A tag that was never pushed, or a registry credential that expired.

The one that came up every week

The single most repeated question I dealt with was some version of "my pod keeps restarting and the logs are empty." Nine times out of ten it was OOMKilled. The developer had copied a memory limit from another service, their service needed more, and the kernel killed it before it could log anything.

The fix that finally reduced the questions was not documentation. It was making the deploy tooling print the pod's last termination reason next to the logs, in plain words: "This container was killed for exceeding its memory limit of 256Mi. Raise memory in your service config or investigate memory usage." The developer never needed to know what a limit was. They needed to know what to change.

3. Put a thin layer in front, not a whole new platform

The tempting answer to all of this is to build an internal platform that hides Kubernetes completely. I have seen that go wrong more often than it goes right. The layer becomes its own thing to learn, it lags behind the features teams need, and when it breaks the developer is now two abstractions away from the actual problem.

The approach I have had the most success with is thinner: a single, opinionated template that turns a small config file into all the Kubernetes objects a service needs.

# service.yaml, the only file a developer edits
name: orders-api
image: registry.internal/orders-api
port: 8080
replicas: 2
memory: 512Mi
cpu: 250m
healthcheck: /healthz
env:
  LOG_LEVEL: info

Behind that file is a Helm chart, or a Kustomize base, or a small script. It does not matter which. What matters is that the developer never writes a Deployment by hand and never sees a label selector, but the objects that get created are ordinary Kubernetes objects that anyone on the platform team can inspect with standard tools.

The template that replaced the copy-paste

We replaced per-service YAML with one shared chart and a values file of roughly ten keys. Migrating existing services was mostly deleting things. The chart set sensible defaults for everything the values file did not mention, so a brand new service needed four lines to deploy.

The trade-off was real. A handful of services needed something the chart did not support, like an extra sidecar or a specific affinity rule. We added an escape hatch that let a service ship raw Kubernetes manifests alongside the values file. Almost nobody used it. The few who did were the people who actually understood Kubernetes, which is exactly who should be writing raw manifests.

4. Do not hide the things they will need at 2 AM

Hiding complexity during the happy path is good. Hiding it during an incident is dangerous. A developer being paged about their own service needs to get to logs, recent events, and a rollback in under a minute, without a platform engineer in the loop.

What that requires, concretely:

A direct link to logs for the service, ideally posted on the pull request or in the deploy notification, not a kubectl incantation.
Events in plain language. "Pod restarted 4 times in 10 minutes, last reason: OOMKilled" beats a raw kubectl describe dump.
Read access by default. Developers should be able to look at anything about their own service without asking.
Rollback as a button or a single command. Not a runbook.

The incident that changed my mind about access

Early on, developers did not have kubectl access to production at all. Everything went through the platform team. The reasoning was safety. The effect was that every incident, however small, needed a platform engineer awake before anyone could even look at what was happening.

After one late-night incident that was a five-minute fix once someone with access finally looked, we gave every team read-only access to their own namespace and a rollback command they could run themselves. The number of incidents did not go up. The time to resolve them went down noticeably, and the platform team stopped being the bottleneck for problems that were never theirs.

5. Defaults are decisions, so make them good ones

Every field a developer does not set is a decision the platform made for them. In Kubernetes, most of the defaults are wrong for a production service: no resource requests, no probes, one replica, no pod disruption budget.

A good template treats defaults as the main product. A service that specifies nothing should still get:

Resource requests and limits that are conservative but not tiny.
A readiness probe on the health endpoint, with a start-up grace period long enough for a slow JVM or a database migration.
Two replicas, spread across nodes.
Rolling updates that keep the old version serving until the new one is ready.

The default that bit us

For a while our template set no resource requests unless the developer added them. Most did not, because they did not know they should. The result was a cluster where the scheduler had no idea how much anything needed, nodes got overcommitted, and the noisiest service on a node would starve its neighbours.

Setting a modest default request for every service, and making the template refuse to render without one, fixed most of the scheduling problems in one change. Nobody on an application team noticed, which was the point.

6. Rollback must be as easy as deploy, and faster

If deploying is a merge and rolling back is a ticket, developers will not roll back. They will try to fix forward under pressure, which is how a small incident becomes a large one.

Rollback should be the same mechanism as deploy, pointed at the previous version. In Kubernetes that can be as simple as kubectl rollout undo, but the developer should not need to know that. They need a command or a button in the same place they deploy from, and it needs to complete in seconds, not minutes.

The test I use: can a developer who joined last week roll back their own service during an incident without asking anyone? If the answer is no, the platform is not done.

Takeaways

Count the concepts. For deploy, logs, and rollback, count how many Kubernetes ideas a developer must understand. Aim for zero on the happy path.
Translate the errors. OOMKilled, failed probes, and empty selectors should surface as plain sentences that say what to change.
Thin template, ordinary objects. A small config file in, standard Kubernetes manifests out, with an escape hatch for the few who need it.
Never hide the incident path. Logs, events, and rollback must be one click away for the service owner, with no platform engineer required.
Good defaults are the product. Requests, probes, replicas, and rolling updates should be right when the developer says nothing.

The part I still have not solved is what happens when the thin layer needs to change. Every template update is a change to every service at once, and I do not yet have a way to roll that out that feels as safe as rolling out a single service.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/05/kubernetes-for-people-who-did-not-ask-for-it/.

CI/CD should be designed around the developer, not the pipeline

vivek Itp — Wed, 09 Sep 2026 18:33:19 +0000

Ask a developer which tool they touch most after their editor, and the honest answer is usually the CI pipeline. Every branch, every pull request, every deploy goes through it. Yet it is almost always designed by people who rarely have to wait on it.

That gap shows. Pipelines get optimised for what is easy to run, easy to secure, and easy for the platform team to maintain. Developer time spent waiting is invisible on every dashboard, so it never gets fixed.

I have been on the platform side of this for most of my career, and I noticed the gap the way most platform engineers do: through the support channel. The same three questions kept coming back. Why is it slow? Why is it red? Can someone re-run it? None of them were about the product. All of them were about the pipeline.

What developers actually want from CI

Strip away the tooling and developers want four things from a pipeline:

They want	Which means
Fast feedback	Know within minutes whether the change is fine
Clear failure	When it is red, know why without opening five logs
Trust	Red means broken. Green means safe to merge. No exceptions
No ceremony	Merging and deploying should not need a meeting, a ticket, or a favour

Everything below is about delivering those four things. Every pipeline feature that does not serve one of them is a cost, not a benefit.

1. Feedback in under ten minutes, or split the pipeline

A pull request check that takes thirty minutes trains developers to context-switch. They open another task, forget the PR, and come back an hour later. The real cost of a slow pipeline is not the thirty minutes. It is the hour of lost focus on both sides of the review.

The fix is rarely a faster machine. It is deciding what must run before merge and what can run after. Lint, unit tests, and a build belong in the PR check. Full integration suites, long security scans, and multi-platform builds can run on the main branch after merge, with a fast path to revert.

What this looked like for me

The slowest pipeline I inherited ran the full integration suite against a real database on every pull request. It took over half an hour on a good day, and a good day meant nobody else was pushing.

We moved integration tests to run on main after merge, and kept lint, unit tests and the build in the PR check. The PR check dropped to under ten minutes. The integration suite still ran on every merge, just not in the developer's way.

One thing did slip through: a database migration that passed unit tests broke the integration run on main. It was caught within minutes and reverted with one click. That was the whole point of the split. A failure on main costs one revert. A slow PR check costs every developer, every day.

2. A failure should tell you what broke in the first screen

The most expensive kind of pipeline failure is the one where the developer has to scroll through two thousand lines of log to find a single AssertionError. Every minute spent hunting for the failure is a minute the pipeline has actively wasted.

What helps, in order of impact:

Fail the job with a one-line summary at the top of the output, not buried at the bottom.
Annotate the PR with the failing test, file, and line. Most CI systems support this. Most pipelines never bother.
Name jobs after what they check, not after the tool. unit-tests beats pytest-run-3.
Link to the fix when the failure is a known category. A lint failure should point to the command that fixes it locally.

The error message that generated the most support requests

For a long stretch, the single most common message in our platform channel was some version of "my tests are failing with exit code 137 and I did not change anything."

Exit code 137 means the process was killed, almost always because the runner ran out of memory. The tests were fine. The test runner was being killed mid-suite because the job was sharing a small machine with a build step that ate all the RAM.

The real fix was giving test jobs their own memory limit. But the cheaper fix, which we shipped the same day, was a wrapper step that caught exit code 137 and printed one line: "Job was killed for exceeding memory. This is a runner problem, not your code. See link." Support requests for that error stopped almost entirely. The developers had never needed the fix. They had needed the explanation.

3. Flaky tests are a platform problem, not a developer problem

Nothing destroys trust in a pipeline faster than a test that fails one run in ten for no reason. Developers learn to click re-run on red without reading. Once that habit sets in, real failures get re-run too, and the pipeline stops meaning anything.

The uncomfortable part is that the platform team usually knows which tests are flaky and leaves them in, because removing a test feels like lowering the bar. It is the opposite. A test nobody believes is not raising the bar. It is noise with a green tick.

What I think works:

Track the re-run rate per job. It is the most honest developer-experience metric a pipeline has.
Give every flaky test an owner and a deadline. Fix it or move it out of the merge path until it is fixed.
Never retry silently. Automatic retries hide the problem and make the pipeline slower for everyone.

How flakiness actually showed up

We did not find our flaky tests by looking at test reports. We found them by counting re-runs. Roughly one in five pull requests was being re-run at least once before it went green, and when we looked at why, the same handful of tests appeared every time.

Most of them had the same root cause: they assumed something about the machine they ran on. A fixed port that was sometimes taken. A timing assertion that held on a laptop and failed on a busy shared runner. A test that depended on the order another test left the database in.

The ones we could fix, we fixed. The ones we could not fix quickly moved to a separate nightly job with a named owner and a date. They still ran. They just stopped blocking merges. The re-run rate fell to a level where a red PR check became worth reading again.

4. The difference between running code locally and in CI should be almost zero

Every gap between a developer's machine and the pipeline is a place where "it works locally" becomes a true statement and a useless one. The developer did nothing wrong. The pipeline did nothing wrong. They were simply running two different things.

Those gaps are rarely dramatic. A different Python version. A dependency resolved differently. An environment variable that exists on the laptop and not on the runner. A file that is present locally and ignored by git. Each one costs a developer an hour of confusion and a support message to the platform team.

The target should be that a developer can run one command locally and get the same result the pipeline will give. Not similar. The same.

The one that took an afternoon to find

A developer pushed a change that used newer language syntax. It ran fine on their laptop. The pipeline went red with a syntax error on a line that was obviously valid. They re-ran it. Red again. They pushed a no-op commit. Red again. Then they asked in the channel.

It took most of an afternoon to find the cause: the CI image was pinned to a runtime two minor versions older than the one on the laptop. Nobody had changed anything. The runtime version was defined in the pipeline config and nowhere in the repository, so local and CI had simply drifted apart over time without anyone noticing.

The fix was three lines. The runtime version went into a version file in the repo that both the developer tooling and the pipeline read. The dependency lockfile was committed and installed with an exact-match flag on both sides. The CI job was changed to run inside the same container image developers could run locally.

The lesson generalises: anything that defines the environment belongs in the repository, versioned, and read by both sides. Anything that lives only on a laptop or only in a CI setting is a bug waiting for its moment.

How to keep the gap closed:

One entry point in the repo, a Makefile, justfile, or task runner, that defines lint, test, and build. The pipeline calls those targets and nothing else.
Pin the runtime version in the repo, not in the pipeline config.
Commit lockfiles and install from them with exact matching on both sides.
Keep secrets and configuration out of the test path. If a test needs a token to pass, it is not a unit test.
When the pipeline fails, the reproduction step is always the same: run the target locally. If that does not reproduce it, the gap is the bug, and it goes to the platform team, not the developer.

5. Shared templates, not copied YAML

In any organisation with more than a handful of repos, pipeline definitions get copied. Then one copy gets a fix and the others do not. Six months later there are eleven slightly different versions of the same deploy job and nobody knows which one is right.

The alternative is a small set of reusable pipeline templates owned by the platform team, versioned, with a changelog. Application repos call the template and pass a few parameters. When the template improves, every repo gets it on the next run.

The trade-off is real: templates centralise control, and a bad template change breaks everyone at once. That is a reason to test templates properly, not a reason to avoid them.

The migration was easy. The customisations were not.

When we consolidated, we had dozens of repos, and almost every one had started life as a copy of another repo's pipeline. Moving the ones that had never been touched was straightforward. Point them at the shared template, run it, done.

The hard part was the repos with small customisations. An extra step here, a different flag there, added by someone who had since left, for a reason nobody remembered. Each one needed a conversation: is this still needed, and if so, should the template support it? Most of them were not needed. A few were, and they made the template better.

What I would do differently is version the templates from day one and publish a changelog. The first time a template change broke a downstream repo, the question was not what changed but when, and we had no clean answer.

6. Deploying should be boring, and rolling back should be more boring

A deploy that needs a runbook is a deploy people avoid. A deploy that happens on merge, with automatic rollback if health checks fail, is one people do ten times a day without thinking.

The signs a deploy process is hurting developer experience:

People batch changes to avoid deploying often.
Deploys happen at fixed times, usually chosen for the platform team's comfort.
Rollback is a different, harder process than deploy.
Someone has to be pinged.

Every one of those is a design choice, and every one of them can be reversed.

From a scheduled deploy window to deploy-on-merge

At its worst, production deploys happened in a window a few times a week, with a platform engineer watching. Developers batched changes to hit the window, which made every deploy bigger, which made every deploy riskier, which was the argument for keeping the window. The process was defending itself.

At its best, a merge to main built the image, ran the post-merge suite, and rolled it out to Kubernetes progressively. Health checks and error rate decided whether the rollout continued. If they failed, the rollout reversed itself and posted a message with the failing check. Nobody watched. Nobody needed to.

The single change that made rollback boring was making it the same operation as deploy. Rolling back meant deploying the previous image tag through the same pipeline. No special runbook, no separate permissions, no person on call to run it.

7. Remove the gates that do not add information

Manual approvals feel like safety. Most of them are theatre. If the approver clicks approve without reading, because they always click approve, the gate is adding delay and no protection.

A gate earns its place when the person approving knows something the pipeline cannot check. Most of the time, what they are checking can be automated: a security scan, a policy check, a required reviewer on the code itself. Automate it, put the result in the PR, and remove the manual step.

The approval that approved everything

Every production deploy needed a sign-off from a change manager. When we looked at the history, the approval rate was effectively one hundred percent, and the median time between request and approval was measured in hours. Not because anyone was reviewing. Because the request had to wait for someone to notice it.

We replaced it with the checks the approver was supposed to be doing: tests passed, security scan passed, code was reviewed by someone on the owning team. All three were already visible in the pull request. The manual gate stayed for exactly one category, database migrations, where a human genuinely did know something the pipeline could not.

The argument for keeping the gate was audit. The answer was that a pipeline log with named checks and a timestamp is a better audit trail than a click.

What this looks like from the developer's chair

Put together, a pipeline built for developer experience looks like this:

Push a branch. Within minutes, lint and unit results are annotated on the PR.
If something is red, the annotation says what and where, and the fix command is one copy-paste away.
Green means safe to merge. Nobody re-runs on principle.
Merge. The main branch builds, runs the slow suite, and deploys to production with health checks.
If health checks fail, it rolls back on its own and posts a message with the failing check.

None of this is exotic. Every piece exists in every major CI system today. The reason most pipelines do not look like this is that nobody made the developer's experience the goal.

Takeaways

Measure the pipeline from the developer's side: time to feedback, re-run rate, time from merge to production.
Anything slow goes after merge. Anything flaky goes out of the merge path until it is fixed.
Local and CI must run the same thing. Pin the runtime, commit the lockfile, share the container image, keep the YAML thin.
Deploys happen on merge. Rollback is the same operation, run backwards.
Every manual gate has to justify itself with information the pipeline cannot get on its own.

The part I am still working out is how to keep a pipeline this way once it exists. Every one of the problems above crept back in slowly, one reasonable-sounding exception at a time. Keeping the developer's experience as the design goal is not a project. It is maintenance.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/05/cicd-designed-for-developers/.