DEV Community: vivek Itp

What an artifact signature proves, and what it does not

vivek Itp — Wed, 30 Sep 2026 04:02:21 +0000

I wrote earlier about verifying commit signatures in the pipeline, and mentioned in passing that the build signs its own output. That sentence deserves a post of its own, because signing an artifact is easy and the claims people make about it are usually larger than what it actually supports.

What we have works. It is also narrower than the phrase "supply chain security" suggests, and the honest version is more useful to anyone building the same thing than a diagram with green ticks on it.

1. Sign the JAR and the image, because only one of them gets deployed

We sign both the JAR and the container image. For a while it was only the JAR, and that was the wrong place to stop.

The JAR is what the build produces. The image is what actually runs. Sign only the JAR and you have proof about an input to a later step and nothing about the thing that reaches the cluster, so anyone who can influence how the image is assembled sits in a gap the signature does not cover.

What you sign	What that lets you check	What is still open
The JAR only	The library came from a trusted build	Nothing about the image that wraps it
The image only	What runs was built by a trusted pipeline	Nothing about the code inside it
Both	Each handoff is checkable on its own	Whether the build itself did what it claimed

Why the second signature got added

The first version signed the JAR at publish time and stopped there. On paper the chain looked finished: signed commits, signed artifact, verification before deploy.

The question that broke it came from outside the platform team: the thing running in production is an image, so what have you proved about the image? The answer was nothing. The image was built in a separate job, from a base image we pulled and a JAR we trusted, and no signature covered the result. Signing the image closed that step — not the whole chain, as the rest of this post gets into, but it moved the boundary somewhere defensible.

2. Only the build can sign, and that is the whole control

Access to the signing key is restricted to the runner's role. No developer has it, no local build can produce a signed artifact, and there is no break-glass path that puts the key on a laptop.

That restriction is the entire security property. Everything else — the verification step, the metadata, the pipeline configuration — is plumbing that only means something because the key is unreachable from anywhere except the build. The moment a person can sign, a signature stops saying "this came from the pipeline" and starts saying "this came from the pipeline, or from someone who was in a hurry".

So the interesting question is not how do we sign but who can schedule a job on a runner that holds the signing role. That is a CI access question wearing cryptographic clothes, and it is where the real risk lives.

The request I keep having to turn down

The recurring ask is for a way to sign locally, and it is always reasonable in the moment: a release is blocked, the pipeline is red for an unrelated reason, the artifact is fine — can someone just sign it and push it through?

Saying no to that is the job. If a human can sign, every signature in the system becomes "probably from the build", and a signature that means probably is not worth the pipeline minutes it costs. What we do instead is keep the pipeline path fast enough that the question is about waiting rather than being blocked — and when that stops being true, the pressure to build a bypass comes straight back.

3. Verify at the point of use, and fail the deploy

The specifics in this section are the pattern rather than a story I am retelling exactly. Verification happens in the deploy job, before the artifact is pulled down and installed. If the signature does not verify, the deploy fails. There is no warning mode and no override flag.

Two details matter more than where the check sits. Fail closed: a step that logs a warning and continues is documentation, not a control. And say why in the first line: "no signature found" and "signed with an unknown key" are different problems with different fixes, and nobody should need a security runbook to tell them apart.

The failure message worth writing properly

The most common verification failure is not an attack. It is an artifact that predates the control, or one promoted between environments by a path that did not carry the signature with it. With a generic message both produce the same outcome: a red deploy, a developer who did nothing wrong, and a support request.

Both are obvious if the message names the artifact, says whether a signature was absent or unrecognised, and links to the one page explaining what to do next. The check is the easy part; the message decides whether people trust it.

4. A signature is not provenance

Here is the part I would want to read if someone else had written this post.

What we record alongside the artifact is the commit and the signature. That is it. There is no signed attestation describing the build, no record of which runner produced it, no statement of what the pipeline configuration was at the time.

So the chain supports this claim: this artifact was signed by our build, and it says it came from this commit. It does not support the claim people tend to hear, which is: this artifact was built from this commit, by this pipeline, with nothing else added. The commit reference is metadata the build wrote about itself. A signature over self-reported metadata proves the signer, not the statement.

That is not a reason to skip signing. It raises the cost of tampering after the build, which is the most likely place for it to happen. It is a reason to be careful about the sentence you put in a compliance document. "Traceable" is true. "Provable" is not, yet.

The distinction that matters in an audit conversation

The useful move in review is to say plainly which claim the setup supports, rather than showing the diagram and letting people read the stronger one into it.

Traceability is genuinely valuable: given an artifact we can get to a commit, and given a commit to a verified author, and every link is checkable. What we cannot do is prove the build did only what the pipeline file said. Closing that means attesting the build itself, which is a larger piece of work than signing ever was.

5. Key rotation is the failure you will actually hit

Not an attacker. A lost key.

When a developer loses their key — a replaced laptop, usually — they generate a new one, and everything signed with the old key is suddenly signed by something the verifier does not recognise. The signature is still valid; the key it points to is no longer registered.

The fix is to keep old public keys rather than replace them. The parameter store holds retired keys alongside the current one, so historic signatures keep verifying while new work is signed with the new key, and revocation becomes a deliberate act for a key you believe is compromised rather than a side effect of someone getting a new machine.

This is the single most useful thing I would tell anyone rolling out signing of any kind. Design for rotation before you design for enforcement. Enforcement is a day of work. Rotation is what determines whether the control survives contact with normal life.

What it looks like when you have not planned for it

The pattern is always the same. Someone gets a new laptop, generates a fresh key, and their next few pieces of work fail verification. From their side the control is broken: they did nothing unusual and the pipeline is refusing their change. Keeping retired keys in the store fixes it, and it costs nothing except deciding once that key history is state you keep rather than state you overwrite.

What still needs a human is deciding whether a key was lost or taken. Those need opposite responses — keep the old key so history verifies, or revoke it so it stops — and no automation tells you which one you are looking at.

6. The part that is still unsolved: knowing which commits are new

The gap I have not closed is not about signing at all. It is about deciding which commits a pipeline run is supposed to check.

When a branch is pushed for the first time, the "before" reference the hook receives is all zeros — there is no previous state to compare against. You cannot tell from that alone where the new work starts. If the branch was taken from another branch rather than from the mainline, walking back from the tip picks up commits that belong to whoever wrote them, not to the person pushing now.

The consequences run both ways, and both are bad:

Too wide: the check covers commits from before the branch point, and a developer is blocked by history they did not write.
Too narrow: the range misses commits, and unverified work reaches a protected branch while the pipeline reports green.

Comparing against the mainline merge base is the obvious answer and it is not reliable either — a branch taken from a branch has a merge base that is not where the developer thinks it is, and a rebase moves it after the fact.

How we live with it today

We compare against the mainline and accept that a branch taken from another branch is a case we get wrong. When it happens the developer sees a failure they cannot explain, asks in the channel, and we look at the range by hand. Calling that a solution would be generous — it is a known rough edge with a manual escape, which is the honest description of most controls at this stage.

What I think the real fix looks like is recording the verified range rather than recomputing it: storing what has already been checked, so the pipeline asks what is new since the last verification instead of inferring the boundary from the push. I have not built it.

Takeaways

Sign every artifact that gets handed on, not only the first one. The JAR and the image are both handoffs.
Keep the signing key reachable only from the build. That restriction is the security property; everything else is plumbing.
Fail the deploy on a bad signature, and make the message distinguish "no signature" from "unknown key".
Be precise about what you have. A signature over self-reported metadata gives you traceability, not attested provenance.
Plan key rotation first. Keep retired public keys so history still verifies, and treat revocation as a deliberate decision.
Work out how you will identify new commits before you enforce anything on them. It is harder than the signing.

The signature part of this was a week of work. The part I am still circling is knowing exactly which commits a given run is responsible for, and until that is solved the enforcement around it is only as trustworthy as the range it was handed.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/30/what-an-artifact-signature-proves/.

Why our Java builds run on Fargate and our Docker builds do not

vivek Itp — Sun, 27 Sep 2026 13:20:34 +0000

Most discussions about self-hosted CI runners start with cost. Managed minutes are expensive, the argument goes, so run your own and save money. That was not why we did it, and cost is usually the weakest reason on the list.

We run our own GitLab runners because of a requirement we could not meet any other way: builds have to sign the artifacts they produce, with a key the build host is allowed to use and nobody else is. Everything below follows from that one constraint, and from the fact that not every build wants the same kind of machine.

1. The reason to self-host is usually a requirement, not a saving

If your builds are ordinary, managed runners are the right answer. Somebody else patches them, scales them, and gets paged when they break. You should need a reason to give that up.

Ours was artifact signing. The publish step signs the JAR with a KMS key before it reaches the artifact repository, and the deploy step refuses anything that does not verify. That only works if the machine doing the signing is one we control: our account, our IAM role, our network path to the key, and no shared tenancy with builds that should never reach it.

Once you own the compute for that reason, the other arguments start to matter — private network access, predictable machine sizes, caches that survive between jobs. But they are consequences, not the case.

What actually forced the decision

We had already moved to signed commits, verified in the pipeline. The next question from the security side was the obvious one: you have proved who wrote the code, so can you prove which build produced the binary?

Answering that meant the runner had to hold an identity: not a secret pasted into a CI variable, but a role with permission to use one specific key, scoped so only jobs from projects meant to sign could assume it. You cannot do that on shared managed compute in any way I was comfortable defending.

2. Split the compute by what the job builds, not by who owns the job

We run everything from one centralised AWS account, with two kinds of executor behind it: ECS Fargate and EC2. The split is not by team or by environment. It is by what the job actually does.

Job type	Runs on	Why
Java builds, tests, artifact publish	ECS Fargate	Ordinary processes, no privileged access needed, scales to zero
Container image builds	EC2	Needs a real Docker daemon and a disk that behaves like a disk

Fargate is the better default. No host to patch, capacity appears when a job appears, and nothing is running means nothing is billed. For a Maven or Gradle build that is all upside.

Container builds are where it stops working. Building an image wants a daemon, layer storage that behaves like a local disk, and occasionally privileges a serverless container platform is specifically designed not to give you. You can fight that with alternative builders, and we looked at it, but EC2 hosts do this without argument and Fargate does it with a fight.

So Java builds go to Fargate, Docker builds go to EC2, and the developer writes neither of those words. They pick a runner tag and the tag decides.

The rule that removed the question

Before we split it, the recurring question in the platform channel was "which runner should my job use?", and the answers people gave each other were folklore copied from whichever project they had worked on last.

One sentence fixed most of it: if the job builds an image it runs on the EC2 tag, and everything else runs on the Fargate tag. That fits in a template comment, and nobody has to understand executor internals to get it right. The jobs that still get it wrong are the ones that do both, which is usually a sign the job is doing too much anyway.

3. Isolation is a decision you make once, early, and live with

Each executor type runs as two stacks, and each stack has two sets of runners separated by a restricted flag. Restricted runners can reach the signing key and the production-facing artifact paths. Unrestricted ones cannot, and that is enforced by the role attached to the runner, not by anything in the project's pipeline file.

The reason for two sets rather than one is simple: if every runner can assume the signing role, every project in the instance can sign, including the one someone created this morning to try something out. Permission on a runner is permission for whoever can schedule a job onto it.

The thing I would tell anyone starting this: decide the isolation boundary before you have twenty projects on the runners, not after. Adding a project to the restricted set is a five-minute change. Taking access away from projects that have quietly come to depend on it is a quarter of conversations.

Why the flag sits on the runner and not in the pipeline

The first version had the signing step guarded by a check inside the pipeline template: if the project was on the approved list, the step ran. That is a check a project can edit its way around, because the pipeline file lives in the project's own repository. The list was in the right place for convenience and the wrong place for security.

Moving the boundary onto the runner's IAM role changed the failure mode. A project that is not supposed to sign does not fail a policy check. It gets an access-denied from AWS, because the machine it runs on genuinely cannot use the key.

4. For critical projects, the pipeline configuration is not the team's to change

Alongside the tagging, we run what we call guarded CI. For a short list of critical projects, the pipeline configuration is hardcoded: the runner tags, the signing step, and the verification step are fixed, and the project cannot override them from its own file.

This is a deliberate reduction in flexibility and it is unpopular in exactly the way you would expect. The argument for it is that the controls on a critical application should not be removable by editing a YAML file in that application's own repository. If a team can turn off the check that protects them, the check is a suggestion.

The cost is real: guarded projects wait on us for changes other teams make themselves. We keep the list short so that stays survivable.

Where guarding earns its keep

The moment that justified it was mundane. A build was failing on the verification step, and the quickest path to green — the one anybody under pressure reaches for — was to comment the step out and come back to it later.

On a normal project that works, and sometimes nobody notices for a month. On a guarded project it does nothing, because the step does not come from the project's file. The team pinged us instead, and the cause was a stale key reference that took minutes to fix. Being unable to bypass the check is what got the right people looking at it the same day.

5. Caching is where self-hosted runners give the time back

The specifics here are the pattern I would recommend rather than a story I am retelling exactly. A Java build that resolves dependencies from scratch every time spends a meaningful chunk of every run redoing work. Self-hosted runners let you fix that, and it is the clearest developer-visible win available.

Two things matter more than the rest:

A shared dependency cache in S3, keyed on the lockfile, so a build only pays full resolution cost when dependencies actually change.
Warm layer storage on the EC2 hosts for image builds, so a rebuild of an application whose base image has not changed reuses the layers it already has.

The trap is treating the cache as free. A cache that is never invalidated eventually serves something stale and costs you an afternoon, and a cache keyed too loosely is a correctness problem wearing a performance costume. Key it on the lockfile, expire it on a schedule, and make cache hits and misses obvious in the job log.

The shape of the win

The pattern to expect is that the first build after a dependency change pays full price and every build after it takes noticeably less. What makes it worth doing is the common case rather than the average: a small code change on a project whose dependencies have not moved in weeks gets fast, and that is where developers spend almost all their time.

6. The maintenance is the part nobody puts in the estimate

Also a recommendation rather than a specific recollection. Building the runner setup gets estimated. The ongoing cost usually does not, and it is not small:

Patching the EC2 hosts, and rotating them often enough that patching is routine rather than an event.
Keeping the runner version in step with the GitLab instance, because drift produces failures that look like project problems.
Watching the autoscaling behaviour, because a capacity problem arrives at the worst moment and looks like a slow pipeline.
Telling platform failures apart from project failures quickly enough that the distinction is useful.

Budget it as a standing share of somebody's time, not as a project with an end date. If nobody owns it, the runners keep working right up until the day they very loudly do not.

7. What developers actually notice

Not the architecture, and not the isolation model. They notice that a build sometimes takes longer to start than it should, and that the reason is invisible from where they sit.

That is a queueing effect and we have not solved it. Concurrency limits are an awkward trade: set them high and a busy afternoon has several heavy builds fighting over the same hosts, so everything slows at once. Set them low and jobs queue while capacity sits idle, because the limit rather than the machine is the constraint. We are still tuning it.

The mitigation that helped most was not a capacity change. It was making the wait visible: if a job is queued rather than running, a developer should see that without asking, because queued and slow are different problems and only one of them is theirs.

Takeaways

Self-host for a requirement you cannot meet otherwise, not for a saving. Ours was signing artifacts with a key only our build hosts can use.
Split executors by what the job builds. Fargate for ordinary builds that scale to zero, EC2 for image builds that want a real daemon and a real disk.
Put the isolation boundary on the runner's role, not in a pipeline file the project can edit.
Guard the configuration for critical projects, keep that list short, and accept that it costs those teams some autonomy.
Cache deliberately, key on the lockfile, and make hits and misses visible in the log.
Budget maintenance as standing time. Patching, version drift and scaling behaviour do not stop arriving.

What I have not worked out is the concurrency question. Every setting we have tried is a choice about who waits and when, and the right answer probably changes with the time of day in a way a single number cannot express.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/27/why-java-builds-run-on-fargate/.

Measuring developer experience without a survey

vivek Itp — Wed, 23 Sep 2026 22:55:37 +0000

The usual first move when someone decides to take developer experience seriously is to send a survey. It is the obvious thing to do and it is rarely the most useful thing available, because the data you actually need is already sitting in systems you run.

I am not against surveys. They capture things instrumentation cannot: whether people feel trusted, whether they think anyone is listening, whether the platform is something they use or something that happens to them. But as a starting point they have three problems, and all three are avoidable.

They are answered by the people who already engage. They measure sentiment at a moment rather than friction over time. And they take weeks to run, which is long enough for the thing that annoyed everyone to be forgotten or fixed.

The numbers are already there

What to measure	Where it already lives	What a bad number means
Time from push to a merge signal	Your CI system	People context-switch while waiting
Re-run rate on checks	Your CI system	Nobody trusts red any more
Time from repository creation to first production deploy	Repo events and deploy logs	Onboarding a service costs days
Support requests per team per month	Your support channel	The platform cannot answer for itself
Time from merge to production	Deploy logs	Shipping is a decision, not a default
Rollback frequency and time to roll back	Deploy logs	Either quality, or fear of deploying

None of those requires a new vendor. They require someone to go and count, and then to keep counting.

1. Pick numbers that describe a developer's day

The test for a developer experience metric is whether a developer would recognise it as describing their own week. Deployment frequency across the organisation does not. Waiting eleven minutes for a pipeline does.

That rules out most of what gets reported upward. Aggregate throughput numbers are useful to someone, but they average away exactly the thing you are trying to see: that one team waits forty minutes and another waits four, and the average of twenty-two describes nobody.

Keep the numbers per team, or per service, and look at the distribution rather than the mean. The interesting signal is almost always in the tail.

What this looked like for me

The first number I tracked properly was time from push to a merge signal, per repository rather than overall. The organisation-wide figure had looked acceptable and had told me nothing.

Broken out per repository, the picture was completely different. A small number of repositories were far slower than the rest, and they belonged to the teams who complained most. The complaint and the data had been saying the same thing for a while; only one of them had been legible.

2. Re-run rate is the most honest number you have

If I could keep only one, it would be this: how often does someone re-run a check without changing anything.

Nobody re-runs a check they believe. A high re-run rate means the pipeline has stopped carrying information, and that the team has learned to treat red as noise. It is the clearest possible measurement of trust, and it is free, because your CI system already records it.

It is also a leading indicator. Trust erodes before anything visible breaks. By the time a real failure gets re-run three times and merged, the erosion happened months earlier.

What the re-run rate exposed

Counting re-runs turned a vague sense that the pipeline was flaky into a short, ordered list of the specific jobs people did not believe. The same few appeared over and over.

That list was more actionable than any survey response about pipeline confidence would have been, because it named the jobs rather than the feeling.

3. Measure the whole path, not your part of it

A platform team naturally measures the parts it owns, which is exactly how the slow parts stay invisible. The queue you control looks fast because it is fast. The wait that hurts is somewhere between two systems, or in another team's inbox.

Measure end to end, from the developer's first action to the outcome they wanted, and treat every handoff as part of the number. If getting a namespace takes two days in someone else's queue, that belongs in time to first deploy whether or not you own the queue.

The dotted spans are the measurements. Both cross boundaries the platform team does not own, and that is the point: they describe the developer's experience rather than the platform's.

The wait that was not in our numbers

When I walked the new-service path myself and timed it, almost all of the elapsed time was waiting in queues belonging to other teams. Every system I owned was fast, and the developer's experience was still days long.

Our dashboards had been honest about our part and silent about the total, which made them reassuring and useless.

4. Count the questions

Support volume is a developer experience metric and almost nobody treats it as one. Every question is a moment where the platform failed to answer for itself.

You do not need tooling for this. A tally of what gets asked, bucketed roughly and counted, will show you a distribution steep enough to act on within a few weeks. The top few buckets are the platform's real backlog.

Watch the trend rather than the total. A channel that gets quieter each quarter is a platform that is learning. A channel with a stable volume and a fast response time is a platform that has got good at answering the same questions forever.

5. Trends, not benchmarks

The question worth asking is whether a number is better than it was last quarter. The question that wastes time is whether it is better than some other company's number.

Published benchmarks come from organisations with different constraints, different regulatory positions, and different definitions of the same word. Comparing against them produces either false comfort or a target nobody can reach for reasons that have nothing to do with effort.

Your own history is the only honest comparison. It shares every confounding factor, which is exactly what makes it useful.

6. Do not turn them into targets

The moment a number becomes a target, it stops measuring what it measured. This is not a new observation and platform teams fall into it anyway, usually with the best intentions and a quarterly goal attached.

If re-run rate becomes a target, the tempting fix is to remove the flaky tests rather than fix them. If time to first deploy becomes a target, the fix is to redefine when the clock starts. Both improve the number and neither improves anyone's day.

Keep them as instruments. They tell you where to look and whether something you changed helped. They are not a scoreboard, and the moment anyone's performance review touches them, they are finished as measurements.

Where I would put the numbers

Putting the measurements where developers could see them, rather than only in a report going upward, changed how they were received. Teams recognised their own figures and occasionally disputed them, which was useful, because a disputed number gets examined.

The times these numbers were least useful were when they appeared in a summary for people who did not work with the platform daily. At that altitude they stop being diagnostic and start being a score.

Then run the survey

Everything above measures friction. It cannot tell you whether people feel they can get things changed, whether they trust the platform team, or whether the documentation is any good.

So run a survey, but run it second, when you already know where the friction is. The questions get sharper, the responses are easier to interpret, and you can ask about the specific things your numbers flagged rather than asking people to rate the developer experience out of ten.

Takeaways

Start with what you already record. Your CI system and deploy logs hold most of it.
Per team, not aggregate, and look at the distribution. The average describes nobody.
Re-run rate is the single most honest number and it costs nothing.
Measure end to end, including the queues you do not own.
Count support questions and watch the trend, not the total.
Compare against your own history, never a published benchmark.
Never make them targets. They are instruments.
Run the survey second, to ask about the things instrumentation cannot see.

What I still do not know how to measure is the work that never gets attempted. Some fraction of good ideas die because someone correctly judges that the platform would make them painful, and none of those decisions leave a trace in any system I run. Everything above measures the friction people pushed through, not the friction that stopped them.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/23/measuring-devex-without-a-survey/.

What migrating logging from Datadog to Dynatrace actually cost us

vivek Itp — Mon, 21 Sep 2026 23:38:56 +0000

Every platform team eventually has to take something away. The technical part is usually straightforward. What goes wrong is everything between announcing it and switching it off, and the cost almost never lands where you expect.

The clearest example I have is moving our logging from Datadog to Dynatrace. The driver was ordinary: cost, plus consolidating logs and metrics onto one platform instead of paying two vendors to hold two halves of the same picture.

Before anything else: this is not a comparison of the two products, and nothing here is a complaint about either. Both do the job. The decision was commercial, the way most tooling decisions are, and what follows is about the migration rather than the tools. I have kept the internal specifics deliberately light.

I expected the hard part to be the applications. It was not. The applications barely noticed. What actually cost us was the monitoring built on top of the logs, and the fact that everyone who used logs daily had to relearn how to ask a question.

What I expected to be hard, and what was

Expected to be hard	Actually hard
Changing every application	The forwarder changed; apps mostly did not
Getting teams to prioritise the work	It rode along with their normal deployments
Log data itself	Custom masking rules tied to the old vendor
Nothing in particular	Every monitor and alert built on the old logs
Nothing in particular	Teams learning to query in the new tool

The two bottom rows are the whole story. Neither was on my list at the start.

1. Find the thing that is not in the obvious place

On paper the change was small. Logs went to a forwarder, and the forwarder needed to point somewhere new. No application code, no library swap, no redeploy required to change a destination.

What was not on paper was everything that had accumulated inside that forwarder. Over the years it had picked up custom masking rules, written against the old vendor's configuration format, quietly redacting sensitive values before logs left our systems. They were not documented as a dependency on the vendor. They were just part of how logging worked.

That is the shape of the hidden dependency in most deprecations. Not the integration everyone knows about, but the small accretions around it that nobody wrote down because they were never a decision, only a series of fixes.

What this looked like for me

The forwarder swap itself was an afternoon of work. The masking rules were not, because they had to be reproduced exactly in a different configuration language, and "exactly" is the operative word when the rules exist to keep sensitive values out of a log store.

Going through them one at a time was tedious and it was also the only responsible option. That was the point where the migration stopped being a config change and became a piece of work with a real review attached.

2. Run both, and validate, before removing anything

The decision that made everything else possible was refusing to cut over. For a couple of months, logs flowed to both platforms at once. Nothing was removed while we compared.

Dual shipping costs money, briefly, and it buys the only thing that makes removal safe: evidence. Not a belief that the new pipeline works, but a side-by-side you can point at. Same volumes, same fields, same masking applied, same events present in both.

The loop in the middle is what a validation period is for. Without it, "are we ready to switch off" is a judgement call made under time pressure. With it, the answer is a comparison anybody can check.

What the overlap actually caught

The overlap did not produce a dramatic discovery, and I count that as the period doing its job rather than being unnecessary. It let us confirm that masking was being applied the same way on both sides, which was the thing I was least willing to guess about.

Had we cut over directly and been wrong about that, we would have found out from the wrong direction entirely.

3. Let it ride on work teams are already doing

We did not run this as a migration project with a deadline and a chase list. Teams picked up the change as part of their normal feature deployments. Whatever they were shipping next carried the new configuration with it.

That choice is worth being honest about, because it trades one problem for another.

What you gain is real. There is no separate piece of work competing with a team's roadmap, no deadline anyone has to defend, and no incentive to rush. The change arrives with something the team wanted to ship anyway.

What you give up is predictability. A team that does not deploy for six weeks has not migrated for six weeks, and the tail is as long as your slowest-moving service. You cannot put a date on the board and be confident about it.

I think it was the right call here, specifically because dual shipping made a long tail harmless. Nothing was breaking while we waited. Take away the overlap and the same approach becomes a slow-motion outage waiting for the least active team.

No chase list

The thing I noticed most was the absence of the usual conversation. Nobody had to be persuaded, because nobody was being asked to stop what they were doing.

The trade shows up at the other end. The final stretch is a small number of services that simply had not deployed recently, and those needed individual conversations rather than a broadcast.

4. The applications were fine. The monitoring was not

Here is the part I would tell anyone planning a similar move: the migration is not the logs, it is everything built on top of them.

Monitors, alerts and dashboards are written against a specific query language, specific field names, and a specific way the platform structures data. None of that survives a vendor change automatically. Every one of them has to be rebuilt, tested, and trusted again.

That work is invisible in the plan, because nobody thinks of an alert as an integration. It is also the work where getting it wrong is worst: an alert that silently stops firing is far more dangerous than a log line that fails to arrive, and it fails quietly by definition.

Rebuilding the things nobody counted

The monitors were the bulk of the real effort and were not what I had scoped at the start. Each one had to be re-expressed in the new platform and then actually verified, because the only way to trust an alert is to make it fire.

If I ran this again, I would treat monitor migration as the main workstream from day one, and the forwarder change as the small task it turned out to be.

5. The real cost was people relearning how to ask

The last thing, and the one I keep thinking about, is that the hardest part of this was not technical at all.

People had years of muscle memory in the old tool. They knew how to get from an alert to the relevant log line in three moves, because they had done it hundreds of times. The new platform can answer all the same questions and it answers them differently, and that difference lands during incidents, when nobody has patience for learning.

That is a genuine cost of any tooling migration and it usually goes unnamed, because it does not appear as a ticket or an outage. It appears as things taking longer for a while, and as a quiet preference for the old tool for as long as the old tool still exists.

What would have helped is treating query fluency as part of the migration rather than an afterthought: a short page of the ten queries people actually run, translated, and a channel where asking how to express something was normal rather than an admission.

6. Leave a tombstone

When the old destination finally went away, what mattered was that anyone who went looking found an explanation rather than silence. What was removed, when, where the data lives now, and how to ask the question they were trying to ask.

Keep it up longer than feels necessary. The person who needs it is the one who only looks at logs when something is badly wrong, and they will arrive months later.

Takeaways

Look for the accretions, not the integration. The forwarder was easy; the masking rules written for the old vendor were not.
Run both and compare before removing anything. The overlap buys evidence rather than confidence.
Riding on normal deployments avoids the chase list and gives up predictability. It is only safe if nothing breaks while you wait.
Scope the monitors as the main work, not the pipeline. An alert that stops firing fails silently.
Name the relearning cost. Query fluency is part of the migration, and it is paid during incidents.
Leave an explanation behind for the person who shows up months later.

What I still have no good answer for is the deprecation with no replacement, where the honest message is that a capability is going away and teams will have to do without it. Everything above works because there was somewhere to go. When there is not, the conversation is about priorities rather than migration, and I have never found a way to make that one go smoothly.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/21/migrating-logging-datadog-to-dynatrace/.

Local development environments that match production closely enough

vivek Itp — Mon, 21 Sep 2026 23:38:51 +0000

Two failure modes, and I have caused both. The first is a local environment so far from production that "it works on my machine" stops being a joke and becomes a daily fact. The second is chasing full parity until starting the thing locally takes fifteen minutes, eats most of a laptop, and developers quietly stop using it.

Neither is a discipline problem. They are both the result of not deciding, explicitly, which differences between a laptop and production are allowed to exist.

I have written before about local and CI needing to run the same commands. This is the neighbouring question and a harder one: how close does a developer's running system need to be to the real one.

Some differences matter and most do not

Difference	Does it change behaviour?
Database major version	Yes. Syntax, query planning, and migrations all move
Runtime version	Yes. This is where most "works locally" bugs live
One replica instead of forty	No, until you are debugging concurrency
Real payment provider vs a fake	No, if the fake matches the contract
No TLS locally	Usually not, until a client library behaves differently
Ten rows instead of ten million	Not for correctness. Absolutely for performance work
Different operating system	Sometimes. Path handling, file watching, case sensitivity

The right-hand column is the whole exercise. Parity is not a slider you push toward one hundred percent. It is a set of individual decisions, each of which should have a reason.

1. Match the contract, not the infrastructure

The thing a service needs from its dependencies is the contract: the same interface, the same error shapes, the same semantics. It almost never needs the same topology, the same scale, or the same managed service.

A local queue that delivers messages at least once, out of order, and occasionally redelivers is a good stand-in for the production one, whatever runs underneath. A local queue that delivers exactly once and in order is worse than useless, because it lets people write code that cannot work in production and gives them confidence while they do it.

This is the test I use for a fake: does it fail in the same ways as the real thing? If not, it is teaching the wrong lessons.

What this looked like for me

We had a local stand-in for a message broker that was much friendlier than the real one. It delivered in order, never redelivered, and never dropped anything.

Code written against it worked locally and then behaved oddly in production, where duplicates and reordering were normal. The stand-in had been chosen because it was easy to run, and the ways in which it was easier were exactly the ways that mattered. Replacing it with something that redelivered and reordered made local development slightly more annoying and caught a category of bug before it shipped.

2. Pin the things that change behaviour

Runtime version and database major version are the two that have caused me the most confusion, and both are cheap to fix. Pin them in the repository, read the same pin everywhere, and the whole class of problem disappears.

The rule I use: if a difference can change the result of running the code, it goes in a file in the repository, and both the laptop and the pipeline read that file. If it cannot, it does not need to match.

# one place, read by the local tooling and by CI
runtime: "3.12.4"
database: "postgres:16.3"

What does not work is documenting the required versions in a README. Nobody re-reads a README after their first week, and versions drift silently from the moment they are written down.

The version that drifted

The most time-consuming local problem I dealt with was not dramatic: a developer's database was a major version behind what production ran, because they had installed it a year earlier and nothing had ever told them to change it.

A migration worked on their machine and failed in the pipeline with an error that pointed at the migration rather than the version. Pinning the database image and having the local tooling refuse to start on a mismatch removed that whole category, and the refusal message did more good than any documentation would have.

3. Decide what you are deliberately not matching

The parts that are not worth matching should be an explicit, written decision rather than an accident. Scale, multi-region behaviour, real third-party accounts, production data. Each of those costs a great deal to reproduce and buys very little for the work a developer does on a normal day.

Writing them down matters more than it sounds. When a developer hits a bug that only appears at scale, the useful response is "yes, local does not cover that, here is the environment that does" rather than an argument about whether local should have caught it.

The branch on the right is where most local setups go wrong. They fake something, the fake is friendlier than reality, and nobody notices until production disagrees.

4. Start-up time is a parity feature

An environment that takes fifteen minutes to start is not a high-parity environment. It is an environment nobody runs, which means its parity is irrelevant.

Every service added to the local stack has a cost paid by every developer, every day, forever. That cost is rarely weighed against the benefit, because the benefit is visible and the cost is spread thin.

The questions worth asking before adding anything to the local stack:

Does a developer need it running to do their normal work, or only occasionally?
Can it start lazily, when something actually calls it?
Can it be a fake that matches the contract rather than the real service?
What does it add to start-up time, and is that worth it every single day?

Cutting the stack down

Our local environment had grown to include everything a service might talk to, because each addition had been individually reasonable. Starting it had become something you did while making coffee, and a lot of people had stopped doing it at all.

Splitting it into a small default set that starts quickly, plus optional extras you opt into when you need them, was not technically interesting and made more difference than anything else on this list. Most developers, most days, needed a fraction of what we had been starting for them.

5. Make the difference visible, not surprising

When local and production do differ, the developer should be able to see it rather than discover it through a failure.

A short banner at start-up listing what is real and what is faked costs nothing and prevents a lot of confusion. "Payments: fake. Queue: local, redelivers. Database: postgres 16.3, schema current. Search: not running." A developer who sees that line does not spend an hour wondering why a payment did not arrive.

The same applies to data. If the local database has a small generated dataset, say so, ideally with a note about what it does not contain. Silence gets interpreted as "this is like production", and it never is.

6. Test the environment itself

A local environment is a piece of software that the whole team depends on, and it usually has no tests at all. It rots quietly: a fake drifts from the real service's contract, a pin goes stale, a start-up script depends on something that was removed.

A small scheduled job that starts the local environment from scratch on a clean machine and runs the smoke tests against it catches this early. When it breaks, it breaks for one person who can fix it, rather than for whoever next tries to onboard.

The setup that only worked if you already had it

Our setup instructions worked perfectly for everyone who had run them before, and reliably failed for new joiners. The scripts had come to depend on tools and state that existing machines happened to have.

Nobody had noticed because nobody started from scratch. Running the whole setup on a clean machine on a schedule surfaced it immediately, and onboarding stopped being a two-day exercise in asking colleagues what else they had installed.

Takeaways

Parity is a set of decisions, not a percentage. For each difference, ask whether it can change the result.
Match the contract, not the infrastructure. A fake that is friendlier than reality teaches the wrong lessons.
Pin what changes behaviour in the repository, read by both the laptop and the pipeline. Never in a README.
Write down what you deliberately do not match, so the gap is a known limit rather than an argument.
Guard start-up time. An environment nobody runs has no parity at all.
Print what is real and what is faked at start-up.
Test the environment on a clean machine on a schedule, or it rots.

The one I keep going back and forth on is production data. Realistic data catches a whole class of bug that generated data never will, and every approach I have used to get it locally, whether subsetting, masking, or anonymising, has either leaked something it should not have or been so lossy that it stopped being realistic. I do not have a good answer there.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/21/local-environments-close-enough-to-production/.

The support channel is your best product research

vivek Itp — Thu, 17 Sep 2026 14:44:38 +0000

Platform teams are usually bad at user research. We do not run interviews, we do not have a researcher, and the survey we send once a year gets answered by the people who already like us. Meanwhile the same team runs a support channel that receives dozens of real, unprompted, specific accounts of the platform failing its users, every single day.

We just do not read it that way. We answer the question, mark it done, and move on. The answer is the least valuable thing in the exchange. The question is the data.

I changed how I worked when I started treating the channel as a research feed instead of a queue. This post is what that looked like.

The exchange has two halves and we keep the wrong one

A support request has a visible half and an invisible half.

Visible	Invisible
"How do I get logs for my staging pod?"	They looked for the answer and could not find it
"Can someone re-run my pipeline?"	They do not have permission, or do not know they do
"Is the registry down?"	The error they saw did not say what was wrong
"What is the right base image?"	There is no obvious default, or there are three
"Who owns the deploy for service X?"	Ownership is not discoverable from the tooling

The visible half gets answered in two minutes and helps one person. The invisible half, if you collect it, tells you what to build. Every single question is a place where the platform failed to answer for itself.

1. Count the questions before you answer them

The cheapest useful thing a platform team can do is keep a tally. Not a ticketing system, not a taxonomy. A tally.

Every time a question comes in, put it in a bucket. Reuse buckets aggressively; if it is roughly the same question, it is the same bucket. After a few weeks the distribution will be extremely uneven, and the top few buckets will account for most of the volume.

That list is your backlog, ordered by how much pain each item causes, measured in the only currency that matters: how often a human had to ask another human.

What this looked like for me

I started with a text file and a habit. Every question I answered got a line. No categories decided up front, just a short description, and when something repeated I merged it into the existing line and added a tick.

After about a month there were maybe thirty lines, and the top five accounted for most of the ticks. None of the top five was on our roadmap. All five were small. That file changed what we worked on for the next quarter more than any planning session did.

2. The repeat question is the bug

A question asked once is a person having a bad day. The same question asked twenty times is a defect in the platform, and it should be treated exactly like one: reproduced, root-caused, fixed, and verified.

The trap is that answering is so cheap. Two minutes, a link, done. Twenty times two minutes is not the real cost, though. The real cost is the twenty people who were blocked until someone was available, plus the ones who did not ask and worked around it instead.

When a question repeats, the fix is almost never documentation. Documentation is where answers go to be not found. The fix is usually to make the question impossible: a better default, a clearer error, a link in the place they were already looking.

The question I answered too many times

The single most repeated question I dealt with was some version of "how do I see the logs for my service in staging". I had a good answer. I gave it dozens of times. I linked the documentation page, which existed and was correct.

What finally stopped it was not a better page. It was putting a direct link to that service's logs into the deploy notification, so the answer arrived before the question. The documentation page did not change at all. The questions stopped almost entirely.

3. Watch for the questions that are not questions

Some of the most valuable signals in a support channel are not phrased as requests. They are asides, complaints, and jokes.

"Ha, the pipeline is red again, business as usual."
"I just re-run it twice, works eventually."
"We stopped using that, it was easier to do it ourselves."
"Do not worry, everyone knows you have to do X first."

None of those is asking for help, so none of them gets logged anywhere. Each one describes a workaround that has become normal. A workaround that has become normal is a feature the platform failed to provide, and nobody will ever file a request for it because they have stopped expecting it to change.

I started keeping these in the same tally as the questions. They were often more useful, because they described problems people had already given up on.

The joke that was a roadmap item

Someone joked in the channel that the correct way to deploy was to "press the button twice and go get a coffee". It got a few laughs, including mine, and then I realised everyone in the thread knew exactly what it meant. The first deploy attempt failed often enough that re-running had become the accepted procedure.

Nobody had ever reported it, because it was not broken in a way you could report. It just did not work the first time, reliably enough that people had built a ritual around it. That went into the tally and turned into real work.

4. Answer in public, always

A support question answered in a direct message helps one person and produces no data. The same question answered in the channel helps everyone watching, gets found by search later, and stays in your tally.

Push everything into the open channel, politely and consistently. When someone asks privately, answer in the channel and link them. Not to shame anyone, but because the private answer is a small gift to one person and a loss to everyone else, including you.

The loop only closes if step three actually happens. A channel where everything is answered and nothing is counted stays exactly as busy next year as it is today.

Moving everything into the open

A lot of our support traffic arrived as direct messages, usually to whoever the person had spoken to last. That felt friendly and it was quietly destructive: the same question was being answered in four private conversations, and none of us knew it was common.

Moving to public-by-default took a few weeks of gently redirecting people. The channel got noisier, which looked like a regression. It was not. The noise had always existed, spread across private messages where it could not be measured.

5. Close the loop out loud

When you fix something that came from the channel, say so in the channel, and name the question it came from.

This is not self-promotion. It is the only way people learn that reporting friction leads to it being fixed. Teams that never see that connection stop reporting, and once they stop, the research feed dries up and you are back to guessing.

The message is short: this thing you all kept hitting, here is what changed, here is what you do now. It costs nothing and it is the difference between a channel that gets quieter every quarter and one that gets more useful every quarter.

What changed when we said it out loud

We had been fixing things from the channel for a while without announcing it, because each fix felt too small to mention. The result was that people kept asking about problems we had already solved, and nobody could tell that the platform was improving.

Posting a short note each time a channel-sourced fix shipped changed the tone noticeably. People started reporting smaller things, including things they had previously worked around silently. The quality of what we heard went up because reporting had visibly started to pay.

6. Be careful what you optimise

Once you start measuring the channel, the tempting metric is response time. It is easy to collect and it looks like service quality.

It is the wrong target. A team optimised for response time gets very good at answering quickly, which makes the channel pleasant and permanent. The goal is not to answer faster. It is to receive fewer questions because fewer things need asking.

The number worth watching is question volume per team per month, or how long the top bucket stays at the top. If the top question is the same one it was six months ago, the platform is not learning, however fast anyone replies.

Takeaways

The question is the data. The answer helps one person; the pattern tells you what to build.
Keep a tally, not a taxonomy. A text file and a habit will beat a tool nobody updates.
A repeat question is a defect. Fix the cause so it cannot be asked, and do not reach for documentation first.
Log the asides and the jokes. A normalised workaround is a request nobody will ever file.
Answer in public so the data exists at all.
Announce channel-sourced fixes or people stop reporting.
Do not optimise response time. Optimise for the question not being needed.

What I still have not solved is the silent majority. Everything above depends on someone asking, and the teams that struggle most are often the ones least likely to post in a public channel at all. I do not have a reliable way to hear from them short of going and asking, which does not scale.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/17/the-support-channel-is-product-research/.

Secrets management that developers do not route around

vivek Itp — Thu, 17 Sep 2026 14:44:33 +0000

Every secrets policy I have seen written down says the same sensible things. Do not commit credentials. Do not paste them in chat. Do not email them. Every organisation I have worked in has done all three, regularly, including the people who wrote the policy.

The instinct is to treat that as a discipline problem and respond with training. I think it is almost always a design problem. People take the shortest path to getting their work done. If the secure path takes twenty minutes and a ticket, and pasting a value into a direct message takes ten seconds, the direct message wins. Not because anyone is careless, but because one route works and the other does not.

The question worth asking is not "how do we stop people doing this" but "what made the wrong way easier".

Where secrets actually leak

Where it leaks	What it means
Pasted into chat	Getting the value the supported way was slower than asking a colleague
Committed in a config file	Local development had no other way to provide it
Baked into a container image	The build needed it and nothing injected it at runtime
Living in someone's shell history	The tool requires a value on the command line
A shared account nobody rotates	Individual access needed approval and the shared one did not

Every row is a workaround for a path that did not exist or was too slow. None of them is solved by a reminder.

1. Measure the secure path in minutes

Before changing anything, time it. From "I need a database password for my new service" to "my code can read it", how long, and how many people are involved?

If that number is measured in days, no amount of policy will hold. Developers will get the value from wherever they can and carry on, and the ones who do it fastest will look like the most effective engineers on the team.

The target is that the supported path is the fastest path available. Not merely acceptable. Fastest. When the secure route is quicker than asking a colleague, the problem mostly solves itself.

What this looked like for me

I timed our own onboarding path once, pretending to be a developer on a new service. Requesting access, waiting for approval, being told which path to use, and finally getting a working value took the better part of two days, spread across three people.

In the same organisation, asking a teammate in a direct message took about four minutes. Nobody needed to be told which one to use. The policy said one thing and the arithmetic said another, and the arithmetic always won.

2. Secrets should reach the process, not the person

The single biggest reduction in leaks comes from a change in shape: a developer should never hold the value at all.

If the runtime fetches secrets itself, using the identity of the workload, then there is nothing to paste, nothing to store locally, and nothing to rotate in a person's password manager. The developer references a name in config and never sees the value behind it.

# service.yaml, what the developer writes
env:
  DATABASE_URL:
    from_secret: orders/db/url

They name what they need. The platform resolves it at start-up using the service's own identity. Whether that is a cloud secrets manager, a vault, or something else matters far less than the fact that no human is in the path.

The change that removed most of the problem

We moved from handing developers values to injecting them at runtime. The developer's config named the secret; the workload's identity was what authorised the read.

What surprised me was how much it simplified their side. There was no longer any question about where to store the value locally, because there was no value. Most of the awkward conversations we had been having about handling procedures stopped being relevant, not because anyone was more careful, but because there was nothing left to handle.

3. Local development is where it breaks

Production is usually fine. Production has identity, injection, and rotation. It is local development that quietly ruins everything, because on a laptop none of that machinery exists and the developer needs something that works right now.

That is where the committed .env file comes from. Not from carelessness, but from a real need with no supported answer.

The options that work, roughly in order of preference:

Do not need the secret. Run against local fakes and a local database with a throwaway password. Most development does not require production credentials.
Short-lived personal credentials. The developer authenticates as themselves and the tooling fetches a value that expires within hours.
A dedicated development secret, clearly separated, low privilege, rotated on a schedule.

What does not work is asking people to keep a long-lived production value on a laptop and remember not to commit it.

The file everyone had

Almost every service repository had an example environment file, and almost every developer had a real one sitting beside it, ignored by git and sometimes not. It was not a rule anyone had broken; it was the only way to run the thing.

Giving the command-line tool the ability to fetch short-lived values for the developer's own identity is what actually emptied those files. The instruction to stop keeping them had been in place for a long time and had changed nothing.

4. Rotation has to be invisible

A rotation policy that requires developers to do something will not be followed past the first quarter. Any secret that a human must remember to change is a secret that does not get changed.

Rotation works when it is a property of the system: the platform issues short-lived credentials, or rotates long-lived ones on a schedule and the application picks up the new value without a redeploy. The developer's involvement should be zero, and the evidence of rotation should be a log line, not a ticket.

The dotted line back is the part most setups skip. If picking up a rotated value requires a restart that someone has to schedule, rotation stops being routine and becomes a small project, which means it stops happening.

5. Make the audit trail useful to the developer

Access logging is usually built for auditors and invisible to everyone else. That is a missed opportunity, because the same data answers questions developers genuinely have: which services read this secret, when was it last used, is anything still depending on the old one.

When the audit view is useful day to day, people look at it, and things get noticed. A secret nobody has read in six months is a secret that can be removed. A service reading a credential it should not need is worth a conversation. None of that surfaces if the log exists only for a yearly review.

The credential nobody was using

When we made access history visible in the same place developers looked at their service configuration, the first useful thing it produced was a list of secrets with no reads at all. Several belonged to services that had been decommissioned; the credentials were still valid and still granting access to things.

Nobody had been ignoring a cleanup process. There had never been a way to see the information. Putting it where people already looked was the whole fix.

6. Detection is the backstop, not the plan

Scanning for committed secrets is necessary and it is not a strategy. By the time a scanner fires, the value is in git history, possibly pushed, possibly public, and the response is a rotation and an awkward afternoon.

Use it, absolutely: a pre-commit hook that catches the value before it is committed is far kinder than a pipeline that catches it afterwards, and both are better than nothing. But treat every hit as a signal about the path, not just an incident to close. Someone needed a value and had nowhere good to put it. Ask what they were trying to do, and fix that.

Takeaways

Time the supported path. If it is slower than asking a colleague, expect people to ask a colleague.
Keep the value away from people. Inject at runtime against the workload's identity so there is nothing to hold.
Solve local development explicitly. Fakes first, then short-lived personal credentials. Never a long-lived production value on a laptop.
Rotate without asking anyone, including the pickup of the new value.
Show access history to developers, not only to auditors. Unused secrets surface themselves.
Treat every scanner hit as a design report, not only an incident.

The part I have not resolved is third-party credentials. Plenty of vendor integrations still hand you a long-lived key, offer no programmatic rotation, and expect a human to copy it from a web console once a year. Everything above assumes the credential can be issued and rotated by a system, and for that category it simply cannot.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/17/secrets-management-nobody-routes-around/.

Observability for developers, not just for on-call

vivek Itp — Tue, 15 Sep 2026 16:16:16 +0000

Almost every dashboard I have built or inherited was designed for an operator. Cluster health, node pressure, error rates across the estate, a wall of graphs that makes sense if you are holding the pager and looking for the one thing that is on fire.

That is a real audience with real needs. It is just not the audience that uses observability most often. The developer who merged a change twenty minutes ago has a much narrower question, asks it far more frequently, and is usually the person least equipped to answer it with the tools we gave them.

Two people, two questions

The operator asks	The developer asks
What is broken right now, across everything?	Did the thing I just shipped break anything?
Which service is causing this?	Is my service healthy, and was it healthy before?
Do I need to page someone?	Can I go to lunch?
Is this worse than an hour ago?	Is this worse than before my change?

The right-hand column is answerable with a fraction of the data in the left. It is also asked ten or twenty times more often. Most platforms optimise entirely for the left column and leave the right to whoever can write a query.

1. The first question is always about the deploy

Right after a release, a developer wants to know one thing: is this worse than it was before I pressed merge? Not the absolute error rate. The comparison.

A dashboard showing a line at two percent errors is useless on its own. Was it two percent this morning? Was it zero? The developer usually does not know, and finding out means changing a time range, which means learning the tool, which means most of them do not bother and instead wait to see if anyone complains.

The fix is to make the deploy the unit of observation. Mark releases on the graphs, default the window to span the last one, and put the before-and-after difference in words, not just in pixels.

What this looked like for me

We had good dashboards and almost nobody outside the platform team opened them. When I asked why, the answer was consistent: people did not know what normal looked like, so a graph told them nothing.

Adding deploy markers to the service graphs changed that more than any new metric did. Suddenly the question was not "is two percent bad" but "did it change at that vertical line". People could answer that without knowing anything about the system.

2. Scope to the service, not the cluster

A developer owns a service. They do not own the cluster, cannot act on node pressure, and should not have to scroll past it.

Every service should have a dashboard that covers only that service and can be reached without choosing anything from a dropdown. Traffic, errors, latency, saturation, restarts, recent deploys. Nothing about the neighbours.

This sounds obvious and is surprisingly rare, usually because the platform team built one excellent dashboard with a service selector at the top. A selector is a small thing for the person who built it and a real barrier for someone opening the tool for the second time this quarter.

The dropdown nobody used

Our main service dashboard had a variable at the top to pick a service. To us it was elegant: one dashboard, every service. To everyone else it was a page that showed the wrong data by default and had to be configured before it meant anything.

We generated a per-service dashboard from the same definition instead. Same graphs, same code, no selection step. Usage went up sharply and the questions we got about it changed from "how do I use this" to actual questions about the data.

3. Link from where they already are

Nobody navigates to an observability tool. They arrive at it from somewhere else, usually in a hurry, usually from a pull request, a deploy notification, or an alert.

Every one of those places should carry a direct link to the right view of the right service, already scoped and time-ranged. A developer should never have to know the tool's URL, let alone its query language, to answer the question they had.

The links are not a nice touch. They are most of the product. An observability platform that has to be found is an observability platform that does not get used by anyone who is not already fluent in it.

The link in the deploy message

For a long time our deploy notification said which version had gone out and nothing else. Anyone who wanted to check on it had to open the tool, find their service, and set a time range.

Adding two links to that message, one to the service view scoped to the deploy and one to that version's logs, was probably an afternoon of work. It did more for how often developers looked at their own telemetry than the previous year of dashboard improvements.

4. Logs must be findable without a query language

Query languages are a wall. They are learnable, and the people who learn them get enormous value, and most developers will not learn one to check on a deploy they are already fairly confident about.

The default path to logs should be a link, filtered to the service and the version, sorted newest first, with no syntax involved. The query language stays available for the people who want it and the incidents that need it. It just cannot be the entry point.

The test is simple: can a developer who has never opened the tool get to their service's recent errors in one click from the deploy notification? If not, logs are effectively a platform-team-only feature.

5. Tell the owner, not just the pager

Alerts usually route to whoever is on call. That is correct for anything user-facing and urgent. It is wrong for the large category of things that are the owning team's problem and nobody else's: a rising error rate on one endpoint, a queue growing slowly, a job that has started failing overnight.

Those should reach the team that owns the service, in their own channel, during their own hours, without waking anyone. Teams that get told about their own service's problems start fixing them before they become incidents. Teams that never hear anything assume everything is fine, because from where they sit it is.

The alerts that went to the wrong people

Everything alertable pointed at the on-call rotation, which meant the platform team saw every application-level problem first and had to work out whose it was and who to tell. We were a routing layer made of humans.

Splitting the alerts into "wakes someone up" and "tells the owning team in their channel" removed a lot of that. The second category was much larger than the first. Most of it had never needed a platform engineer at all; it had just never had anywhere else to go.

6. Do not make them learn your cardinality problems

There is a strong temptation to teach developers the internals: why they cannot label a metric with a user ID, what a histogram bucket is, why their query is slow. That knowledge is genuinely useful and it is not their job.

The platform should provide instrumentation that is hard to misuse, sensible defaults, and a clear error when someone does something expensive. Not a training course. Every hour a product developer spends learning the observability stack's failure modes is an hour not spent on the thing they were hired for, and they will forget most of it before they need it again.

Takeaways

Make the deploy the unit of observation. Mark releases and default to the window around the last one.
One dashboard per service, no selector. Generate them; do not ask people to configure anything.
Link from the pull request, the deploy message, and the alert. Nobody navigates to the tool.
Logs in one click, no query language for the default path.
Route non-urgent alerts to the owning team, not to the pager.
Do not teach developers your storage constraints. Make the safe thing the default.

What I still get wrong is knowing when a developer-facing view has become too simple to be honest. A single green tick after a deploy is exactly what people want and it hides a lot. I have not found the line between reassuring and misleading, and I suspect it moves depending on how much the team already trusts the platform.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/15/observability-for-developers/.

Golden paths that people actually take

vivek Itp — Fri, 11 Sep 2026 18:06:57 +0000

Every platform team I have been on has built a golden path: a template, a paved road, a "blessed" way to build and ship a service. Every one of them was well used for a few months. The interesting question is never whether people start on the path. It is whether they are still on it a year later, and what made them leave.

I have watched paths get abandoned quietly, one service at a time, while the adoption slide still said one hundred percent. This post is about why that happens and what I have found actually keeps people on a path without forcing them.

Why people leave the path

Nobody leaves a golden path because they enjoy maintaining their own pipeline. They leave because, at some specific moment, staying on the path was more work than stepping off it. If you can find those moments, you can fix them.

The moment	What the developer said	What it actually signals
Needed something the template did not support	"I just added a step to my copy"	The path was too narrow
Template got an update	"I'll upgrade later"	Upgrading was a migration, not a click
Something broke inside the template	"I could not see what it was doing"	The path was opaque
A new joiner copied an old service	"That is what the last person did"	The path was not the obvious starting point
Platform team was slow to respond	"We could not wait"	The path had one gatekeeper

Each row is a design decision the platform team made, or failed to make. None of them is the developer's fault.

1. A golden path is a product, not a policy

The fastest way to get everyone onto a path is to mandate it. The fastest way to get everyone quietly off it is the same mandate. A policy gets compliance on the day it is checked and workarounds on every other day.

A path that people take voluntarily has to be better than the alternative for the person using it, on the day they use it. That is a product question: who is this for, what do they get, and why would they choose it over copying last month's service?

What this looked like for me

The first path I helped build was rolled out with a mandate: all new services use the template, no exceptions. Adoption was immediate and the dashboard looked great.

About six months in, I looked at what the services actually contained. Most of them had started from the template and then diverged: an extra step here, a swapped-out base image there, a pipeline job commented out because it was slow. They were on the path in name only. The mandate had been met on day one and ignored from day two, because nothing about the path made staying on it easier than leaving.

2. It has to win on day one and on day one hundred

A template is easy to make attractive for a brand new service. Four lines of config and you are deployed. That is day one, and most paths are designed entirely around it.

Day one hundred is different. The service has a queue consumer now, a scheduled job, a second database, a customer-specific quirk. The question on day one hundred is whether the path still fits, and whether the platform team has shipped anything in the meantime that the service can pick up without effort.

Paths that only win on day one lose on day one hundred, and day one hundred is where every service lives most of its life.

The template with no upgrade story

Our first template was a repository you copied. Copying it was trivial. Updating it was impossible, because once copied, each service owned its own copy and nothing linked it back.

When we shipped an improvement to the template, exactly one service got it: the next new one. Everything already running stayed on whatever version it had started with. Within a year the template had a dozen versions in the wild and no way to tell which service was on which. The path did not have a day one hundred at all.

3. Escape hatches keep people on the path

The instinct is to make the path strict, so that nobody can deviate. In practice a strict path with no exit means the first unsupported need pushes the whole service off it. Now the developer is not deviating in one place. They are maintaining everything themselves.

A better design has a clear, supported way to step off the path for one thing while staying on it for everything else. Add a custom step here. Override this one value. Ship a raw manifest alongside the generated ones. The escape hatch is small, visible, and counted.

Counting matters more than allowing. Every use of the escape hatch is a signal that the path is missing something. If ten services use the same hatch for the same reason, that reason belongs in the path.

The hatch that became a roadmap

When we added an escape hatch to the deploy template, I expected it to be abused. It mostly was not. What it did instead was give us a list.

Within a few months the hatch usage clustered around three needs: a sidecar for a particular internal agent, a cron-style job, and a different health check shape. All three became first-class template features, and the services using the hatch for them moved back onto the standard path without being asked. The hatch had not weakened the path. It had told us what to build next.

4. Upgrades must be a pull request, not a project

If moving from template version 3 to version 4 requires reading a migration guide, the service will stay on version 3. The only upgrade mechanism that works at scale is one where the platform team does the work and the service team reviews it.

That means the template is not a thing you copy. It is a dependency you reference, with a version. Upgrading is a pull request, opened automatically, with a diff small enough to read and a pipeline run that proves it works. The service team merges it or asks a question. Either way, the effort is minutes.

The second branch is the important one. When the upgrade breaks a service, the fix goes into the template so that it does not break the next one. The service team never has to become experts in the thing they are upgrading.

From copied repo to referenced version

Moving off the copied-repository model was the single change that made the path survivable. Services referenced a versioned template, and a bot opened a pull request in each one whenever a new version was tagged.

The first few rounds were rough, because the template had never been upgraded in place before and every hidden assumption surfaced at once. But the fixes went into the template, and after a few releases most upgrade pull requests merged without a comment. The dozen-versions-in-the-wild problem stopped growing, then started shrinking.

5. Measure drift, not adoption

Adoption is the easy number and the wrong one. A service that started on the path and has been diverging for a year still counts as adopted.

The number that tells you whether the path is working is drift: how far each service is from the current template. Versions behind, escape hatches in use, files that differ from the generated ones. If drift is low and stable, the path is winning. If drift climbs, people are leaving, whatever the adoption number says.

Drift is also actionable in a way adoption is not. A service three versions behind is a conversation. Twenty services all using the same override is a feature request.

The dashboard that replaced the adoption slide

We replaced the adoption percentage with a drift view: every service, which template version it was on, and which escape hatches it used. It was a less flattering picture and a far more useful one.

It changed what the platform team worked on. Instead of building new features and hoping people would take them, we looked at the services with the highest drift and asked why. The answers were usually specific and usually fixable.

6. Talk to the people who left

The services that stepped off the path are the best source of information about it, and they are the ones the platform team is least likely to talk to. Leaving feels like a rejection, so both sides avoid the conversation.

It should be the opposite. A service that left had a reason, and the reason is almost always something the path should have handled. Ask, without judgement, and fix the reason. Then make it easy to come back.

The one missing feature

One team left the path entirely over a single need: their service required a specific network policy the template could not express. Rather than use the escape hatch, which did not cover it, they rebuilt their pipeline from scratch and maintained it themselves for months.

When we finally asked, the fix took a week. We added the option to the template, they moved back, and two other services that had quietly worked around the same limitation moved back with them. Nobody had raised it because nobody thought the platform team would want to hear it.

Takeaways

Treat the path as a product. People stay on it because it is better, not because it is required.
Design for day one hundred, not just the first deploy. Most of a service's life is maintenance.
Build a small, counted escape hatch. Its usage is your roadmap.
Make upgrades automatic pull requests. If upgrading is a project, nobody upgrades.
Measure drift, not adoption. Adoption hides the problem. Drift points at it.
Ask the people who left why. Then fix it and make returning easy.

The part I have not figured out is what to do with the services that will never come back: the old ones with no active team, too different to upgrade and too important to ignore. A golden path is for the services that are still moving, and I do not yet have a good answer for the ones that are not.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/11/golden-paths-that-people-actually-take/.

Verifying commit signatures in the CI pipeline

vivek Itp — Fri, 11 Sep 2026 15:09:04 +0000

Most teams that "require signed commits" have ticked a box in branch protection and stopped there. That box checks that a signature exists. It does not check that the key belongs to someone you trust, and it does nothing once the code is past the merge button.

I worked on an application where that was not good enough. The codebase did cryptographic operations, the security policy was zero trust, and the requirement was blunt: every line of code reaching production has to be traceable to a verified person, and every artifact has to be traceable to a verified build. This post is how I made the pipeline enforce that, and the parts that hurt.

What "signed" actually proves

A commit signature proves that whoever held a particular private key signed that exact commit content. That is all. It does not prove who holds the key, and it does not prove the key is still trusted today.

Check	Branch protection alone	Pipeline verification
A signature is present on each commit	Yes	Yes
The signature is cryptographically valid	Yes	Yes
The key belongs to a known, identity-verified person	Only if the key is registered with the hosting platform	Yes, against your own key registry
Every commit in a promotion, not just the tip, is verified	No	Yes
The build artifact is tied to a verified build	No	Yes, if the build signs its output

The right-hand column is what zero trust actually asks for. The left-hand column is what most teams settle for.

1. Pick SSH signing, and own the key registry

We used SSH keys for signing rather than GPG. Every developer already had an SSH key, the tooling is built into git, and there was no separate keyring to teach people.

git config --global gpg.format ssh
git config --global user.signingkey ~/.ssh/id_ed25519.pub
git config --global commit.gpgsign true

The important decision was not the key type. It was where the public keys lived. We did not rely on the keys uploaded to the git hosting platform. We kept our own registry: each developer's verified public key stored in a parameter store, under their identity, managed by the platform team.

That registry is the source of truth. The pipeline reads from it, not from the hosting platform. If a key is not in the parameter store, the commit is not trusted, regardless of what the web UI says.

Why the registry had to be ours

The git platform will happily show a "Verified" badge next to a commit if the key is attached to any account. That says nothing about whether we had checked who owned the key. For a codebase handling crypto operations, "someone uploaded this key to their profile" was not an acceptable identity check.

Holding the keys in our own parameter store meant we decided what "trusted" meant, and the pipeline enforced our definition rather than the platform's.

2. Verify in the pipeline, not only at the branch

Branch protection stayed on. It is a cheap first gate and it rejects unsigned pushes before they waste anyone's time. But the real check ran inside the GitLab runner.

On every merge request pipeline, the job did the following:

List the commits new to that merge request.
For each commit, read the author identity.
Fetch that person's public key from the parameter store.
Build an allowed_signers file from the keys, and run git verify-commit against it.
Fail the job on the first commit that does not verify.

# Build the allowed-signers file from the registry
for author in $(git log --format='%ae' "$BASE..$HEAD" | sort -u); do
  key=$(fetch_public_key_from_parameter_store "$author") || exit 1
  echo "$author $key" >> allowed_signers
done

git config gpg.ssh.allowedSignersFile allowed_signers

# Verify every new commit, not just the tip
for sha in $(git rev-list "$BASE..$HEAD"); do
  git verify-commit "$sha" || { echo "Unverified commit: $sha"; exit 1; }
done

The BASE..HEAD range is the whole point. Checking only the tip commit lets an unsigned commit ride in underneath a signed one. Every commit in the range has to verify, or the job fails.

Two ranges, two gates

On merge request pipelines, the range was "commits new to this MR". That kept the check fast and gave developers a clear signal on their own work.

On promotion to staging, the range was different: every commit between the previous merge to staging and the current one. That second pass was the one that mattered for the audit trail. It meant nothing could reach staging that had not been verified against the registry, even if it had somehow slipped through an earlier gate.

3. Sign what you build, not just what you commit

Verifying commits proves where the source came from. It says nothing about whether the artifact in the repository manager was built from that source by a trusted process.

So the build signed its output too. When the GitLab runner published the JAR to Nexus, it signed the artifact with a KMS key that only the runner's role could use. Deployments verified that signature before installing anything.

With both halves in place, the chain was continuous: a verified person wrote the code, a verified pipeline built it, and the thing running in production could be traced back through both.

4. Everything that can break, will

I would like to say the rollout was smooth. It was not. Every failure mode you can think of happened.

Rebases and squash merges rewrote commits and dropped signatures. The rewritten commit was now unsigned and failed the check.
Automation commits from bots and pipelines had no key at all.
Key rotation meant a developer's older commits verified against a key that was no longer their current one.
New laptops meant a developer signing with a key nobody had registered yet.

The fixes were mostly boring, which is the good kind:

When a rewrite dropped signatures, the developer re-signed and force-pushed their branch. Force push had to be allowed on feature branches for this to work, which took some convincing.
During key rotation, the registry held both the old and the new key for the same person. Old commits verified against the old key, new ones against the new key, and the old key was removed once nothing depended on it.

The force-push argument

The instinct on the security side was to forbid force pushes everywhere. But a signature check that rejects rewritten commits, combined with a policy that forbids rewriting them again, leaves a developer with no way out. Their only option is to open a new branch and cherry-pick, which is worse for everyone.

Allowing force push on feature branches, while keeping it locked on protected branches, was the compromise. The signature check on the protected branch was the real control. The feature branch was a workspace.

5. Roll it out with a cutoff date, not a flag day

Turning signature verification on for a repository with years of history would have failed on every old commit at once. Instead we did two things.

First, we verified every developer's key before enforcement. Each person's public key was checked against their identity, the way you would verify a customer before opening an account, and only then was it written to the parameter store.

Second, we set an enforcement date. The pipeline checked signatures only on commits made after that date and ignored everything before it. That gave people a defined window to get their keys registered and their signing configured, and it meant history did not need rewriting.

CUTOFF=$(date -d "2025-01-01" +%s)
for sha in $(git rev-list "$BASE..$HEAD"); do
  ts=$(git show -s --format=%ct "$sha")
  if [ "$ts" -ge "$CUTOFF" ]; then
    git verify-commit "$sha" || exit 1
  fi
done

The grace period did most of the work

Announcing a date and then enforcing it on that date, with nothing retroactive, meant almost nobody was surprised. The people who had not set up signing found out on their first push after the cutoff, with a clear failure, and fixed it in a few minutes. Without the cutoff, the same rollout would have meant either rewriting history or exempting so much that the check meant nothing.

What I would still change

The rule I would keep is the one about owning the registry. It is the difference between "signed" and "signed by someone we checked".

The part I have not solved is maintenance. Keys expire, people change laptops, people leave. Every one of those is a manual update to the parameter store today. That is fine at a small scale and a liability at a large one. What this needs is automation: key registration tied to identity provisioning, rotation that updates the registry without a ticket, and revocation that happens the moment someone's access is removed. I have not built that yet.

Takeaways

A "Verified" badge is not identity verification. Keep your own registry of public keys and decide for yourself what trusted means.
Verify in the pipeline, over the full commit range. Branch protection is a first gate, not the control.
Sign the artifact too. Source provenance without build provenance is half a chain.
Expect rewrites, bots, rotation, and new machines to break it. Allow re-signing on feature branches and hold old and new keys during rotation.
Enforce from a date, not from the beginning of time. Verify keys first, then set a cutoff.

The open question is still key maintenance. Until registration, rotation, and revocation are automated, the trust model is only as current as the last person who remembered to update the parameter store.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/11/verifying-commit-signatures-in-the-ci-pipeline/.

Error messages are part of the platform

vivek Itp — Wed, 09 Sep 2026 18:47:15 +0000

Nobody reads a platform's documentation on a good day. On a good day the pipeline is green, the deploy went out, and the developer never thinks about the platform at all. The platform only gets read when something breaks, and what gets read is the error message.

That makes the error message the most-read text a platform team ever writes. It is also, almost always, the text nobody on the platform team wrote on purpose. It is whatever the underlying tool printed, passed through untouched, with a stack trace attached.

I spent a long time treating errors as something that happened to the platform rather than something the platform produced. This post is what changed my mind, and what I do differently now.

The gap between what it said and what it meant

Here are the kinds of messages developers actually see, next to what the platform team knows they mean. The gap between the two columns is where support requests come from.

What the developer saw	What it actually meant
`ImagePullBackOff`	The image tag you deployed was never pushed. Your build job probably failed.
`exit code 137`	The runner ran out of memory. Not your code.
`Error: forbidden: User cannot list resource "pods"`	You are in the wrong namespace, or you were never granted access to this one.
`KeyError: 'DATABASE_URL'`	The secret for this environment was never created.
`x509: certificate signed by unknown authority`	The internal CA is not installed in your base image.
`Error: no matches for kind "Ingress" in version "extensions/v1beta1"`	The template you copied is three years old.

Every row in the right column is something the platform team could have printed instead. None of them did, because the message came from a tool underneath the platform and nobody caught it on the way up.

1. An error has three jobs

A useful error message answers three questions, in this order:

What broke? In one plain sentence, from the developer's point of view.
Why? The most likely cause, stated as a fact if it is known and as a guess if it is not.
What do I do now? A command, a link, or a name. Something the developer can act on in the next minute.

If a message does only the first job, the developer knows they are stuck. If it does all three, the developer is usually unstuck before they think of asking anyone.

What this looked like for me

The first error I rewrote was the image pull failure. The raw version was ImagePullBackOff with a pod name. Developers would see it, check that their code compiled, re-run the deploy, see it again, and then ask in the channel.

The rewritten version, printed by the deploy step, read something like: "Deploy failed: image tag abc1234 does not exist in the registry. This usually means the build job for this commit failed or has not finished. Check the build job here: link."

The first line was the same information. The second and third lines were the difference between a support request and a self-fix. Questions about that error mostly stopped after the change.

2. Write for the person who did not build the system

Most error messages are written, implicitly, for the person who wrote the code that raised them. They use internal names, assume knowledge of the architecture, and skip context that the author never needed because they already had it in their head.

A developer on a product team does not have that context. They do not know that "reconciler" means the deploy controller, that "upstream" means the service behind the proxy, or that error code 4012 is the one about quotas.

The test I use: would a competent engineer who joined last week and has never read our platform code understand this message? If not, it is not finished.

The message with an internal name in it

We had a validation step that rejected service configs with a message naming the internal component that did the checking. Something like "rejected by policy-gate: rule R7 failed." Everyone on the platform team knew what R7 was. Nobody else did, and there was nothing in the message to search for.

The fix was to make the rule print its own explanation: "Service config rejected: memory must be at most 4Gi for services without a capacity exception. Yours is 8Gi. To request an exception, see: link." Same rule, same check. The developer just got to read the reason instead of the rule number.

3. Put the fix in the message

The most valuable thing an error can contain is the next command to run. Not a description of the fix. The fix.

A lint failure should print the command that auto-fixes it.
A missing secret should print the command or link that creates it.
A permissions error should name the group that grants the permission and where to request it.
A deprecated config field should print the new field name and the one-line change.

This is also the cheapest place to put documentation, because it is the only documentation that is guaranteed to be read at the moment it is needed.

# Before
Error: validation failed for field 'healthcheck'

# After
Error: 'healthcheck' must be a path starting with '/', got 'healthz'.
Fix:   change it to '/healthz' in service.yaml.
Docs:  https://internal/docs/service-config#healthcheck

The link that replaced a wiki page

For a while the most-linked page in our support channel was a wiki article explaining how to request access to a new environment. Someone would hit a permissions error, ask, and get the link. Dozens of times.

We put the link in the error message. The wiki page did not change. The number of people who needed to be told about it dropped close to zero, because the tool told them at the exact moment they needed it.

4. Lead with the plain sentence, keep the raw detail

The instinct when wrapping an error is to hide the original. That is a mistake. The original error is often the only thing that helps when the plain-language guess is wrong, and someone on the platform team will eventually need it.

The structure that works: plain sentence first, cause and action next, raw detail last and clearly labelled.

The developer reads the top. The platform engineer, if it gets that far, reads the bottom. Nobody has to scroll through a stack trace to find out that a tag was missing.

The wrapper that hid too much

My first attempt at friendlier errors over-corrected. The deploy tool caught every failure and printed a short summary, and dropped the original output entirely. It was clean until the summary was wrong. Then the developer had a confident sentence pointing at the wrong cause and no way to see what had actually happened.

We put the raw output back, below a separator, under a heading that said it was for debugging. The summaries stayed. The dead ends went away.

5. Treat the most common errors as bugs

Every platform has a small set of errors that account for most of the confusion. You can find them without any tooling: read a month of the support channel and count. The same five or six messages will come up over and over.

Each of those is a bug in the platform, even if the underlying tool is behaving correctly. The bug is that the platform let a confusing message reach a developer. Put them on the backlog, in priority order by how often they appear, and fix them the way you would fix any other bug.

The list on the wall

At one point I kept a short list of the errors that generated the most repeat questions, ordered by count. It had maybe eight entries. We worked down it during quiet weeks, one rewritten message at a time.

It was some of the highest-return work the platform team did that year, and almost none of it involved changing what the platform actually did. It only changed what it said.

6. Test error messages like features

If a message matters, it deserves a test. Not a test that the error is raised, which most codebases already have, but a test that the message says what it should: names the field, includes the fix, links to the right place.

This sounds excessive until the first time someone refactors the validation code and the helpful message quietly reverts to a generic one. Nobody notices, because errors are not on the happy path, until the support channel fills up again.

Takeaways

The error message is the platform's real interface. It is read more than any docs page, at the exact moment it matters.
Three jobs: what broke, why, what to do next. A message that does one is not finished.
Write for someone who joined last week and has never seen the platform code.
Put the fix in the message. A command or a link beats a description every time.
Plain sentence on top, raw error on the bottom. Never hide the original.
Count the repeat errors and fix them as bugs. It is cheap, and it compounds.

What I still have not worked out is errors from tools I do not own. When a managed cloud service or a third-party CLI returns something cryptic, the platform can wrap it, but the wrapping is a guess, and the guess goes stale every time the vendor changes their wording.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/09/error-messages-are-part-of-the-platform/.

Self-service on a slide vs self-service in practice

vivek Itp — Wed, 09 Sep 2026 18:42:05 +0000

Every internal platform I have seen described in a slide deck was self-service. Developers would "spin up a new service in minutes" with "no tickets" and "no waiting on the platform team". I have never seen one of those slides that was accurate on the day it was presented.

That is not because the people writing the slides were lying. It is because self-service is a property of the whole path a developer walks, and the slide only describes the part the platform team built. The parts they did not build, the access request, the DNS entry, the secret that has to be created by someone else, the approval that lives in a different tool, are invisible from the platform side and completely visible from the developer's.

This post is about measuring that gap honestly, and about what actually closes it.

The only definition that matters

Self-service means a developer with a normal amount of access can go from "I need a new service" to "it is running in production and I can see it" without asking a human for anything.

Not "without asking the platform team". Without asking anyone. The moment there is a ticket, a Slack message, or a form that someone else has to act on, the path is not self-service, no matter how good the tooling is on either side of that gap.

That definition is strict on purpose. A path that is self-service except for one step is not ninety percent self-service. It is blocked at that step, and every developer will wait there.

What the path usually looks like

Here is the shape of the path I have seen most often, regardless of company or tooling. The platform team owns the middle. The friction lives at the edges.

Step	Who does it	Self-service?
Create the repository from a template	Developer	Yes
Get the pipeline running	Developer, using the shared template	Yes
Get a namespace or environment	Platform team, via ticket	No
Get a database or queue	Another team, via a different ticket	No
Get secrets into the environment	Security or platform, via request	No
Get a DNS entry and certificate	Network team, via email	No
Get production deploy permission	Manager approval, via form	No
First production deploy	Developer	Yes

The three "yes" rows are what the slide describes. The five "no" rows are what the developer remembers.

What this looked like for me

The first time I actually walked the path myself, start to finish, as if I were a developer on a product team, it took the better part of a week. Not because any single step was slow. Because each step was a request to a different person, and each person had a queue.

The platform tooling in the middle was genuinely good. Repository template, pipeline, deploy, all worked first time. I still spent most of the week waiting. Nobody on the platform team had ever counted the waiting, because none of it happened in our tools.

1. Count tickets, not features

The most useful number a platform team can track is not how many features the platform has. It is how many times a developer has to ask another human for something on the way to a first production deploy.

I call it the ticket count, even though half of those requests are never actual tickets. A Slack message asking for a namespace is a ticket. An email to the network team is a ticket. An approval form is a ticket. If a human has to act before the developer can continue, it counts.

Then track the second number: how long each of those requests waits. Not how long it takes to fulfil, which is usually minutes. How long it waits in someone's queue before anyone looks at it, which is usually days.

The number that changed the conversation

When I first put the ticket count in front of the people who owned the platform roadmap, it was somewhere around six human touch points for a new service, with a median wait of a day or two each. That was the first time the roadmap conversation shifted from "what feature should we add" to "which of these requests can we make disappear".

It also surfaced something uncomfortable: the platform team was not the bottleneck. We were the fastest queue in the chain. The slow ones belonged to teams that had never been told they were part of the developer experience at all.

2. Most tickets exist because of a default nobody set

When you look at why each request exists, a pattern shows up. Most of them are not there for safety. They are there because nobody decided what the default should be, so a human decides it every time.

A namespace request exists because nobody decided that every repository gets a namespace automatically.
A resource quota request exists because nobody decided what a reasonable starting quota is.
A database request exists because nobody decided that a service can provision its own small database within limits.
A production access request exists because nobody decided that the team that owns a service can deploy it.

Each of these is a policy question that a human answers by hand, dozens of times, usually the same way. Writing the answer down once and letting the tooling apply it is the whole job.

The request that was always approved

One of the requests in our chain was for a new namespace with a default quota. When I looked at the history, every single one had been approved, almost always with the same quota. The approval step existed because at some point someone had worried about cluster capacity, and the worry had turned into a permanent human gate.

We replaced it with a rule: every repository created from the template gets a namespace and a standard quota at creation time, with a documented way to ask for more. The approval disappeared, the ticket disappeared, and the cluster did not run out of capacity. The worry had been real. The gate had never been the right answer to it.

3. Pre-approve the safe path, and only the safe path

The objection to removing gates is always the same: what stops someone doing something dangerous? The answer is not a human. It is a narrow path that is safe by construction, with the gates kept only for leaving that path.

In practice that means:

The template produces a service that is safe to deploy as-is: no public exposure, conservative resources, no elevated permissions.
Everything on that path is pre-approved. Nobody reviews it, because the review already happened when the template was written.
Anything off the path, a public endpoint, a privileged container, a larger quota, still needs a human. But that human is now reviewing exceptions, not routine.

This is the part that makes security and platform teams comfortable with removing gates. They are not removing review. They are moving it from every request to the template, where it happens once and applies everywhere.

Where the pre-approval broke down

The thing I got wrong the first time was making the safe path too narrow. The template covered a stateless HTTP service and nothing else. Anything with a queue consumer, a scheduled job, or a database needed the exception route, which meant most real services went through the exception route, which meant we had rebuilt the ticket queue with extra steps.

The fix was widening the path until it covered what most teams actually built, and treating each new exception request as a signal that the path was still too narrow. The exception route is supposed to be rare. If it is not, the template is wrong, not the developers.

4. Own the whole path, or at least measure it

The hardest part is that most of the tickets belong to teams the platform team does not control. Networking, security, database operations, a change board. Telling those teams to remove their gates does not work. Showing them where they sit in the developer's week sometimes does.

What I have seen work:

Publish the path as a single diagram, with every human touch point marked and the median wait for each. Put it somewhere visible.
Ask each owning team one question: what would need to be true for this step to be automatic? Usually the answer is a policy nobody has written down.
Offer to build the automation for them. Most teams are not against self-service. They do not have time to build it.

Each green node was, at one point, a ticket to a different team. None of them stopped being someone's responsibility. They stopped needing a human in the loop for the routine case.

The team that was never asked

The slowest step on our path was a DNS and certificate request that went to a networking team by email. From the platform side it looked like an immovable external dependency. When I finally talked to that team, they had no idea their queue was on the critical path for every new service. They had automation for internal zones already. It just was not connected to anything developers could reach.

Connecting it took a couple of weeks and one conversation. The step went from a multi-day wait to a couple of minutes, and the platform team did not build most of it.

5. Measure it the way a developer feels it

If the goal is honest self-service, the metric has to be something a developer would recognise. The ones I keep coming back to:

Time from repository creation to first production deploy. Wall clock, not effort. This is the number that captures the waiting.
Human touch points on that path. The ticket count from above. Target: zero for the standard path.
Exception rate. What fraction of new services need the off-path route. If it is high, the path is too narrow.
Support requests per new service. How many times the developer had to ask a question that the tooling should have answered.

None of these need a new tool. They need someone to walk the path occasionally, as a developer would, and write down what happened.

Takeaways

Self-service means no human in the loop, not just no platform engineer in the loop. One gate blocks the whole path.
Count the tickets, including the ones that are not tickets. Slack messages, emails, and approval forms all count.
Most gates are undecided defaults. Decide the default once, encode it, and the gate disappears.
Pre-approve the safe path and keep review for exceptions. Then widen the path until exceptions are rare.
The slow steps usually belong to someone else. Show them the path. Offer to build the automation.

What I still do not have a good answer for is the last step on most paths: the production approval that exists for compliance reasons rather than technical ones. I have made it faster and made it clearer, but I have not yet made it go away, and I am not sure it should.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/07/self-service-on-a-slide-vs-in-practice/.