DEV Community: vivek Itp

The first question to ask a red pipeline

vivek Itp — Sun, 08 Nov 2026 04:34:12 +0000

A green pipeline contains nothing worth reading. Nobody opens the log of a passing build, and they are right not to. All of the value in a pipeline's output is in the failures, and almost none of that output is arranged to help.

So this is about the failures, and specifically about one question worth asking before you read anything: did my code fail, or did the thing running my code fail? Those are different problems with different owners, and the fastest way to waste an afternoon is to start debugging one while looking at the other.

To be clear about where the difficulty comes from: build logs are hard to read because they are the raw output of a dozen tools that were never asked to produce something a person would read. That is a property of the logs, not of the people reading them. Platform engineers get quick at it only by having seen a few thousand of them, which is not a skill anyone should have to acquire by attrition.

1. Two kinds of red, and they are not interchangeable

	Code failure	System failure
What happened	Your change did not do what it should	The environment running your change gave up
Example	A failing assertion, a type error, a lint finding	A runner out of memory, a registry timeout, a full disk
Who can fix it	You	The platform team
Useful next step	Read the error, reproduce locally	Report it, with the job link

Telling these apart first is not a formality. If it is a code failure, everything you need is in the log and the fix is yours. If it is a system failure, there is nothing in your diff to find, and the most productive thing you can do is hand it to whoever owns the runners — with the job link, which saves them the only tedious part.

The failure that taught me to write this down

For a long stretch the most common message in our platform channel was some version of "my tests are failing with exit code 137 and I did not change anything."

Exit code 137 means the process was killed, nearly always because it ran out of memory. The tests were fine. The test runner was being killed mid-suite because the job shared a small machine with a build step that ate the available RAM.

What strikes me now is that every person who asked was doing the right thing: they read the log, found nothing about their code in it, and asked someone. The cost was that they each spent twenty minutes first, checking a change that was never the problem, because nothing in the output said this is not about your code.

2. The signals, in the order they are quickest to check

Four checks, roughly in ascending order of effort. Most failures are settled by the first two.

Which step failed? Before reading any output, look at where it stopped. A failure in checkout, dependency install, container pull or image build — anything before your code is executed — is a system failure almost by definition. Your code had not run yet.

What is the exit code? A few are diagnostic on their own. 137 is a process killed, usually for memory. 143 is a termination signal, often a timeout or a cancelled job. Anything about "no space left on device", DNS, connection refused or a registry name is infrastructure, whatever step it appeared in.

Did your change touch the thing that broke? A failure in a module your diff does not reach is worth a second look before you accept it as yours. It can still be yours — a shared dependency, a changed interface — but the question is cheap to ask.

Does it reproduce? Run the same target locally. If it fails locally too, it is yours and you now have a fast loop. If it passes locally and fails only in the pipeline, the gap between the two environments is the actual bug, and that gap is usually the platform's.

Why 'which step' comes first

The reason I put the step before the output is that the output is long and the step name is one word. Checking it costs a second and resolves a large share of failures on its own.

It is also the check people skip, because a wall of red text pulls your eye into reading it. The habit worth building is to look at where it stopped before reading what it said.

3. Read the log from the end, but do not stop at the end

The last lines of a failing job are often the least useful: a runner cleaning up, a summary count, a non-zero exit being reported by something three layers removed from the cause.

What you want is the first line that is a real error rather than a consequence of one. Everything after the original failure tends to be wreckage — cascading errors, teardown problems, a test runner reporting that it could not write a report for a suite that already died.

So: start at the bottom to confirm it failed and how, then search upward for the earliest thing that looks like a root cause. In a long log, searching for the first occurrence of an error marker gets you closer than scrolling ever will.

The shape of a log that wastes your time

The pattern I see most often is a job whose final two hundred lines are all consequence. A dependency failed to resolve early, every subsequent step failed because the dependency was missing, and the visible end of the log is the last of those downstream failures.

Read from the bottom and you are looking at the fifth symptom. The actual line you need is far above, and it is usually short and unremarkable — which is exactly why it does not catch the eye.

4. Re-running is a test, not a remedy

Clicking re-run is reasonable when you have a reason: you believe it is a system failure and want to know whether it is transient. That is a hypothesis being tested.

What is worth avoiding is re-running as the first move, before looking at anything, because that is where two costs come from. It delays the fix on a code failure by the length of a pipeline run. And on a genuine system failure it destroys the evidence — the next run may land on a different machine, and the thing the platform team needed to see is gone.

If you are going to re-run a job you think is broken infrastructure, it is worth copying the job link somewhere first. That link is the difference between a report someone can act on and a conversation that starts with "it was red earlier".

What makes a report immediately actionable

From the other side of this, the reports I can act on straight away have three things: the job link, what step failed, and whether it has happened more than once. That is thirty seconds of effort and it skips an entire round trip.

The ones that take longest are not badly written — they are usually just "the pipeline is broken", sent by someone being considerate about not writing a long message. The irony is that the short version costs them more time than the detailed one would have.

5. Most of this should not be the developer's job

Everything above is a workaround. A developer should not need a mental decision tree to find out whether a failure is theirs, and the fact that this post can exist is a criticism of the tooling rather than a gap in anyone's knowledge.

What the platform side can do, roughly in order of how much it helps:

Label the failure class in the job itself. If a wrapper can recognise exit code 137, it can print one line saying the job was killed for memory, that this is a runner problem rather than a code problem, and where to report it.
Name jobs after what they check, so the step name alone carries information: unit-tests rather than stage-3.
Put the first real error at the top of the output, or in the merge request annotation, so the log is a reference rather than a search problem.
Make the failure class visible across runs, so a recurring system failure is a number somebody owns rather than a run of individual support messages.

The cheap fix that worked better than the real one

With the memory failure, the real fix was giving test jobs their own memory limit, and we did that. But the change that stopped the support requests arrived the same day and was much smaller: a wrapper step that caught exit code 137 and printed one line — this job was killed for exceeding memory. This is a runner problem, not your code. See link.

Nobody had needed the fix in order to carry on with their day. They had needed to know it was not theirs.

Takeaways

Ask which kind of failure it is before reading the log. Code failure, yours; system failure, the platform's. The answer usually takes seconds.
Check which step failed before reading its output. Anything that broke before your code ran is not about your code.
Learn a handful of exit codes. 137 is out of memory, 143 is a termination signal, and anything naming disk, DNS or a registry is infrastructure.
Find the first real error, not the last one. The end of a failing log is usually wreckage.
Re-run to test a hypothesis, not as a reflex — and save the job link before you do, or the evidence goes with it.
The tooling should answer this, not the reader. Every category a wrapper can recognise is a category nobody has to learn.

What I have not solved is where the labels should live. Printing them in the log helps the person reading it and does nothing for the person who never opens it. Putting them in the merge request helps more people and means the pipeline has to know something about the code review system. I have built neither properly, which is why this post is still a reading guide rather than a description of something automatic.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/11/08/the-first-question-to-ask-a-red-pipeline/.

The most boring file in our application

vivek Itp — Sun, 01 Nov 2026 14:04:17 +0000

A user three time zones away loads a page and it feels quick. Nothing about that is an accident, and almost none of it is the application's doing. The part the application team contributes is a single static file that does nothing, and getting that file wrong is the easiest way to switch the whole thing off without noticing.

The feature is Akamai SureRoute. It tests several network paths between the edge and the origin and sends real traffic down the one that performed best. This post is the application side of it, because that is the side I keep having to explain.

1. The internet does not route for speed

Traffic between two points on the internet takes a path chosen by routing protocols, and those protocols optimise for reachability and policy, not for how fast your page feels. The default path works. It is frequently not the quickest one available.

SureRoute's answer is to stop assuming. A request for content that cannot be served from cache triggers a race: the edge fetches a small test object from the origin over three candidate routes at once, and the winner becomes the primary route until the next race. Results are held for a while — around half an hour by default — then re-raced.

Note what that excludes. Cacheable content is already served from the edge, so it never races and does not need to. This feature only ever helps the requests that have to reach your origin, which in a modern application is the API calls and the personalised responses — exactly the ones a user notices waiting for.

Why I stopped explaining this as a CDN feature

When I describe it as "the CDN makes your site faster", application teams hear caching, and reasonably conclude it is somebody else's problem.

Describing it as route selection for the requests that cannot be cached lands differently, because those are the requests the team already gets complaints about. The page shell was never the slow part. The call behind the spinner was.

2. The whole of the application team's job is one static file

Here is the entire contribution, and it is genuinely this small: serve a static test object the edge can fetch to run its races.

Requirement	Why
Returns HTTP 200	A race with a failed fetch measures nothing
Static, no back-end processing	The measurement is of the network path, not your application
No authentication	The edge fetches it without credentials
Content type `text/html`, a few KB of plain text	It needs to resemble a real uncacheable response in size
Same origin hostname as the real traffic	A path to a different host is a measurement of the wrong path

That last row is the one that catches people. The object has to sit behind the same hostname as the traffic being optimised, because otherwise you have measured a route you are not using. Where several origins sit behind a load balancer, the balancer can serve the object itself.

The file that broke the measurement by being helpful

The failure I would expect — and the reason I now review this file specifically — is someone making it useful. A test object that queries a database, or returns a dynamically generated timestamp, or checks downstream dependencies, is no longer measuring the network. It is measuring your application and attributing the result to a route.

Worse, it fails in the direction that looks like success: every route now appears slow and roughly equal, so the race picks something close to arbitrary and the feature quietly stops helping. Nothing is red. There is no alert for "your route optimisation has become a coin toss".

3. Boring is the specification, not a shortcut

This is the part that sits badly with application engineers, and I understand why. We spend our working lives being told endpoints should validate, authenticate, log and report. Here the requirement is the opposite: return the same bytes immediately, every time, to anyone who asks, and do nothing else.

It is worth being precise about why that is safe. The object is a few kilobytes of static text with no information in it. It exposes no data because it contains none. The thing that would make it dangerous — making it reveal something about the system — is also the thing that would make it useless.

The review comment I expect to write forever

Any suggestion to add logging, a version string, build information or a dependency check to this file gets the same answer: every one of those makes the measurement worse and the file more interesting to an attacker, which is the wrong direction on both counts.

The one change I would accept is making it the right size. It should resemble a real uncacheable response, because a race against a trivially small file does not exercise the path the way real traffic will.

4. It is not your health check

The two get conflated constantly, including in how this was first described to me, and they are different objects with incompatible requirements.

A health check is supposed to be opinionated: it should check the database, verify dependencies and fail when the instance is unfit to serve. A race test object must do none of that. Pointing route optimisation at a health check endpoint gives you a measurement polluted by everything that endpoint touches.

Two files, named so nobody merges them

The thing I would insist on is that these are two separate files with names that make their purpose obvious, rather than one endpoint serving both duties.

They will drift otherwise. Someone adds a dependency check to the health endpoint, correctly, for health-checking reasons — and if that endpoint is also the race target, they have degraded a performance feature while improving an operational one, with no way to know they did it.

5. What it costs

Being straight about this, because it does have a cost: the races put extra requests on the origin. They arrive looking like ordinary traffic, over the same ports with the same certificates, and they recur.

For a static file served without processing that is a small cost, and it is the price of the feature. But it means two things for the application team. The test object has to be cheap to serve, because it will be served more often than you expect. And if you are counting requests to the origin for capacity or for billing, some of them are races rather than users.

Where I would look if origin request counts looked wrong

If someone asked me why origin request volume did not match user analytics, race traffic is on the short list of explanations — along with health checks, uptime monitoring and crawlers.

None of those are users, all of them hit the origin, and a team that does not know they exist will spend an afternoon looking for the missing people.

Takeaways

Route optimisation only helps uncacheable requests. Cacheable content comes from the edge already. That makes this an API-latency feature, not a page-weight one.
The application team's contribution is one static file, served over the same hostname as the real traffic.
It must be dull by specification: 200, static, unauthenticated, no processing, a realistic few kilobytes. Every improvement makes it worse.
It is not a health check, and sharing one endpoint for both guarantees one of them gets degraded by a reasonable change to the other.
Races add origin traffic. Keep the file cheap, and remember that some requests in your origin counts were never people.

What I do not have is a way to tell whether it is working. A good race and a broken one produce the same colour of nothing: no error, no alert, traffic flowing. You can infer it from latency over time, which is slow and confounded by everything else that changes. A check that fetches the test object and asserts it is still static, still unauthenticated and still the right size would catch the common regression, and I have not built it.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/11/01/the-most-boring-file-in-our-application/.

No inbound ports, no peering: putting an edge in front of our servers

vivek Itp — Sun, 25 Oct 2026 14:38:50 +0000

The usual way to put an application on the internet is to open a port. You put a load balancer in front, you write an inbound rule, and from then on anything on the internet can at least knock on the door. Every control after that is about deciding who gets in.

We went a different way: a connector on the host that dials out to an edge network, and not one inbound rule anywhere. The thing I did not expect is how much it let us delete — not just firewall rules, but the cloud provider's own peering between accounts and its managed DNS service as well.

1. The connection goes the other way

The mechanism is simple enough that it took me a while to believe it. A lightweight daemon runs on the host and opens an outbound connection to the edge network. Requests from the internet arrive at the edge first, and the edge sends them back down the connection that is already open.

Because the connection was established from the inside, nothing needs to listen for it. There is no inbound listener on the origin at all.

The connector keeps several connections open to different edge locations rather than one, so losing a location is not an outage. The transport is QUIC over UDP where it can be, falling back to HTTP/2 over TCP on the same port when UDP is unavailable.

	Port opened on our side	What can reach the host
Load balancer and inbound rule	Inbound, from the internet	Anything that can route to the address
Outbound connector	None	Only what the edge sends down an existing connection

Reading the security group afterwards

The moment it clicked was looking at the instance's inbound rules after the cutover and finding nothing in them. Not a narrow rule, not one restricted to the edge's address ranges. An empty list.

That is a different statement from "we only allow traffic from these addresses". A narrow rule is still a door, and the discipline of keeping it narrow — reviewing it, auditing it, catching the one someone widened for an afternoon and forgot — exists because the door exists. Remove the door and the discipline goes with it, which is worth more than the rule it replaced.

2. Outbound to the edge is the whole requirement

What we do allow is outbound, to the edge network's connector endpoints. Publicly, that is one port for the tunnel transport itself — both TCP and UDP, because of the QUIC-and-fallback behaviour — and optionally HTTPS for things like update checks and token validation, which the connector runs without if they are blocked.

That is a short list, and it is the kind of rule an outbound policy can be specific about: a named destination and a named port, rather than a range opened because something somewhere needs it.

Why this is an easier conversation with security

An inbound rule is permission for the world to attempt something. An outbound rule is permission for one of our processes to reach one named place. Those are not the same risk and they do not need the same review.

The second one I can describe in a sentence and defend without a diagram, which matters more than it should when a control has to be explained to people who will not read the diagram.

3. Access decisions move to the edge, and that is the point

With no inbound path, the question "who may reach this site" stops being a network question. It becomes a policy question at the edge, in front of the application, where it can be expressed in terms of people and groups rather than addresses.

That is what made this worth doing rather than just clever. Address-based access control has always been a proxy for what we actually mean: we do not care what network someone is on, we care who they are. Moving the decision to the edge lets the rule say the thing we mean.

The consequence worth stating plainly: traffic arriving this way does not pass the host's own firewall rules on the way in, because it is not arriving from the network side. If your access control lives in those rules, it has to move to the edge as part of this change rather than after it.

What we stopped having to model

The cross-account case is where this paid off most. Reaching a service in another account used to mean the provider's own peering: connections between networks, route tables, non-overlapping address ranges, and a change in two accounts every time something moved.

With connectors on both sides and policy at the edge, the networks never have to meet. Each dials out, and the access decision is made in one place rather than expressed as the intersection of two sets of routes.

I would not claim peering is obsolete. I would say a surprising amount of what we used it for was reachability obtainable another way, with less to keep in step.

4. It replaced the managed DNS service too

This one surprised me. Once the edge is in front of everything, the authoritative DNS may as well live there: a hostname points at the tunnel, the edge resolves it and routes it. We moved off the cloud provider's managed DNS entirely.

Two properties are worth knowing before you do the same:

The record and the tunnel are independent. You can publish a hostname pointing at a tunnel that is not running, and visitors get an error rather than a failure that looks like DNS. That is clearer than it sounds — the failure names the tunnel rather than sending you to look at resolution.
A tunnel's identifier is not a credential. Routing to it only works from within the same account, so knowing the identifier does not let someone else send traffic to your origin.

The consolidation nobody asked for

We did not set out to change DNS. It came up because after the cutover the managed DNS service was doing nothing except pointing at something that no longer existed, and keeping it meant keeping two places where a hostname could be defined.

Collapsing it was less about cost than about the number of systems that have to agree for a hostname to work. That number went from two to one, and the class of outage where they disagree went away with it.

5. What it saves, in resources and in effort

Two kinds of saving, and they are not the same kind of argument.

The first is resources that stop existing. Each is a billed thing in a native cloud design, and some are billed per environment, so the count multiplies quietly as you add a staging and a pre-production:

Native cloud design	With an outbound connector
Peering relationships between accounts, with route tables and address-range planning	Not required — each side dials out
A separate managed DNS service, and its queries	Collapsed into the edge
Inbound rules on every entry point, carried through every review	None to write, review or re-narrow

Worth being clear about what does not go away. A load balancer is still in the picture, terminating TLS in front of the application, and the host still needs its outbound path. This is not a design that deletes your networking bill. It removes a specific category of it — the things that exist to make inbound reachability safe — and leaves the rest alone.

So the honest summary of the money is: it saves some, mostly by retiring things that were there to manage reachability rather than to serve traffic. The larger saving is time, which nobody bills you for:

No inbound rules to review, widen by accident, or forget to narrow again.
No route tables or address-range coordination to keep in step across accounts.
One place where a hostname is defined, so no class of outage where two systems disagree about it.
One place where access policy is expressed, in terms of identity rather than addresses.
Host replacement does not involve a network change: the new host dials out the way the old one did.

That last one is the quiet win. Replacing an instance used to touch something in the network — a target registration, a rule, an address. Now it touches the connector and nothing else.

Where the effort saving actually shows up

The resource list is the part that gets into a business case, and it is the less interesting half. The per-environment multiplication is real — a peering relationship is a thing you need one of per environment, and the count grows without anyone deciding it should — but it is a number somebody can argue with.

The time saving is harder to put in a slide and easier to feel. It is the review that does not have to happen, the two-account change that is now a one-account change, the address-range conversation nobody needs to have. None of that appears on an invoice, and all of it was coming out of the same engineers' weeks.

There is a second-order one I did not expect: the things that go away are the things most likely to be wrong. A stale inbound rule, a route that no longer matches the address range, a hostname defined in two places — those were a steady trickle of small incidents, and removing the mechanism removed the category.

The costing here is structural rather than measured: I am describing which line items and which recurring tasks stop existing, not a figure. Anyone making this case should count their own, because the balance depends entirely on how many environments and accounts they are running.

6. What you take on in exchange

This section is the part I would want to read if someone else had written this post.

A dependency at the edge. The edge network is now in the path of everything. That was true of a load balancer too, but this is a different vendor with a different failure mode, and the inbound path you would otherwise fall back to is gone by design. The answer is failover at the edge rather than a reopened door, which means a second location to route through — and that belongs in the plan from the start, not after the first time you need it.

Access control in one place means one place to get wrong. Everything that used to be distributed across network rules now lives in policy at the edge. That is better, and it concentrates the consequence of a mistake.

Outbound UDP has to actually work. If the transport port is blocked for UDP, the connector falls back to TCP and keeps working, which is good — and means a network restriction can quietly cost you performance rather than announce itself as an outage. Worth checking which transport you are actually on rather than assuming.

The host firewall is no longer in the request path. Said above, worth repeating, because it is the mistake I would expect someone to make: leaving access rules on the host and believing they still apply to traffic arriving through the tunnel.

Takeaways

A connector that dials out removes the door, rather than narrowing it. There is no inbound listener to protect.
The requirement on your side is one outbound destination and port, which is a much easier control to describe and defend than an inbound rule.
Access control becomes a policy about people, not addresses — and it has to move to the edge as part of the change, not after it.
Cross-account reachability may not need peering. A lot of what peering gets used for is reachability obtainable with less to keep in step.
Count the saving in two columns. Some money, from retiring the things that existed to manage inbound reachability — peering, a separate DNS service, the inbound rules themselves — and rather more time, which nobody bills you for. Be clear about what stays: a balancer still terminates TLS, and the outbound path still costs what it costs.
Expect the DNS consolidation. Once the edge is in front, a separate managed DNS service is one more place a hostname can be defined.
You are trading a set of network controls for a dependency on the edge. Make that a decision, not a side effect, and design the edge failover alongside it rather than afterwards.

The question I am working on now is failover at the edge itself. Removing the inbound path was deliberate, so the answer cannot be a door kept closed-but-present — that is precisely what this design exists to avoid. The direction I am taking is a second edge location to route through, so that the fallback is another path to the same closed origin rather than a way around it. Designing that properly, and proving it fails over the way you expect rather than the way the diagram suggests, is not something I would want to find out about under pressure.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/10/25/no-inbound-ports-no-peering/.

What the platform should not do

vivek Itp — Sun, 18 Oct 2026 13:03:53 +0000

Most writing about platform engineering is about what to build. Very little of it is about what to refuse, which is strange, because refusing is most of the job.

Every platform team I have worked on has had a remit larger than its headcount. That is not a staffing problem to be fixed later. It is the permanent condition, and the teams that do well are the ones that treat deciding what not to do as real work rather than as an admission of failure.

1. The remit is always bigger than the team

The arithmetic does not work and it never will. A platform team supports every other team, so its potential work grows with the organisation while its headcount grows with whatever budget argument was won last year.

What follows from that is uncomfortable: if you do not choose what to drop, the choice gets made for you, by whoever asks loudest or most recently. That is the worst possible prioritisation function, because volume correlates with frustration rather than with value.

The alternative is not a better backlog tool. It is being willing to say, out loud and in public, that something is not going to be done.

What queue-driven prioritisation looks like from the inside

This section is an argument rather than a specific recollection. The failure mode is not dramatic. Every individual request is reasonable. Each one takes a day or two. Nobody is being unreasonable and nothing is being mismanaged.

Six months later the team has shipped nothing it planned, every engineer can list four things they are halfway through, and the thing everyone agreed was the most valuable work has not been started.

No single decision caused that. The absence of decisions caused it.

2. Three questions before taking something on

What I use, roughly in this order:

Question	If no
Will more than one team use it?	It belongs to the team that wants it
Is it harder to explain than to build?	Write the documentation instead
Would several teams otherwise build it badly?	Let them build it, and watch

The third one is the only argument I find genuinely convincing for centralising something. Not efficiency, and not consistency for its own sake — those are arguments platform teams like, and application teams correctly hear as "we want control". The real argument is that some things are hard enough to get wrong that five independent attempts will produce five subtly broken versions, and the cost of that lands on everyone later.

Signing is like that. Firewall rules are like that. A deployment script for one team's service is not.

The one that is genuinely ours

Artifact signing is the clearest case I have. It is not that application teams could not implement signing. It is that the thing being protected — a key only the build can use — stops meaning anything the moment it is implemented five different ways with five different sets of assumptions about who can reach it.

That is a control where one correct implementation is worth more than five reasonable ones, and that is the test. If the value of a thing comes from everybody doing it the same way, it belongs to the platform. If the value is local, it belongs to the team.

3. Taking something on is forever

The part that gets underestimated is not the build. It is that every piece of platform surface area is a permanent claim on the team's future capacity: support, upgrades, the week it breaks, the migration when the underlying thing changes.

So the question is never "can we build this". It is "are we prepared to own this in three years, with the team we will have then". A yes given casually in a planning meeting is a commitment nobody revisits, and the team that inherits it will not know why it was made.

This is also why "we'll build it and hand it over" almost never works. Handing over a system that nobody else asked to own is a slow way of keeping it.

Keeping the guarded list short on purpose

The clearest version of this for me is the set of projects whose pipeline configuration we control centrally, where the team cannot change its own runner tags or remove the signing and verification steps.

That list is deliberately short, and it stays short because every project on it is a project that waits on us for changes other teams make themselves. Adding one is not a config change. It is taking on a permanent obligation to be responsive for that team, forever.

Every addition is a conversation rather than a default, and that is not bureaucracy. It is the only thing keeping the list at a size where we can actually be responsive to the projects on it.

4. A no needs a reason the other person can use

Most platform noes are technically correct and socially useless. "That's not on our roadmap" tells someone nothing except that they should ask again in a different way, or go around you.

A useful no has three parts: what you are not going to do, why, and what the person should do instead. The third part is the one that gets skipped, and it is the one that decides whether the team walks away with a path forward or with a grievance.

"We're not building that" — a dead end.
"We're not building that, because it would only be used by your team. Here's the template to build it yourselves, and we'll review it." — a path.

The second takes ten minutes longer and it is the difference between a platform team that is trusted and one that is routed around.

The noes that come back

Also a pattern rather than a specific incident. A no that is not understood does not end the request. It relocates it: to a different channel, to a manager, to a workaround built quietly that you find out about a year later when it breaks.

The ones that stay answered are the ones where the person left the conversation able to explain the reasoning to someone else. If they cannot repeat your reason, you have not said no — you have just declined, and the request is still live.

5. Self-service is how you say no at scale

Everything above is about individual requests. The structural version is self-service: the request you never receive because the person did it themselves.

This is the only mechanism I know that actually changes the arithmetic in section 1. Every other approach — better triage, more rigorous prioritisation, clearer intake forms — manages the queue. Self-service removes items from it permanently.

The catch is that self-service is itself platform surface area, and it is subject to the same rule: you own it forever. A self-service tool that is half-built generates more support load than the manual process it replaced, because now people need help with the thing and with the tool.

Where moving a control creates a queue

A prediction from the firewall work rather than something I have measured. When a control moves off the application server and into infrastructure the platform team owns, the security position improves and a new queue appears: teams now wait on us for changes they used to make themselves.

If that queue is not addressed, the control becomes the thing people design around. The answer is not to give the access back. It is to make the common change self-service — reviewable, auditable, and not gated on a conversation with us.

That work is the actual cost of moving the control, and it is usually not in the estimate.

6. What I get wrong

I am better at the first half of this than the second. Deciding what not to take on, I can do. Saying it in a way that leaves the other team with a path, I get wrong often enough that I notice.

The specific failure is that a no feels final to me and provisional to them. I say it once, consider it settled, and move on. They hear a position that might soften if raised again next quarter. Both of us leave thinking the conversation ended.

Takeaways

The remit is always bigger than the team. If you do not choose what to drop, volume and recency choose for you.
Centralise what is dangerous to get wrong in parallel, not what is merely repeated. Consistency alone is a weak argument and teams hear it as control.
Every yes is permanent. Ask whether you will own it in three years with the team you will have then.
A no needs a reason the person can repeat to someone else, and an alternative. Without those it is not answered, just relocated.
Self-service is the only thing that changes the arithmetic — and it is platform surface area too, so half-building it costs more than not starting.

What I have not worked out is how to make these decisions visible. They happen in conversations and in the heads of whoever has been on the team longest, which means they are inconsistent across people and invisible to everyone outside it. A platform with a clear scope that nobody can read is not much better than one without a scope at all.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/10/18/what-the-platform-should-not-do/.

Moving the firewall off the application server

vivek Itp — Sun, 04 Oct 2026 13:13:48 +0000

For a long time the firewall in front of one of our applications was CSF, running on the EC2 instance itself. It worked. Changing a rule meant connecting to the box and running a command, and that sentence is the entire reason we moved away from it.

The replacement is AWS Network Firewall, sitting outside the application server, with the instances moved into a restricted subnet behind it and a new network load balancer in front. The security argument for that is easy. The part worth writing about is what it changes for the people who used to be able to fix things themselves.

1. A firewall on the box is only as strong as access to the box

CSF's problem was not its rules. It was where it lived.

A host firewall is configured by someone with a shell on the host. That means the set of people who can change the firewall is the set of people who can log in, and those two sets should not be the same. It also means the configuration lives in a place nothing else reads: not in a repository, not in a template, not in anything you can diff. The current state of the rules is whatever the last person to run a command left behind.

	Firewall on the instance	Firewall outside it
Who can change a rule	Anyone with shell access	Anyone who can change the firewall resource
Where the rules live	On the box	In infrastructure configuration
How you audit them	Log in and look	Read the resource
If the host is compromised	The attacker can edit the firewall	The firewall is unaffected

The last row is the one that ends the argument. A control that an attacker on the host can turn off is not protecting you from an attacker on the host.

The question I could not answer well

What finally moved this was being asked to produce the current rule set for review. The honest answer was that I could tell them what the rules were right now, by going and looking, but not who had last changed them or why.

There was no history. CSF's configuration was state on an instance, and instances get replaced. Every rebuild carried the rules forward through whatever mechanism had put them there in the first place, and that mechanism was never quite the same twice.

2. The new shape: restricted subnet, firewall outside, load balancer in front

The topology changed more than the tooling did. Network Firewall was already in the account, used alongside EC2. The move was to put the application instances into a restricted subnet behind the firewall, and to front the whole thing with a new NLB.

Three things follow from that picture:

The instance stops being the boundary. It sits in a subnet that cannot be reached except through the firewall, so the host no longer needs to defend itself.
The rules become a resource. They are described, versioned and reviewed like any other piece of infrastructure, rather than typed into a terminal.
There is a new component in the path. The NLB is one more thing that can be misconfigured, and one more place to look when traffic does not arrive.

That last point is a real cost. We replaced a simple path with a longer one, and a longer path has more places to be wrong.

What the new piece bought us

The NLB was not there to balance load. With the instances inside a restricted subnet, something has to terminate the connection on the outside and hand it on, and that something needs to be a resource we can point the firewall at.

It also gave us a single place where traffic enters, which is what made the geo rules meaningful. Before, an instance with a public address had its own front door. Several instances meant several front doors and several copies of the same rules, kept in step by hand.

3. "Allow this country" is not what a geo rule should say

The thing people assume about geo-IP filtering is that you pick countries and allow them. That is a very blunt instrument and it would not have passed review here.

Our rules are scoped to a specific region on a specific port. Not "traffic from this country", but "traffic from this region, to this port". Everything else is denied, and specific addresses get their own allow entries where a known source needs to reach something the geo rule does not cover.

The reason to be narrow is that geography is a weak signal. An allow rule on a country is enormous: it admits every compromised machine in that country, and it still fails to admit a legitimate user who happens to be travelling. Pinning it to a port at least means the rule only opens what the application actually serves, so a broad source range cannot be used to go looking for something else.

Where the specific-IP entries come from

The geo rule covers ordinary traffic. The individual address entries are for the known, named sources — the systems that have to reach the application and do not fit the pattern.

Those entries are the part that needs discipline. Each one is a small exception, and a list of small exceptions is how an allow-list quietly becomes an allow-all. The rule I would hold to is that every address entry needs to name what it is for, and anything that cannot be named should not be there.

4. Run both at once, and resist the urge to cut over

We did not switch. For the migration, CSF and Network Firewall both ran, both enforcing, with the host firewall still in place while traffic moved to the new path.

Running two firewalls in parallel is awkward on purpose. It means either one can block traffic, and it means that when something is refused you have two places to check. That is the cost, and it buys the one thing that matters: you are never in a state where the old control is gone and the new one has not been proved.

The sequence that worked:

Build the new path — subnet, firewall, load balancer — with rules intended to match what CSF already allowed.
Move traffic to the new path with CSF still enforcing on the host.
Watch for anything the new rules deny that the old ones allowed.
Only once that has been quiet for a while, take the host firewall out.

Step 3 is where the real work is. Everything else is configuration.

Why the overlap is worth the confusion

The temptation is always to do it in one change, because two firewalls is a confusing state to be in and nobody enjoys explaining it.

But a cutover means the first time your new rules face real traffic is also the moment you have nothing to fall back to. Any gap between what CSF allowed and what you think it allowed shows up as an outage rather than as a denied-traffic log entry you can go and read.

Keeping both on turns that class of mistake from an incident into a finding.

5. Who can change a rule now, and how long it takes

This section is the pattern I would argue for rather than a process I am reporting. This is the part that costs the application team something, and it is worth being honest about rather than presenting the move as pure gain.

Before: someone with access to the instance ran a command. Minutes, no ticket, no review — and no record either.

After: the rule is a change to infrastructure owned by whoever owns the firewall. That is slower by construction, and if the application team does not own that infrastructure, they now wait on someone else for something they used to do themselves.

The trap is treating the delay as acceptable because the control is better. Developers do not experience a security improvement; they experience the thing that used to take five minutes now taking a day. If a routine rule change needs a conversation, people start designing around the firewall instead of asking it for what they need, and you end up with traffic routed somewhere easier rather than somewhere correct.

What I would aim for: the application team can propose a rule change as a reviewable change, it goes through the same pipeline as any other infrastructure, and the review is about whether the rule is right rather than about whether the request is allowed to exist.

6. What to expect to break

Also a prediction rather than a report. Two failure modes are worth watching for, and neither of them looks like a firewall problem when it happens.

Traffic that worked by accident. Something was reaching the application on a path nobody documented, because the host firewall allowed it or because the instance had a public address. The new path denies it, and the first symptom is a system that stopped working for reasons unrelated to any change in itself.
A source that does not fit the geo rule. A legitimate caller outside the allowed region, or on a port the rule does not cover, which previously got in through a host rule that someone added once and never wrote down.

Both are the same underlying problem: the old rule set was never fully known, so "match what CSF allowed" is an approximation. The parallel run is what turns those into log entries you read rather than outages you explain, which is the whole argument for section 4.

Takeaways

A firewall on the host can be edited by anyone who reaches the host, including an attacker. That is the reason to move it, not tidiness.
Move the instances behind the boundary, so the host does not have to defend itself. Accept that the path gets longer and has more places to be wrong.
Scope geo rules to a region and a port, not a country. Keep specific-IP exceptions named and few.
Run both firewalls during the migration. It is confusing and it turns a class of outage into a log entry.
Watch the cost to the application team. A rule change that used to take minutes and now takes a day is a developer experience problem, whatever it did for the security posture.

What I am still unsure about is whether we have found the right place for the rules to be reviewed. Putting them in infrastructure configuration fixed the audit problem completely, and it moved the delay onto the people who need a rule changed. Those are not the same trade, and I do not think we have finished making it.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/10/04/firewall-off-the-application-server/.

What an artifact signature proves, and what it does not

vivek Itp — Wed, 30 Sep 2026 04:02:21 +0000

I wrote earlier about verifying commit signatures in the pipeline, and mentioned in passing that the build signs its own output. That sentence deserves a post of its own, because signing an artifact is easy and the claims people make about it are usually larger than what it actually supports.

What we have works. It is also narrower than the phrase "supply chain security" suggests, and the honest version is more useful to anyone building the same thing than a diagram with green ticks on it.

1. Sign the JAR and the image, because only one of them gets deployed

We sign both the JAR and the container image. For a while it was only the JAR, and that was the wrong place to stop.

The JAR is what the build produces. The image is what actually runs. Sign only the JAR and you have proof about an input to a later step and nothing about the thing that reaches the cluster, so anyone who can influence how the image is assembled sits in a gap the signature does not cover.

What you sign	What that lets you check	What is still open
The JAR only	The library came from a trusted build	Nothing about the image that wraps it
The image only	What runs was built by a trusted pipeline	Nothing about the code inside it
Both	Each handoff is checkable on its own	Whether the build itself did what it claimed

Why the second signature got added

The first version signed the JAR at publish time and stopped there. On paper the chain looked finished: signed commits, signed artifact, verification before deploy.

The question that broke it came from outside the platform team: the thing running in production is an image, so what have you proved about the image? The answer was nothing. The image was built in a separate job, from a base image we pulled and a JAR we trusted, and no signature covered the result. Signing the image closed that step — not the whole chain, as the rest of this post gets into, but it moved the boundary somewhere defensible.

2. Only the build can sign, and that is the whole control

Access to the signing key is restricted to the runner's role. No developer has it, no local build can produce a signed artifact, and there is no break-glass path that puts the key on a laptop.

That restriction is the entire security property. Everything else — the verification step, the metadata, the pipeline configuration — is plumbing that only means something because the key is unreachable from anywhere except the build. The moment a person can sign, a signature stops saying "this came from the pipeline" and starts saying "this came from the pipeline, or from someone who was in a hurry".

So the interesting question is not how do we sign but who can schedule a job on a runner that holds the signing role. That is a CI access question wearing cryptographic clothes, and it is where the real risk lives.

The request I keep having to turn down

The recurring ask is for a way to sign locally, and it is always reasonable in the moment: a release is blocked, the pipeline is red for an unrelated reason, the artifact is fine — can someone just sign it and push it through?

Saying no to that is the job. If a human can sign, every signature in the system becomes "probably from the build", and a signature that means probably is not worth the pipeline minutes it costs. What we do instead is keep the pipeline path fast enough that the question is about waiting rather than being blocked — and when that stops being true, the pressure to build a bypass comes straight back.

3. Verify at the point of use, and fail the deploy

The specifics in this section are the pattern rather than a story I am retelling exactly. Verification happens in the deploy job, before the artifact is pulled down and installed. If the signature does not verify, the deploy fails. There is no warning mode and no override flag.

Two details matter more than where the check sits. Fail closed: a step that logs a warning and continues is documentation, not a control. And say why in the first line: "no signature found" and "signed with an unknown key" are different problems with different fixes, and nobody should need a security runbook to tell them apart.

The failure message worth writing properly

The most common verification failure is not an attack. It is an artifact that predates the control, or one promoted between environments by a path that did not carry the signature with it. With a generic message both produce the same outcome: a red deploy, a developer who did nothing wrong, and a support request.

Both are obvious if the message names the artifact, says whether a signature was absent or unrecognised, and links to the one page explaining what to do next. The check is the easy part; the message decides whether people trust it.

4. A signature is not provenance

Here is the part I would want to read if someone else had written this post.

What we record alongside the artifact is the commit and the signature. That is it. There is no signed attestation describing the build, no record of which runner produced it, no statement of what the pipeline configuration was at the time.

So the chain supports this claim: this artifact was signed by our build, and it says it came from this commit. It does not support the claim people tend to hear, which is: this artifact was built from this commit, by this pipeline, with nothing else added. The commit reference is metadata the build wrote about itself. A signature over self-reported metadata proves the signer, not the statement.

That is not a reason to skip signing. It raises the cost of tampering after the build, which is the most likely place for it to happen. It is a reason to be careful about the sentence you put in a compliance document. "Traceable" is true. "Provable" is not, yet.

The distinction that matters in an audit conversation

The useful move in review is to say plainly which claim the setup supports, rather than showing the diagram and letting people read the stronger one into it.

Traceability is genuinely valuable: given an artifact we can get to a commit, and given a commit to a verified author, and every link is checkable. What we cannot do is prove the build did only what the pipeline file said. Closing that means attesting the build itself, which is a larger piece of work than signing ever was.

5. Key rotation is the failure you will actually hit

Not an attacker. A lost key.

When a developer loses their key — a replaced laptop, usually — they generate a new one, and everything signed with the old key is suddenly signed by something the verifier does not recognise. The signature is still valid; the key it points to is no longer registered.

The fix is to keep old public keys rather than replace them. The parameter store holds retired keys alongside the current one, so historic signatures keep verifying while new work is signed with the new key, and revocation becomes a deliberate act for a key you believe is compromised rather than a side effect of someone getting a new machine.

This is the single most useful thing I would tell anyone rolling out signing of any kind. Design for rotation before you design for enforcement. Enforcement is a day of work. Rotation is what determines whether the control survives contact with normal life.

What it looks like when you have not planned for it

The pattern is always the same. Someone gets a new laptop, generates a fresh key, and their next few pieces of work fail verification. From their side the control is broken: they did nothing unusual and the pipeline is refusing their change. Keeping retired keys in the store fixes it, and it costs nothing except deciding once that key history is state you keep rather than state you overwrite.

What still needs a human is deciding whether a key was lost or taken. Those need opposite responses — keep the old key so history verifies, or revoke it so it stops — and no automation tells you which one you are looking at.

6. The part that is still unsolved: knowing which commits are new

The gap I have not closed is not about signing at all. It is about deciding which commits a pipeline run is supposed to check.

When a branch is pushed for the first time, the "before" reference the hook receives is all zeros — there is no previous state to compare against. You cannot tell from that alone where the new work starts. If the branch was taken from another branch rather than from the mainline, walking back from the tip picks up commits that belong to whoever wrote them, not to the person pushing now.

The consequences run both ways, and both are bad:

Too wide: the check covers commits from before the branch point, and a developer is blocked by history they did not write.
Too narrow: the range misses commits, and unverified work reaches a protected branch while the pipeline reports green.

Comparing against the mainline merge base is the obvious answer and it is not reliable either — a branch taken from a branch has a merge base that is not where the developer thinks it is, and a rebase moves it after the fact.

How we live with it today

We compare against the mainline and accept that a branch taken from another branch is a case we get wrong. When it happens the developer sees a failure they cannot explain, asks in the channel, and we look at the range by hand. Calling that a solution would be generous — it is a known rough edge with a manual escape, which is the honest description of most controls at this stage.

What I think the real fix looks like is recording the verified range rather than recomputing it: storing what has already been checked, so the pipeline asks what is new since the last verification instead of inferring the boundary from the push. I have not built it.

Takeaways

Sign every artifact that gets handed on, not only the first one. The JAR and the image are both handoffs.
Keep the signing key reachable only from the build. That restriction is the security property; everything else is plumbing.
Fail the deploy on a bad signature, and make the message distinguish "no signature" from "unknown key".
Be precise about what you have. A signature over self-reported metadata gives you traceability, not attested provenance.
Plan key rotation first. Keep retired public keys so history still verifies, and treat revocation as a deliberate decision.
Work out how you will identify new commits before you enforce anything on them. It is harder than the signing.

The signature part of this was a week of work. The part I am still circling is knowing exactly which commits a given run is responsible for, and until that is solved the enforcement around it is only as trustworthy as the range it was handed.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/30/what-an-artifact-signature-proves/.

Why our Java builds run on Fargate and our Docker builds do not

vivek Itp — Sun, 27 Sep 2026 13:20:34 +0000

Most discussions about self-hosted CI runners start with cost. Managed minutes are expensive, the argument goes, so run your own and save money. That was not why we did it, and cost is usually the weakest reason on the list.

We run our own GitLab runners because of a requirement we could not meet any other way: builds have to sign the artifacts they produce, with a key the build host is allowed to use and nobody else is. Everything below follows from that one constraint, and from the fact that not every build wants the same kind of machine.

1. The reason to self-host is usually a requirement, not a saving

If your builds are ordinary, managed runners are the right answer. Somebody else patches them, scales them, and gets paged when they break. You should need a reason to give that up.

Ours was artifact signing. The publish step signs the JAR with a KMS key before it reaches the artifact repository, and the deploy step refuses anything that does not verify. That only works if the machine doing the signing is one we control: our account, our IAM role, our network path to the key, and no shared tenancy with builds that should never reach it.

Once you own the compute for that reason, the other arguments start to matter — private network access, predictable machine sizes, caches that survive between jobs. But they are consequences, not the case.

What actually forced the decision

We had already moved to signed commits, verified in the pipeline. The next question from the security side was the obvious one: you have proved who wrote the code, so can you prove which build produced the binary?

Answering that meant the runner had to hold an identity: not a secret pasted into a CI variable, but a role with permission to use one specific key, scoped so only jobs from projects meant to sign could assume it. You cannot do that on shared managed compute in any way I was comfortable defending.

2. Split the compute by what the job builds, not by who owns the job

We run everything from one centralised AWS account, with two kinds of executor behind it: ECS Fargate and EC2. The split is not by team or by environment. It is by what the job actually does.

Job type	Runs on	Why
Java builds, tests, artifact publish	ECS Fargate	Ordinary processes, no privileged access needed, scales to zero
Container image builds	EC2	Needs a real Docker daemon and a disk that behaves like a disk

Fargate is the better default. No host to patch, capacity appears when a job appears, and nothing is running means nothing is billed. For a Maven or Gradle build that is all upside.

Container builds are where it stops working. Building an image wants a daemon, layer storage that behaves like a local disk, and occasionally privileges a serverless container platform is specifically designed not to give you. You can fight that with alternative builders, and we looked at it, but EC2 hosts do this without argument and Fargate does it with a fight.

So Java builds go to Fargate, Docker builds go to EC2, and the developer writes neither of those words. They pick a runner tag and the tag decides.

The rule that removed the question

Before we split it, the recurring question in the platform channel was "which runner should my job use?", and the answers people gave each other were folklore copied from whichever project they had worked on last.

One sentence fixed most of it: if the job builds an image it runs on the EC2 tag, and everything else runs on the Fargate tag. That fits in a template comment, and nobody has to understand executor internals to get it right. The jobs that still get it wrong are the ones that do both, which is usually a sign the job is doing too much anyway.

3. Isolation is a decision you make once, early, and live with

Each executor type runs as two stacks, and each stack has two sets of runners separated by a restricted flag. Restricted runners can reach the signing key and the production-facing artifact paths. Unrestricted ones cannot, and that is enforced by the role attached to the runner, not by anything in the project's pipeline file.

The reason for two sets rather than one is simple: if every runner can assume the signing role, every project in the instance can sign, including the one someone created this morning to try something out. Permission on a runner is permission for whoever can schedule a job onto it.

The thing I would tell anyone starting this: decide the isolation boundary before you have twenty projects on the runners, not after. Adding a project to the restricted set is a five-minute change. Taking access away from projects that have quietly come to depend on it is a quarter of conversations.

Why the flag sits on the runner and not in the pipeline

The first version had the signing step guarded by a check inside the pipeline template: if the project was on the approved list, the step ran. That is a check a project can edit its way around, because the pipeline file lives in the project's own repository. The list was in the right place for convenience and the wrong place for security.

Moving the boundary onto the runner's IAM role changed the failure mode. A project that is not supposed to sign does not fail a policy check. It gets an access-denied from AWS, because the machine it runs on genuinely cannot use the key.

4. For critical projects, the pipeline configuration is not the team's to change

Alongside the tagging, we run what we call guarded CI. For a short list of critical projects, the pipeline configuration is hardcoded: the runner tags, the signing step, and the verification step are fixed, and the project cannot override them from its own file.

This is a deliberate reduction in flexibility and it is unpopular in exactly the way you would expect. The argument for it is that the controls on a critical application should not be removable by editing a YAML file in that application's own repository. If a team can turn off the check that protects them, the check is a suggestion.

The cost is real: guarded projects wait on us for changes other teams make themselves. We keep the list short so that stays survivable.

Where guarding earns its keep

The moment that justified it was mundane. A build was failing on the verification step, and the quickest path to green — the one anybody under pressure reaches for — was to comment the step out and come back to it later.

On a normal project that works, and sometimes nobody notices for a month. On a guarded project it does nothing, because the step does not come from the project's file. The team pinged us instead, and the cause was a stale key reference that took minutes to fix. Being unable to bypass the check is what got the right people looking at it the same day.

5. Caching is where self-hosted runners give the time back

The specifics here are the pattern I would recommend rather than a story I am retelling exactly. A Java build that resolves dependencies from scratch every time spends a meaningful chunk of every run redoing work. Self-hosted runners let you fix that, and it is the clearest developer-visible win available.

Two things matter more than the rest:

A shared dependency cache in S3, keyed on the lockfile, so a build only pays full resolution cost when dependencies actually change.
Warm layer storage on the EC2 hosts for image builds, so a rebuild of an application whose base image has not changed reuses the layers it already has.

The trap is treating the cache as free. A cache that is never invalidated eventually serves something stale and costs you an afternoon, and a cache keyed too loosely is a correctness problem wearing a performance costume. Key it on the lockfile, expire it on a schedule, and make cache hits and misses obvious in the job log.

The shape of the win

The pattern to expect is that the first build after a dependency change pays full price and every build after it takes noticeably less. What makes it worth doing is the common case rather than the average: a small code change on a project whose dependencies have not moved in weeks gets fast, and that is where developers spend almost all their time.

6. The maintenance is the part nobody puts in the estimate

Also a recommendation rather than a specific recollection. Building the runner setup gets estimated. The ongoing cost usually does not, and it is not small:

Patching the EC2 hosts, and rotating them often enough that patching is routine rather than an event.
Keeping the runner version in step with the GitLab instance, because drift produces failures that look like project problems.
Watching the autoscaling behaviour, because a capacity problem arrives at the worst moment and looks like a slow pipeline.
Telling platform failures apart from project failures quickly enough that the distinction is useful.

Budget it as a standing share of somebody's time, not as a project with an end date. If nobody owns it, the runners keep working right up until the day they very loudly do not.

7. What developers actually notice

Not the architecture, and not the isolation model. They notice that a build sometimes takes longer to start than it should, and that the reason is invisible from where they sit.

That is a queueing effect and we have not solved it. Concurrency limits are an awkward trade: set them high and a busy afternoon has several heavy builds fighting over the same hosts, so everything slows at once. Set them low and jobs queue while capacity sits idle, because the limit rather than the machine is the constraint. We are still tuning it.

The mitigation that helped most was not a capacity change. It was making the wait visible: if a job is queued rather than running, a developer should see that without asking, because queued and slow are different problems and only one of them is theirs.

Takeaways

Self-host for a requirement you cannot meet otherwise, not for a saving. Ours was signing artifacts with a key only our build hosts can use.
Split executors by what the job builds. Fargate for ordinary builds that scale to zero, EC2 for image builds that want a real daemon and a real disk.
Put the isolation boundary on the runner's role, not in a pipeline file the project can edit.
Guard the configuration for critical projects, keep that list short, and accept that it costs those teams some autonomy.
Cache deliberately, key on the lockfile, and make hits and misses visible in the log.
Budget maintenance as standing time. Patching, version drift and scaling behaviour do not stop arriving.

What I have not worked out is the concurrency question. Every setting we have tried is a choice about who waits and when, and the right answer probably changes with the time of day in a way a single number cannot express.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/27/why-java-builds-run-on-fargate/.

Measuring developer experience without a survey

vivek Itp — Wed, 23 Sep 2026 22:55:37 +0000

The usual first move when someone decides to take developer experience seriously is to send a survey. It is the obvious thing to do and it is rarely the most useful thing available, because the data you actually need is already sitting in systems you run.

I am not against surveys. They capture things instrumentation cannot: whether people feel trusted, whether they think anyone is listening, whether the platform is something they use or something that happens to them. But as a starting point they have three problems, and all three are avoidable.

They are answered by the people who already engage. They measure sentiment at a moment rather than friction over time. And they take weeks to run, which is long enough for the thing that annoyed everyone to be forgotten or fixed.

The numbers are already there

What to measure	Where it already lives	What a bad number means
Time from push to a merge signal	Your CI system	People context-switch while waiting
Re-run rate on checks	Your CI system	Nobody trusts red any more
Time from repository creation to first production deploy	Repo events and deploy logs	Onboarding a service costs days
Support requests per team per month	Your support channel	The platform cannot answer for itself
Time from merge to production	Deploy logs	Shipping is a decision, not a default
Rollback frequency and time to roll back	Deploy logs	Either quality, or fear of deploying

None of those requires a new vendor. They require someone to go and count, and then to keep counting.

1. Pick numbers that describe a developer's day

The test for a developer experience metric is whether a developer would recognise it as describing their own week. Deployment frequency across the organisation does not. Waiting eleven minutes for a pipeline does.

That rules out most of what gets reported upward. Aggregate throughput numbers are useful to someone, but they average away exactly the thing you are trying to see: that one team waits forty minutes and another waits four, and the average of twenty-two describes nobody.

Keep the numbers per team, or per service, and look at the distribution rather than the mean. The interesting signal is almost always in the tail.

What this looked like for me

The first number I tracked properly was time from push to a merge signal, per repository rather than overall. The organisation-wide figure had looked acceptable and had told me nothing.

Broken out per repository, the picture was completely different. A small number of repositories were far slower than the rest, and they belonged to the teams who complained most. The complaint and the data had been saying the same thing for a while; only one of them had been legible.

2. Re-run rate is the most honest number you have

If I could keep only one, it would be this: how often does someone re-run a check without changing anything.

Nobody re-runs a check they believe. A high re-run rate means the pipeline has stopped carrying information, and that the team has learned to treat red as noise. It is the clearest possible measurement of trust, and it is free, because your CI system already records it.

It is also a leading indicator. Trust erodes before anything visible breaks. By the time a real failure gets re-run three times and merged, the erosion happened months earlier.

What the re-run rate exposed

Counting re-runs turned a vague sense that the pipeline was flaky into a short, ordered list of the specific jobs people did not believe. The same few appeared over and over.

That list was more actionable than any survey response about pipeline confidence would have been, because it named the jobs rather than the feeling.

3. Measure the whole path, not your part of it

A platform team naturally measures the parts it owns, which is exactly how the slow parts stay invisible. The queue you control looks fast because it is fast. The wait that hurts is somewhere between two systems, or in another team's inbox.

Measure end to end, from the developer's first action to the outcome they wanted, and treat every handoff as part of the number. If getting a namespace takes two days in someone else's queue, that belongs in time to first deploy whether or not you own the queue.

The dotted spans are the measurements. Both cross boundaries the platform team does not own, and that is the point: they describe the developer's experience rather than the platform's.

The wait that was not in our numbers

When I walked the new-service path myself and timed it, almost all of the elapsed time was waiting in queues belonging to other teams. Every system I owned was fast, and the developer's experience was still days long.

Our dashboards had been honest about our part and silent about the total, which made them reassuring and useless.

4. Count the questions

Support volume is a developer experience metric and almost nobody treats it as one. Every question is a moment where the platform failed to answer for itself.

You do not need tooling for this. A tally of what gets asked, bucketed roughly and counted, will show you a distribution steep enough to act on within a few weeks. The top few buckets are the platform's real backlog.

Watch the trend rather than the total. A channel that gets quieter each quarter is a platform that is learning. A channel with a stable volume and a fast response time is a platform that has got good at answering the same questions forever.

5. Trends, not benchmarks

The question worth asking is whether a number is better than it was last quarter. The question that wastes time is whether it is better than some other company's number.

Published benchmarks come from organisations with different constraints, different regulatory positions, and different definitions of the same word. Comparing against them produces either false comfort or a target nobody can reach for reasons that have nothing to do with effort.

Your own history is the only honest comparison. It shares every confounding factor, which is exactly what makes it useful.

6. Do not turn them into targets

The moment a number becomes a target, it stops measuring what it measured. This is not a new observation and platform teams fall into it anyway, usually with the best intentions and a quarterly goal attached.

If re-run rate becomes a target, the tempting fix is to remove the flaky tests rather than fix them. If time to first deploy becomes a target, the fix is to redefine when the clock starts. Both improve the number and neither improves anyone's day.

Keep them as instruments. They tell you where to look and whether something you changed helped. They are not a scoreboard, and the moment anyone's performance review touches them, they are finished as measurements.

Where I would put the numbers

Putting the measurements where developers could see them, rather than only in a report going upward, changed how they were received. Teams recognised their own figures and occasionally disputed them, which was useful, because a disputed number gets examined.

The times these numbers were least useful were when they appeared in a summary for people who did not work with the platform daily. At that altitude they stop being diagnostic and start being a score.

Then run the survey

Everything above measures friction. It cannot tell you whether people feel they can get things changed, whether they trust the platform team, or whether the documentation is any good.

So run a survey, but run it second, when you already know where the friction is. The questions get sharper, the responses are easier to interpret, and you can ask about the specific things your numbers flagged rather than asking people to rate the developer experience out of ten.

Takeaways

Start with what you already record. Your CI system and deploy logs hold most of it.
Per team, not aggregate, and look at the distribution. The average describes nobody.
Re-run rate is the single most honest number and it costs nothing.
Measure end to end, including the queues you do not own.
Count support questions and watch the trend, not the total.
Compare against your own history, never a published benchmark.
Never make them targets. They are instruments.
Run the survey second, to ask about the things instrumentation cannot see.

What I still do not know how to measure is the work that never gets attempted. Some fraction of good ideas die because someone correctly judges that the platform would make them painful, and none of those decisions leave a trace in any system I run. Everything above measures the friction people pushed through, not the friction that stopped them.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/23/measuring-devex-without-a-survey/.

What migrating logging from Datadog to Dynatrace actually cost us

vivek Itp — Mon, 21 Sep 2026 23:38:56 +0000

Every platform team eventually has to take something away. The technical part is usually straightforward. What goes wrong is everything between announcing it and switching it off, and the cost almost never lands where you expect.

The clearest example I have is moving our logging from Datadog to Dynatrace. The driver was ordinary: cost, plus consolidating logs and metrics onto one platform instead of paying two vendors to hold two halves of the same picture.

Before anything else: this is not a comparison of the two products, and nothing here is a complaint about either. Both do the job. The decision was commercial, the way most tooling decisions are, and what follows is about the migration rather than the tools. I have kept the internal specifics deliberately light.

I expected the hard part to be the applications. It was not. The applications barely noticed. What actually cost us was the monitoring built on top of the logs, and the fact that everyone who used logs daily had to relearn how to ask a question.

What I expected to be hard, and what was

Expected to be hard	Actually hard
Changing every application	The forwarder changed; apps mostly did not
Getting teams to prioritise the work	It rode along with their normal deployments
Log data itself	Custom masking rules tied to the old vendor
Nothing in particular	Every monitor and alert built on the old logs
Nothing in particular	Teams learning to query in the new tool

The two bottom rows are the whole story. Neither was on my list at the start.

1. Find the thing that is not in the obvious place

On paper the change was small. Logs went to a forwarder, and the forwarder needed to point somewhere new. No application code, no library swap, no redeploy required to change a destination.

What was not on paper was everything that had accumulated inside that forwarder. Over the years it had picked up custom masking rules, written against the old vendor's configuration format, quietly redacting sensitive values before logs left our systems. They were not documented as a dependency on the vendor. They were just part of how logging worked.

That is the shape of the hidden dependency in most deprecations. Not the integration everyone knows about, but the small accretions around it that nobody wrote down because they were never a decision, only a series of fixes.

What this looked like for me

The forwarder swap itself was an afternoon of work. The masking rules were not, because they had to be reproduced exactly in a different configuration language, and "exactly" is the operative word when the rules exist to keep sensitive values out of a log store.

Going through them one at a time was tedious and it was also the only responsible option. That was the point where the migration stopped being a config change and became a piece of work with a real review attached.

2. Run both, and validate, before removing anything

The decision that made everything else possible was refusing to cut over. For a couple of months, logs flowed to both platforms at once. Nothing was removed while we compared.

Dual shipping costs money, briefly, and it buys the only thing that makes removal safe: evidence. Not a belief that the new pipeline works, but a side-by-side you can point at. Same volumes, same fields, same masking applied, same events present in both.

The loop in the middle is what a validation period is for. Without it, "are we ready to switch off" is a judgement call made under time pressure. With it, the answer is a comparison anybody can check.

What the overlap actually caught

The overlap did not produce a dramatic discovery, and I count that as the period doing its job rather than being unnecessary. It let us confirm that masking was being applied the same way on both sides, which was the thing I was least willing to guess about.

Had we cut over directly and been wrong about that, we would have found out from the wrong direction entirely.

3. Let it ride on work teams are already doing

We did not run this as a migration project with a deadline and a chase list. Teams picked up the change as part of their normal feature deployments. Whatever they were shipping next carried the new configuration with it.

That choice is worth being honest about, because it trades one problem for another.

What you gain is real. There is no separate piece of work competing with a team's roadmap, no deadline anyone has to defend, and no incentive to rush. The change arrives with something the team wanted to ship anyway.

What you give up is predictability. A team that does not deploy for six weeks has not migrated for six weeks, and the tail is as long as your slowest-moving service. You cannot put a date on the board and be confident about it.

I think it was the right call here, specifically because dual shipping made a long tail harmless. Nothing was breaking while we waited. Take away the overlap and the same approach becomes a slow-motion outage waiting for the least active team.

No chase list

The thing I noticed most was the absence of the usual conversation. Nobody had to be persuaded, because nobody was being asked to stop what they were doing.

The trade shows up at the other end. The final stretch is a small number of services that simply had not deployed recently, and those needed individual conversations rather than a broadcast.

4. The applications were fine. The monitoring was not

Here is the part I would tell anyone planning a similar move: the migration is not the logs, it is everything built on top of them.

Monitors, alerts and dashboards are written against a specific query language, specific field names, and a specific way the platform structures data. None of that survives a vendor change automatically. Every one of them has to be rebuilt, tested, and trusted again.

That work is invisible in the plan, because nobody thinks of an alert as an integration. It is also the work where getting it wrong is worst: an alert that silently stops firing is far more dangerous than a log line that fails to arrive, and it fails quietly by definition.

Rebuilding the things nobody counted

The monitors were the bulk of the real effort and were not what I had scoped at the start. Each one had to be re-expressed in the new platform and then actually verified, because the only way to trust an alert is to make it fire.

If I ran this again, I would treat monitor migration as the main workstream from day one, and the forwarder change as the small task it turned out to be.

5. The real cost was people relearning how to ask

The last thing, and the one I keep thinking about, is that the hardest part of this was not technical at all.

People had years of muscle memory in the old tool. They knew how to get from an alert to the relevant log line in three moves, because they had done it hundreds of times. The new platform can answer all the same questions and it answers them differently, and that difference lands during incidents, when nobody has patience for learning.

That is a genuine cost of any tooling migration and it usually goes unnamed, because it does not appear as a ticket or an outage. It appears as things taking longer for a while, and as a quiet preference for the old tool for as long as the old tool still exists.

What would have helped is treating query fluency as part of the migration rather than an afterthought: a short page of the ten queries people actually run, translated, and a channel where asking how to express something was normal rather than an admission.

6. Leave a tombstone

When the old destination finally went away, what mattered was that anyone who went looking found an explanation rather than silence. What was removed, when, where the data lives now, and how to ask the question they were trying to ask.

Keep it up longer than feels necessary. The person who needs it is the one who only looks at logs when something is badly wrong, and they will arrive months later.

Takeaways

Look for the accretions, not the integration. The forwarder was easy; the masking rules written for the old vendor were not.
Run both and compare before removing anything. The overlap buys evidence rather than confidence.
Riding on normal deployments avoids the chase list and gives up predictability. It is only safe if nothing breaks while you wait.
Scope the monitors as the main work, not the pipeline. An alert that stops firing fails silently.
Name the relearning cost. Query fluency is part of the migration, and it is paid during incidents.
Leave an explanation behind for the person who shows up months later.

What I still have no good answer for is the deprecation with no replacement, where the honest message is that a capability is going away and teams will have to do without it. Everything above works because there was somewhere to go. When there is not, the conversation is about priorities rather than migration, and I have never found a way to make that one go smoothly.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/21/migrating-logging-datadog-to-dynatrace/.

Local development environments that match production closely enough

vivek Itp — Mon, 21 Sep 2026 23:38:51 +0000

Two failure modes, and I have caused both. The first is a local environment so far from production that "it works on my machine" stops being a joke and becomes a daily fact. The second is chasing full parity until starting the thing locally takes fifteen minutes, eats most of a laptop, and developers quietly stop using it.

Neither is a discipline problem. They are both the result of not deciding, explicitly, which differences between a laptop and production are allowed to exist.

I have written before about local and CI needing to run the same commands. This is the neighbouring question and a harder one: how close does a developer's running system need to be to the real one.

Some differences matter and most do not

Difference	Does it change behaviour?
Database major version	Yes. Syntax, query planning, and migrations all move
Runtime version	Yes. This is where most "works locally" bugs live
One replica instead of forty	No, until you are debugging concurrency
Real payment provider vs a fake	No, if the fake matches the contract
No TLS locally	Usually not, until a client library behaves differently
Ten rows instead of ten million	Not for correctness. Absolutely for performance work
Different operating system	Sometimes. Path handling, file watching, case sensitivity

The right-hand column is the whole exercise. Parity is not a slider you push toward one hundred percent. It is a set of individual decisions, each of which should have a reason.

1. Match the contract, not the infrastructure

The thing a service needs from its dependencies is the contract: the same interface, the same error shapes, the same semantics. It almost never needs the same topology, the same scale, or the same managed service.

A local queue that delivers messages at least once, out of order, and occasionally redelivers is a good stand-in for the production one, whatever runs underneath. A local queue that delivers exactly once and in order is worse than useless, because it lets people write code that cannot work in production and gives them confidence while they do it.

This is the test I use for a fake: does it fail in the same ways as the real thing? If not, it is teaching the wrong lessons.

What this looked like for me

We had a local stand-in for a message broker that was much friendlier than the real one. It delivered in order, never redelivered, and never dropped anything.

Code written against it worked locally and then behaved oddly in production, where duplicates and reordering were normal. The stand-in had been chosen because it was easy to run, and the ways in which it was easier were exactly the ways that mattered. Replacing it with something that redelivered and reordered made local development slightly more annoying and caught a category of bug before it shipped.

2. Pin the things that change behaviour

Runtime version and database major version are the two that have caused me the most confusion, and both are cheap to fix. Pin them in the repository, read the same pin everywhere, and the whole class of problem disappears.

The rule I use: if a difference can change the result of running the code, it goes in a file in the repository, and both the laptop and the pipeline read that file. If it cannot, it does not need to match.

# one place, read by the local tooling and by CI
runtime: "3.12.4"
database: "postgres:16.3"

What does not work is documenting the required versions in a README. Nobody re-reads a README after their first week, and versions drift silently from the moment they are written down.

The version that drifted

The most time-consuming local problem I dealt with was not dramatic: a developer's database was a major version behind what production ran, because they had installed it a year earlier and nothing had ever told them to change it.

A migration worked on their machine and failed in the pipeline with an error that pointed at the migration rather than the version. Pinning the database image and having the local tooling refuse to start on a mismatch removed that whole category, and the refusal message did more good than any documentation would have.

3. Decide what you are deliberately not matching

The parts that are not worth matching should be an explicit, written decision rather than an accident. Scale, multi-region behaviour, real third-party accounts, production data. Each of those costs a great deal to reproduce and buys very little for the work a developer does on a normal day.

Writing them down matters more than it sounds. When a developer hits a bug that only appears at scale, the useful response is "yes, local does not cover that, here is the environment that does" rather than an argument about whether local should have caught it.

The branch on the right is where most local setups go wrong. They fake something, the fake is friendlier than reality, and nobody notices until production disagrees.

4. Start-up time is a parity feature

An environment that takes fifteen minutes to start is not a high-parity environment. It is an environment nobody runs, which means its parity is irrelevant.

Every service added to the local stack has a cost paid by every developer, every day, forever. That cost is rarely weighed against the benefit, because the benefit is visible and the cost is spread thin.

The questions worth asking before adding anything to the local stack:

Does a developer need it running to do their normal work, or only occasionally?
Can it start lazily, when something actually calls it?
Can it be a fake that matches the contract rather than the real service?
What does it add to start-up time, and is that worth it every single day?

Cutting the stack down

Our local environment had grown to include everything a service might talk to, because each addition had been individually reasonable. Starting it had become something you did while making coffee, and a lot of people had stopped doing it at all.

Splitting it into a small default set that starts quickly, plus optional extras you opt into when you need them, was not technically interesting and made more difference than anything else on this list. Most developers, most days, needed a fraction of what we had been starting for them.

5. Make the difference visible, not surprising

When local and production do differ, the developer should be able to see it rather than discover it through a failure.

A short banner at start-up listing what is real and what is faked costs nothing and prevents a lot of confusion. "Payments: fake. Queue: local, redelivers. Database: postgres 16.3, schema current. Search: not running." A developer who sees that line does not spend an hour wondering why a payment did not arrive.

The same applies to data. If the local database has a small generated dataset, say so, ideally with a note about what it does not contain. Silence gets interpreted as "this is like production", and it never is.

6. Test the environment itself

A local environment is a piece of software that the whole team depends on, and it usually has no tests at all. It rots quietly: a fake drifts from the real service's contract, a pin goes stale, a start-up script depends on something that was removed.

A small scheduled job that starts the local environment from scratch on a clean machine and runs the smoke tests against it catches this early. When it breaks, it breaks for one person who can fix it, rather than for whoever next tries to onboard.

The setup that only worked if you already had it

Our setup instructions worked perfectly for everyone who had run them before, and reliably failed for new joiners. The scripts had come to depend on tools and state that existing machines happened to have.

Nobody had noticed because nobody started from scratch. Running the whole setup on a clean machine on a schedule surfaced it immediately, and onboarding stopped being a two-day exercise in asking colleagues what else they had installed.

Takeaways

Parity is a set of decisions, not a percentage. For each difference, ask whether it can change the result.
Match the contract, not the infrastructure. A fake that is friendlier than reality teaches the wrong lessons.
Pin what changes behaviour in the repository, read by both the laptop and the pipeline. Never in a README.
Write down what you deliberately do not match, so the gap is a known limit rather than an argument.
Guard start-up time. An environment nobody runs has no parity at all.
Print what is real and what is faked at start-up.
Test the environment on a clean machine on a schedule, or it rots.

The one I keep going back and forth on is production data. Realistic data catches a whole class of bug that generated data never will, and every approach I have used to get it locally, whether subsetting, masking, or anonymising, has either leaked something it should not have or been so lossy that it stopped being realistic. I do not have a good answer there.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/21/local-environments-close-enough-to-production/.

The support channel is your best product research

vivek Itp — Thu, 17 Sep 2026 14:44:38 +0000

Platform teams are usually bad at user research. We do not run interviews, we do not have a researcher, and the survey we send once a year gets answered by the people who already like us. Meanwhile the same team runs a support channel that receives dozens of real, unprompted, specific accounts of the platform failing its users, every single day.

We just do not read it that way. We answer the question, mark it done, and move on. The answer is the least valuable thing in the exchange. The question is the data.

I changed how I worked when I started treating the channel as a research feed instead of a queue. This post is what that looked like.

The exchange has two halves and we keep the wrong one

A support request has a visible half and an invisible half.

Visible	Invisible
"How do I get logs for my staging pod?"	They looked for the answer and could not find it
"Can someone re-run my pipeline?"	They do not have permission, or do not know they do
"Is the registry down?"	The error they saw did not say what was wrong
"What is the right base image?"	There is no obvious default, or there are three
"Who owns the deploy for service X?"	Ownership is not discoverable from the tooling

The visible half gets answered in two minutes and helps one person. The invisible half, if you collect it, tells you what to build. Every single question is a place where the platform failed to answer for itself.

1. Count the questions before you answer them

The cheapest useful thing a platform team can do is keep a tally. Not a ticketing system, not a taxonomy. A tally.

Every time a question comes in, put it in a bucket. Reuse buckets aggressively; if it is roughly the same question, it is the same bucket. After a few weeks the distribution will be extremely uneven, and the top few buckets will account for most of the volume.

That list is your backlog, ordered by how much pain each item causes, measured in the only currency that matters: how often a human had to ask another human.

What this looked like for me

I started with a text file and a habit. Every question I answered got a line. No categories decided up front, just a short description, and when something repeated I merged it into the existing line and added a tick.

After about a month there were maybe thirty lines, and the top five accounted for most of the ticks. None of the top five was on our roadmap. All five were small. That file changed what we worked on for the next quarter more than any planning session did.

2. The repeat question is the bug

A question asked once is a person having a bad day. The same question asked twenty times is a defect in the platform, and it should be treated exactly like one: reproduced, root-caused, fixed, and verified.

The trap is that answering is so cheap. Two minutes, a link, done. Twenty times two minutes is not the real cost, though. The real cost is the twenty people who were blocked until someone was available, plus the ones who did not ask and worked around it instead.

When a question repeats, the fix is almost never documentation. Documentation is where answers go to be not found. The fix is usually to make the question impossible: a better default, a clearer error, a link in the place they were already looking.

The question I answered too many times

The single most repeated question I dealt with was some version of "how do I see the logs for my service in staging". I had a good answer. I gave it dozens of times. I linked the documentation page, which existed and was correct.

What finally stopped it was not a better page. It was putting a direct link to that service's logs into the deploy notification, so the answer arrived before the question. The documentation page did not change at all. The questions stopped almost entirely.

3. Watch for the questions that are not questions

Some of the most valuable signals in a support channel are not phrased as requests. They are asides, complaints, and jokes.

"Ha, the pipeline is red again, business as usual."
"I just re-run it twice, works eventually."
"We stopped using that, it was easier to do it ourselves."
"Do not worry, everyone knows you have to do X first."

None of those is asking for help, so none of them gets logged anywhere. Each one describes a workaround that has become normal. A workaround that has become normal is a feature the platform failed to provide, and nobody will ever file a request for it because they have stopped expecting it to change.

I started keeping these in the same tally as the questions. They were often more useful, because they described problems people had already given up on.

The joke that was a roadmap item

Someone joked in the channel that the correct way to deploy was to "press the button twice and go get a coffee". It got a few laughs, including mine, and then I realised everyone in the thread knew exactly what it meant. The first deploy attempt failed often enough that re-running had become the accepted procedure.

Nobody had ever reported it, because it was not broken in a way you could report. It just did not work the first time, reliably enough that people had built a ritual around it. That went into the tally and turned into real work.

4. Answer in public, always

A support question answered in a direct message helps one person and produces no data. The same question answered in the channel helps everyone watching, gets found by search later, and stays in your tally.

Push everything into the open channel, politely and consistently. When someone asks privately, answer in the channel and link them. Not to shame anyone, but because the private answer is a small gift to one person and a loss to everyone else, including you.

The loop only closes if step three actually happens. A channel where everything is answered and nothing is counted stays exactly as busy next year as it is today.

Moving everything into the open

A lot of our support traffic arrived as direct messages, usually to whoever the person had spoken to last. That felt friendly and it was quietly destructive: the same question was being answered in four private conversations, and none of us knew it was common.

Moving to public-by-default took a few weeks of gently redirecting people. The channel got noisier, which looked like a regression. It was not. The noise had always existed, spread across private messages where it could not be measured.

5. Close the loop out loud

When you fix something that came from the channel, say so in the channel, and name the question it came from.

This is not self-promotion. It is the only way people learn that reporting friction leads to it being fixed. Teams that never see that connection stop reporting, and once they stop, the research feed dries up and you are back to guessing.

The message is short: this thing you all kept hitting, here is what changed, here is what you do now. It costs nothing and it is the difference between a channel that gets quieter every quarter and one that gets more useful every quarter.

What changed when we said it out loud

We had been fixing things from the channel for a while without announcing it, because each fix felt too small to mention. The result was that people kept asking about problems we had already solved, and nobody could tell that the platform was improving.

Posting a short note each time a channel-sourced fix shipped changed the tone noticeably. People started reporting smaller things, including things they had previously worked around silently. The quality of what we heard went up because reporting had visibly started to pay.

6. Be careful what you optimise

Once you start measuring the channel, the tempting metric is response time. It is easy to collect and it looks like service quality.

It is the wrong target. A team optimised for response time gets very good at answering quickly, which makes the channel pleasant and permanent. The goal is not to answer faster. It is to receive fewer questions because fewer things need asking.

The number worth watching is question volume per team per month, or how long the top bucket stays at the top. If the top question is the same one it was six months ago, the platform is not learning, however fast anyone replies.

Takeaways

The question is the data. The answer helps one person; the pattern tells you what to build.
Keep a tally, not a taxonomy. A text file and a habit will beat a tool nobody updates.
A repeat question is a defect. Fix the cause so it cannot be asked, and do not reach for documentation first.
Log the asides and the jokes. A normalised workaround is a request nobody will ever file.
Answer in public so the data exists at all.
Announce channel-sourced fixes or people stop reporting.
Do not optimise response time. Optimise for the question not being needed.

What I still have not solved is the silent majority. Everything above depends on someone asking, and the teams that struggle most are often the ones least likely to post in a public channel at all. I do not have a reliable way to hear from them short of going and asking, which does not scale.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/17/the-support-channel-is-product-research/.

Secrets management that developers do not route around

vivek Itp — Thu, 17 Sep 2026 14:44:33 +0000

Every secrets policy I have seen written down says the same sensible things. Do not commit credentials. Do not paste them in chat. Do not email them. Every organisation I have worked in has done all three, regularly, including the people who wrote the policy.

The instinct is to treat that as a discipline problem and respond with training. I think it is almost always a design problem. People take the shortest path to getting their work done. If the secure path takes twenty minutes and a ticket, and pasting a value into a direct message takes ten seconds, the direct message wins. Not because anyone is careless, but because one route works and the other does not.

The question worth asking is not "how do we stop people doing this" but "what made the wrong way easier".

Where secrets actually leak

Where it leaks	What it means
Pasted into chat	Getting the value the supported way was slower than asking a colleague
Committed in a config file	Local development had no other way to provide it
Baked into a container image	The build needed it and nothing injected it at runtime
Living in someone's shell history	The tool requires a value on the command line
A shared account nobody rotates	Individual access needed approval and the shared one did not

Every row is a workaround for a path that did not exist or was too slow. None of them is solved by a reminder.

1. Measure the secure path in minutes

Before changing anything, time it. From "I need a database password for my new service" to "my code can read it", how long, and how many people are involved?

If that number is measured in days, no amount of policy will hold. Developers will get the value from wherever they can and carry on, and the ones who do it fastest will look like the most effective engineers on the team.

The target is that the supported path is the fastest path available. Not merely acceptable. Fastest. When the secure route is quicker than asking a colleague, the problem mostly solves itself.

What this looked like for me

I timed our own onboarding path once, pretending to be a developer on a new service. Requesting access, waiting for approval, being told which path to use, and finally getting a working value took the better part of two days, spread across three people.

In the same organisation, asking a teammate in a direct message took about four minutes. Nobody needed to be told which one to use. The policy said one thing and the arithmetic said another, and the arithmetic always won.

2. Secrets should reach the process, not the person

The single biggest reduction in leaks comes from a change in shape: a developer should never hold the value at all.

If the runtime fetches secrets itself, using the identity of the workload, then there is nothing to paste, nothing to store locally, and nothing to rotate in a person's password manager. The developer references a name in config and never sees the value behind it.

# service.yaml, what the developer writes
env:
  DATABASE_URL:
    from_secret: orders/db/url

They name what they need. The platform resolves it at start-up using the service's own identity. Whether that is a cloud secrets manager, a vault, or something else matters far less than the fact that no human is in the path.

The change that removed most of the problem

We moved from handing developers values to injecting them at runtime. The developer's config named the secret; the workload's identity was what authorised the read.

What surprised me was how much it simplified their side. There was no longer any question about where to store the value locally, because there was no value. Most of the awkward conversations we had been having about handling procedures stopped being relevant, not because anyone was more careful, but because there was nothing left to handle.

3. Local development is where it breaks

Production is usually fine. Production has identity, injection, and rotation. It is local development that quietly ruins everything, because on a laptop none of that machinery exists and the developer needs something that works right now.

That is where the committed .env file comes from. Not from carelessness, but from a real need with no supported answer.

The options that work, roughly in order of preference:

Do not need the secret. Run against local fakes and a local database with a throwaway password. Most development does not require production credentials.
Short-lived personal credentials. The developer authenticates as themselves and the tooling fetches a value that expires within hours.
A dedicated development secret, clearly separated, low privilege, rotated on a schedule.

What does not work is asking people to keep a long-lived production value on a laptop and remember not to commit it.

The file everyone had

Almost every service repository had an example environment file, and almost every developer had a real one sitting beside it, ignored by git and sometimes not. It was not a rule anyone had broken; it was the only way to run the thing.

Giving the command-line tool the ability to fetch short-lived values for the developer's own identity is what actually emptied those files. The instruction to stop keeping them had been in place for a long time and had changed nothing.

4. Rotation has to be invisible

A rotation policy that requires developers to do something will not be followed past the first quarter. Any secret that a human must remember to change is a secret that does not get changed.

Rotation works when it is a property of the system: the platform issues short-lived credentials, or rotates long-lived ones on a schedule and the application picks up the new value without a redeploy. The developer's involvement should be zero, and the evidence of rotation should be a log line, not a ticket.

The dotted line back is the part most setups skip. If picking up a rotated value requires a restart that someone has to schedule, rotation stops being routine and becomes a small project, which means it stops happening.

5. Make the audit trail useful to the developer

Access logging is usually built for auditors and invisible to everyone else. That is a missed opportunity, because the same data answers questions developers genuinely have: which services read this secret, when was it last used, is anything still depending on the old one.

When the audit view is useful day to day, people look at it, and things get noticed. A secret nobody has read in six months is a secret that can be removed. A service reading a credential it should not need is worth a conversation. None of that surfaces if the log exists only for a yearly review.

The credential nobody was using

When we made access history visible in the same place developers looked at their service configuration, the first useful thing it produced was a list of secrets with no reads at all. Several belonged to services that had been decommissioned; the credentials were still valid and still granting access to things.

Nobody had been ignoring a cleanup process. There had never been a way to see the information. Putting it where people already looked was the whole fix.

6. Detection is the backstop, not the plan

Scanning for committed secrets is necessary and it is not a strategy. By the time a scanner fires, the value is in git history, possibly pushed, possibly public, and the response is a rotation and an awkward afternoon.

Use it, absolutely: a pre-commit hook that catches the value before it is committed is far kinder than a pipeline that catches it afterwards, and both are better than nothing. But treat every hit as a signal about the path, not just an incident to close. Someone needed a value and had nowhere good to put it. Ask what they were trying to do, and fix that.

Takeaways

Time the supported path. If it is slower than asking a colleague, expect people to ask a colleague.
Keep the value away from people. Inject at runtime against the workload's identity so there is nothing to hold.
Solve local development explicitly. Fakes first, then short-lived personal credentials. Never a long-lived production value on a laptop.
Rotate without asking anyone, including the pickup of the new value.
Show access history to developers, not only to auditors. Unused secrets surface themselves.
Treat every scanner hit as a design report, not only an incident.

The part I have not resolved is third-party credentials. Plenty of vendor integrations still hand you a long-lived key, offer no programmatic rotation, and expect a human to copy it from a web console once a year. Everything above assumes the credential can be issued and rotated by a system, and for that category it simply cannot.

Originally published at https://vivek-itp.github.io/mydevexblog/blog/2026/09/17/secrets-management-nobody-routes-around/.