DEV Community: Michał Iżewski

The Health Check That Watched the Wrong Thing

Michał Iżewski — Sun, 16 Aug 2026 13:34:36 +0000

The last post ended on a promise: background workers fall into the same correlated-failure trap as request services, from a different angle. This is that angle, and it starts one level earlier: a health check that reports green while the work has stopped.

I floated a heartbeat file for a queue worker, a small marker the worker rewrites as it goes, with a watchdog checking how stale it is. The reply came back fast: "just put a /health on it." Fair reflex. It's what every request service does, what every tutorial shows, what every load balancer expects. It's also the one thing that hides the failure I was trying to catch. An HTTP /health on a worker will return 200 all day while the worker sits there doing nothing.

Process is up, actual work has stopped. That gap in between is what the whole post is about.

Where the HTTP reflex comes from

"Health check means an HTTP endpoint" is baked in for a good reason. Every service that handles requests exposes one, every orchestrator polls one, every tutorial you've read wires up GET /health returning 200. For a request handler that's not just convention, it's correct. The reflex only misfires when you carry it, unexamined, onto a process that doesn't handle requests at all. Same shape as the session library from the last series entry: a default that's right in the context it was built for, and surprising the moment you step outside that context. Nobody was wrong to reach for it. It's answering a question the worker doesn't ask.

The failure the green endpoint can't see

Picture the worker. It runs a loop: pull a job off the queue, do the work, pull the next one. To satisfy the "just add /health" reflex, you bolt an HTTP listener onto it, and that listener answers independently of the loop, from its own thread or process or whatever your runtime uses.

Now the loop wedges. A job deadlocks, the connection pool is exhausted and every task is stuck behind one that never returns, a lock is held by a thread that's gone. Timeouts help, and most libraries ship with them, but a default one isn't a tuned one, so the loop can sit far longer than the work really needs before anything gives up. The work stops. The listener doesn't: it keeps accepting connections and returning 200, because answering a trivial GET has nothing to do with whether the loop is moving.

So the orchestrator sees green. The queue behind the worker climbs. And the exact failure you put the health check there to catch, a worker that's alive but stuck, is the one failure the check is structurally blind to. The endpoint is answering exactly the question it was asked, and that question was about the HTTP server, not the loop.

Proof of life isn't proof of work

The obvious patch is to make the check smarter. The HTTP /health only proves the listener answers, fine, so have it also confirm the worker process is actually running: check the PID, ask the supervisor, systemctl is-active. Not hard to wire up. And still the wrong signal, because a dead process is often not the failure you're actually worried about. Far more often the process is up and the loop is parked on a job that will never return.

The signal has to come from the work

The reason the HTTP reflex is right for a request service and wrong for a worker is the same fact read two ways.

In a request service, handling the request is the work. When the health check hits an endpoint and gets a response, the act of responding is itself proof that the request path is alive: routing, framework, handler, all of it just ran. The check can't easily lie, because doing the thing it checks is how it passes.

A worker breaks that coupling. The work is the loop, and an HTTP listener bolted on beside it answers independently of the loop, so by default the check proves the listener is up, not that the loop is moving. Nothing stops you from wiring the two together: have the loop update a shared timestamp and let /health return 503 when that timestamp goes stale. Do that and the endpoint is honest again. HTTP can carry a real signal; it only has to originate in the loop, something only the loop can refresh as it makes progress, and stop the moment the loop stops. Where you expose it, a file, a metric your monitoring already scrapes, an HTTP route, a socket, matters less than whether it's tied to the work.

A heartbeat the loop writes, a watchdog that reads it

The simplest thing with that property: the loop writes a small marker after it does real work, and a separate watchdog checks how stale the marker is.

# inside the worker loop
while true; do
  job="$(reserve_job)"        # blocks until there is work
  process "$job"
  touch /run/worker.alive     # heartbeat: proof the loop advanced
done

A file under /run or /dev/shm carries real weight here: the marker means something precisely because only the loop can refresh it. A separate listener can't fake it, because it never touches the loop.

The watchdog runs on cron, out of process, and does one thing. If the marker is older than a threshold, mark the instance unhealthy and let the platform replace it.

# watchdog on cron: replace the instance if the loop has gone quiet
threshold=$(( max_job_seconds + slack ))       # longest healthy job, plus margin
age=$(( $(date +%s) - $(stat -c %Y /run/worker.alive) ))
if (( age > threshold )); then
  mark_instance_unhealthy                      # let the platform replace it
fi

Then the naive version bites. Look at where the touch sits: after process. The marker only advances when a job completes. Now think about an empty queue. reserve_job blocks, nothing arrives, the loop never reaches the touch, and to the watchdog that's indistinguishable from a loop wedged on a stuck job. Both stop refreshing the file. A worker that's perfectly healthy and simply out of work gets cycled for no reason. This assumes reserve_job returns on a timeout: it long-polls for a few seconds, then comes back empty. If instead it blocks forever on an empty queue, the loop stalls there, the marker goes stale, and even the fixed heartbeat won't save an idle worker from being killed. The poll has to be able to time out for any of this to hold.

So the marker should prove one thing: the loop is still turning. Move the heartbeat so it fires on every iteration, idle poll included, and pick the threshold above your longest legitimate job so a slow-but-healthy job isn't read as a wedge:

while true; do
  job="$(reserve_job)"                 # blocks up to a poll timeout, then returns
  [ -n "$job" ] && process "$job"
  touch /run/worker.alive              # proof the loop turned, job or no job
done

That fixes the single-threaded loop. But most real workers aren't single-threaded, and that's the next assumption to pull out. They run a pool, a handful of threads or goroutines or async tasks pulling jobs in parallel, and one touch from a shared spot lies about all of them. Say ten workers share the process and one wedges on a deadlocked query while the other nine keep churning through small jobs. The marker stays fresh, the instance stays green, and you've quietly lost ten percent of your throughput with a connection stuck open the whole time. That's a grey failure, and a central heartbeat is exactly blind to it, for the same reason the HTTP check was: it's measuring something beside the work, not the work. The signal has to come from all of them at once. Each one refreshes its own timestamp and the watchdog reads the oldest live one, or the pool supervisor checks its children before it touches the file.

Whether the loop is doing useful work (draining a queue that's actually backing up) is a different question, and liveness is the wrong place to answer it. That belongs to queue-depth monitoring, one level up. Keep the two apart, or you'll wire queue depth into liveness and rebuild the exact correlated-failure trap from the last post, this time on your workers. And even a healthy heartbeat only proves the loop turns, not that the turning gets anything done, which is a separate failure this post doesn't close.

A few more details bite. First, startup: a worker that just booted hasn't turned the loop yet, so the marker is missing on a healthy instance. Seed it at boot, or give the watchdog a grace window that covers boot plus the first poll cycle before it's allowed to judge a fresh instance. Otherwise every deploy shoots its own new workers. And the threshold cuts both ways: it has to clear your longest honest job, so if that job runs forty-five minutes, a genuinely wedged worker also sits forty-five minutes before the watchdog notices. Long jobs and fast recovery pull against each other here, and the way out isn't a bigger threshold, it's the job itself reporting progress as it runs, a different signal living one level up.

Second, restart versus replace. The tempting fix when the watchdog fires is to bounce the worker in place through the process supervisor: SIGTERM, wait, then SIGKILL if it's still there. The pragmatic move is to skip the ceremony and just rotate the whole instance. It's simpler and clears more, a wedged socket, a leaked handle, memory corrupted into a corner, all gone with the box. What it won't clear is a cause that travels with the work: a job that wedges whatever picks it up, or a shared dependency that's still down. But those don't survive an in-place restart either, so there's nothing to weigh. Rotating is slower, so wire the infra to bring the replacement up before it pulls the wedged one out, unless you like explaining capacity dips.

Why put HTTP on a worker at all

Once the signal is a file the loop touches, a separate HTTP endpoint mostly earns its keep on a request service, where the listener is already there and a /health route is free. A worker usually has no listener at all, so bolting one on means a server, a port, and sometimes a second container in the pod whose whole job is to answer 200. If the file already carries the signal, that's a lot of machinery to expose the same fact.

On Kubernetes you can skip the listener entirely. The worker doesn't sit behind a Service, so there's no traffic to gate with a readiness probe, which leaves liveness as the whole question here (the last post pulled those two apart). Nothing routes to an HTTP port anyway, and the liveness probe reads the marker directly with an exec:

livenessProbe:
  exec:
    # probe runs with no shell, so wrap the test yourself
    command: ["sh", "-c", "[ $(( $(date +%s) - $(stat -c %Y /run/worker.alive) )) -lt 60 ]"]
  periodSeconds: 10
  failureThreshold: 3
# slow first turn? use a startupProbe, not a fat initialDelaySeconds

Two catches with the exec approach. First, the image has to carry sh, stat and date, and carry the flags the command expects, which a distroless image won't and a busybox one bends, so a probe that passes on your laptop can misfire on Alpine. Verify it against the real base image, or ship a tiny static helper that does the check and exits non-zero. Second, exec isn't free: every probe forks three processes, so ten seconds a pod across a big fleet is a steady drip of spawns and kubelet overhead.

There's a way around both, and it's where the signal being a metric pays off. If the loop already exports its heartbeat as a metric your monitoring collects, whatever you run, an alarm defined on that metric, going stale past a threshold, can fire the reaction from outside the pod: no forks, no shell, no exec on the box at all. The shift is that this detects and alerts rather than restarts, the alarm pages someone or triggers automation, where a liveness probe would have the kubelet kill the pod on the spot. So it catches the wedge but doesn't reset it by itself. Pair it with a probe if you want the automatic restart, or run it alone where a human or a separate automation closes the loop. A file with exec is simplest and self-contained, an in-process handler on a shared timestamp skips the fork tax, and a metric moves the check off the box entirely at the cost of that directness.

On EC2 with an ASG the file wins more easily, because a watchdog already runs on the box reading that same marker. A cron job comparing the marker's age to a threshold is enough; systemd has a native watchdog that does the same thing if you'd rather not run cron. Either way, an HTTP endpoint on top buys nothing the watchdog doesn't already have. I haven't run workers on ECS myself, so I'll leave its specifics alone rather than guess, but the split is the same wherever you land: the signal comes from the loop, not the transport.

Long-running processes don't get an honest signal for free

The worker is one case of a wider shape. Crons, queue consumers, stream processors, anything long-running that isn't already sitting behind something that checks it, none of them get an honest health signal for free the way a request handler does. And the failure follows a similar pattern every time: a green endpoint, a running process, an exit code of zero, standing in for work that quietly stopped. The transport says alive. The work says nothing, because nobody wired the work to speak.

The fix generalizes as cleanly as the trap does. Put the signal somewhere only real work can emit it, and when it stops arriving in the window you expect, rotate the process. Whether the loop is doing anything useful beyond staying alive, draining a backlog, actually committing rows, is a second signal living one level up, and worth building separately. Get the first one right and a stuck worker stops looking healthy. Get both and you can see it slow down before it stops.

A worker wedged on a dead job used to sail through its health check and pile work up behind a cheerful 200. Wire the signal to the loop, and it trips the check the moment it stops moving, which is the moment you wanted to know.

Two things this leaves open, both about the same blind spot. A worker can turn its loop forever without making progress, and it can hang precisely because it's beating on a dependency that needs room to recover. Same root: a signal that says "alive" over a process doing no useful work. Which of those bites you first decides what's worth building next.

The Health Check That Caused the Outage

Michał Iżewski — Tue, 21 Jul 2026 19:43:44 +0000

Your database dropped out for a few minutes. Your health check turned that into a full outage.

That's not a hypothetical situation I made up for effect. It happened to us twice, in two different shapes. Here's the first.

We had a health check that reached DocumentDB. One day a Security Group rollout didn't go to plan, and the app could no longer reach DocumentDB. It sat behind maybe a tenth of our endpoints, so the damage should've been small.

It wasn't. The processes never crashed, the app was alive on every instance. But the health check couldn't reach DocumentDB, so it flagged every instance as unhealthy at the same moment, and the platform started cycling the whole HTTP tier out, faster than the replacements could come back. The workers kept running with nothing to do, because the thing that fed them had gone dark. Took us a while to trace it back to one Security Group change. A dependency that maybe a tenth of the product needed had taken down all of it.

Putting that database check in the health endpoint feels responsible, and that's why so many teams do it.

The assumption every orchestrator makes

Start with what a health check is for. The load balancer, the autoscaling group, the Kubernetes control plane: all of them run the same loop. Find the unhealthy instance, take it out of rotation, put a healthy one in. Route around the broken thing.

That loop has a hidden assumption baked in. It assumes failures are independent. And it works because when one instance breaks, the others are fine, so there's somewhere to send traffic and something to replace the dead node with.

A deep health check breaks that assumption. The moment your check queries a dependency that every instance shares, you've tied the health of all of them to one external thing. When that thing goes down, every instance fails its check in the same second. The machinery built to route around one broken instance now has nothing healthy to route to. It was never designed for correlated failure, and you created correlated failure on purpose.

That's the core of it. The rest of this post is what it looks like on EC2, ECS and EKS, and the way out, which turns out to be one principle applied a little differently each time.

Liveness vs readiness: the same mistake, two outcomes

Liveness and readiness are the two jobs a health check can have: is the process alive, and should it get traffic right now. Kubernetes splits them into separate probes; most other setups blur them into one.

Where you wire the deep check decides how badly it hurts.

On readiness, all instances go NotReady together when the dependency blips. The load balancer sees zero healthy targets. Now even requests that never touched the database get nothing back, because there's no instance left in rotation to serve them. A local problem becomes a total one.

On liveness, it's worse. The orchestrator doesn't only stop sending traffic. It decides the app is dead and starts killing and restarting processes, because a dependency is slow. So at the exact moment the system is fragile, you add cold starts, fresh connection storms, and more pressure on the dependency that was already struggling. You're feeding the fire you were trying to put out.

Liveness answers one question: is this process alive and able to serve a trivial request. It doesn't touch business dependencies. If the process is alive, liveness keeps succeeding and the orchestrator leaves it running. Whether traffic is routed to it is a separate, readiness concern. Endpoints that actually need the database return 503 from the app's error handler, per request, when the database is down. That 503 comes from your code, not from the infrastructure pulling the whole instance out of service.

The same trap on EC2 + ASG + ALB

Most writing on this assumes Kubernetes. The trap is the same on plain EC2, slower and more expensive, and both of our incidents ran there: behind an ALB, with an Auto Scaling Group.

The setup is ordinary. The ALB target group runs a health check against an endpoint. The ASG has health check type = ELB, so it replaces any instance the load balancer marks unhealthy. Reasonable, until the checked endpoint reaches a shared dependency. Then one downstream failure fails every target at once, the ASG marks the whole fleet unhealthy, and it does what you configured: terminate and replace.

Here's where it turns into a loop that's hard to break from the inside. When the ASG launches a replacement, CodeDeploy runs the deployment lifecycle on that fresh instance before it enters service, validation hook included. If that validation hook also reaches the dependency, it fails too, because the dependency is still down. So the replacement never passes validation and never joins the fleet. The old instances are cycled out by the health check, and the new ones can't get in past the deploy gate. The fleet can't even rebuild itself with the artifact it already has, because the thing failing is the dependency check, not the code. That's the rotation the DocumentDB outage put us in. You don't recover until the dependency does.

An ASG doesn't understand any of this. It reasons about a single number: desired capacity minus current capacity. It sees a missing instance and launches one, with no idea that every instance is failing for the same reason. So it keeps throwing fresh machines at a wall, each one hitting it in the same spot.

One AWS detail matters here, because the earlier framing is too simple for the ALB specifically. An ALB doesn't black-hole traffic the moment every target is unhealthy. It fails open: with no healthy target in a group, it routes to all of them anyway, on the assumption that the check is the thing at fault. So on EC2 the lost capacity isn't the ALB refusing to route. It's the ASG terminating the instances the ALB flagged, and the failed replacements never coming up behind them.

The shallow version is undramatic. The ALB check hits a lightweight endpoint that touches no external dependencies, the ASG replaces an instance only when the shallow check says that instance itself is unhealthy, and the maintenance policy launches a replacement before terminating the old instance, so capacity never dips.

# ALB target group: shallow check, nothing downstream behind it
resource "aws_lb_target_group" "app" {
  health_check {
    path    = "/health"   # process-alive only, never the database
    matcher = "200"
  }
  deregistration_delay = 30   # drain in-flight requests on the way out
}

# ASG: replace when the shallow check fails, launch before terminating
resource "aws_autoscaling_group" "app" {
  health_check_type = "ELB"

  instance_maintenance_policy {
    min_healthy_percentage = 100
    max_healthy_percentage = 110
  }
}

None of it helps if the endpoint behind path = "/health" reaches a shared dependency.

The second incident: when more instances made it worse

Underneath, this outage and the DocumentDB one were the same: a connectivity problem that automation escalated into a fleet-wide one, both rooted in infra config. What made this one harder to climb out of was an autoscaler that took the cascade and pulled it tighter. It's the same machinery as the earlier replacement loop, pointed the other way: instead of swapping instances out, it kept piling new ones on.

We'd been scaling up to chew through a big backlog, and the capacity got ahead of what the database could take. As I remember it, enough workers came up to use up the database's connection limit, and RDS stopped accepting new connections. From there it was a feedback loop.

You might expect a stalled database to leave the app idle, blocked on I/O, with the CPU relaxed. It went the other way, and the scaling rule fired on CPU. I was on the sidelines for this one, so I'm reconstructing the compute side, but the shape is familiar: the box wasn't waiting quietly, it was working hard at failing. Workers piled up blocked on connections that never came, every failed request threw and was logged, and a log-shipping agent churned through the flood. None of it is useful work, but all of it burns CPU, and to a scaling rule a pinned CPU looks like demand.

So it scaled out, the worst thing it could do. Every new instance opened its own connections on boot, so the moment a slot freed up a rising instance grabbed it before anything stable could. The fleet climbed to its ceiling and kept rotating, and none of it helped, because the bottleneck was the database, not the compute.

The fix had to go where the bottleneck was, at the dependency: more connections, a bigger instance, the usual capacity work. A connection proxy would've softened it too, by pooling connections instead of letting every instance open its own, but softening is not removing. A proxy still fronts a finite database, and not every dependency has one. The cleaner answer is to stop generating the pile-on.

What stuck with me was the shape of it. More instances is the standard answer to load. Here, more instances kicked us in the face. The autoscaler assumed every new instance would help. But every new instance made the starvation worse.

ECS sits in the middle

ECS lands between the manual EC2 setup and full Kubernetes, and the trap shows up there too, under different names.

ECS gives you two health signals. The container health check, the HEALTHCHECK in your task definition, behaves like liveness: fail it and ECS replaces the task. The target group health check, run by the ALB, behaves like readiness: fail it and the task is deregistered from traffic, and replaced. Put a dependency check in either one and you get the same cascade as everywhere else, with ECS rescheduling the whole set of tasks against a dependency that's still down.

The one ECS-specific knob here is healthCheckGracePeriodSeconds, which gives a freshly started task a window before the ALB health check can mark it unhealthy. Notice what it does and doesn't cover: it buys time for a slow boot, nothing more. It says nothing about whether a downstream is healthy. Those are two different problems, and ECS has no separate signal for the second one, so the workaround later in this post applies here too.

Kubernetes splits the one signal into three

I'm still getting to know Kubernetes. I come from the EC2 and ASG side, where you wire rotation and replacement together by hand. What struck me while learning EKS is that the platform separates three questions EC2 forces you to answer with a single signal: is this alive, should it get traffic, is it still starting.

Each question maps to a different action. Kubernetes restarts a container that fails liveness. It holds traffic off a pod that hasn't passed readiness. It waits through a slow boot with the startup probe instead of killing the pod for being slow. A rolling update never sends traffic to a pod until it reports ready.

It's easy to overstate the startup probe. It covers slow startup and nothing else. None of the three probes was built to express dependency health. Each answers one thing about the instance itself: liveness, the process is alive; readiness, this pod can take traffic; startup, the boot has finished. People do wire a downstream check into readiness, and sometimes that's right, but it's something you add on top. What the platform gives you by default is those three concerns kept separate. That's the real difference from EC2, where there's one signal, so everything gets crammed into it, which is also why the deep-check mistake is so easy to make there.

Readiness is about useful work

Readiness has a real job, and it's not checking dependencies. It gates traffic during rolling deploys and graceful drain. A pod starts, the liveness check returns 200 right away because the process is up, but the caches are cold and the connection pools are empty, so the first requests would be slow or fail. Readiness holds traffic off until the pod is warmed up, and during shutdown it lets you drain in-flight requests before the process exits. It's about this instance's own state, not the health of anything downstream.

Which is why the common correction misses. "Don't check the database in liveness, check it in readiness instead" just moves the deep check. On a shared dependency the correlated failure comes along for the ride: the dependency blips, every instance reports not-ready at once, and the load balancer is back to zero healthy targets, with all the operational mess that brings.

There's at least one shape where a readiness check on a dependency is the right call. Picture a small, single-purpose service: it takes a message and writes it to a queue, and that's all it does. If the queue is gone, the service has no useful work left. It can do nothing except return 503, and every one of those failed requests is another connection thrown at a dependency that's trying to recover. A healthy-but-useless instance hammering an unhealthy dependency can be the very thing that keeps it from coming back. Here, dropping the instance from rotation simply helps. There's no partial availability to protect, because every request needs the one thing that's down, and pulling out of routing takes pressure off the dependency while it recovers.

The distinction is footprint. A readiness check on a dependency is right when that dependency is the service's whole reason to exist. It becomes the outage from the start of this post the moment the dependency sits behind only part of a larger app and gets treated as a prerequisite for all of it. So readiness answers "can I do useful work," not "is my shared downstream alive."

When the platform gives you one signal

What do you do when you can't express "alive but degraded" at the instance level, because there's one health signal and it's binary. Maybe you're on EC2 with a single ALB check, maybe your platform doesn't separate the probes the way EKS does. The move is to push the dependency awareness down one level, from the instance to the request.

Keep /health shallow. It returns 200 if the process can serve a trivial request, and that signal alone decides rotation and replacement. Then let each endpoint decide for itself. An endpoint that needs the database tries the database and returns 503 if it's down. An endpoint that doesn't need the database keeps returning 200 and keeps serving. The blip degrades only the endpoints that depend on the broken thing. The rest stay up.

This has a name worth using: partial availability. The goal is to keep serving the requests that don't need the dependency, not to pretend it's healthy. If 20% of your requests depend on the failed component, fail those 20% and keep the other 80% working. A shallow check plus per-request errors gets you that. A deep check gets you zero percent, because it takes the whole fleet down over a dependency that most requests never touch.

One honest limit of the per-request approach: it still reaches for the dependency on every request. Each call tries, waits, and fails, so a struggling dependency keeps getting poked while it recovers. The usual tool for cutting that off is the circuit breaker: past a failure threshold it stops trying, returns the error at once, and probes in the background until the dependency is back. That's a post in itself, so I'll leave it here as a name to look up.

The real fix is keeping shared dependencies out of the signal that drives replacement and routing. A shallow /health is usually the simplest way to do that. There's a blunter fallback when you can't change the check itself: set health check type = EC2, so the ASG replaces only on instance-level failure and a failing ALB check can no longer trigger termination. It breaks the churn loop, but treat it as a last resort. The ELB check is what catches a process that's running but wedged: a deadlocked app, a server returning stale 500s, something alive but no longer answering. Drop to type = EC2 and you lose that, and have to rebuild it with a watchdog of your own.

Correlated failure is the pattern

Most of us design the happy path well. The question that gets asked too rarely is "what happens when a dependency gets slow or goes away." A health check is one of the few places where that question has a default answer already baked in, and the default that feels most responsible, prove the whole stack works on every check, is the one that converts a small dependency blip into a system-wide outage.

The pattern runs through every layer. Health checks, load balancing, auto-replacement, auto-scaling: all of it assumes failures are roughly independent, and that adding or replacing capacity helps. The machinery routes around the broken instance because the other instances are fine. Tie every instance to the same shared dependency and that assumption is gone. One slow downstream takes the whole fleet together, and the system built to save you either has nothing left to route to, or it scales straight into the bottleneck and makes recovery impossible.

The checks earn their keep by staying simple. Liveness: is this process alive. Readiness: can this instance do useful work yet. Neither asks whether a shared downstream is healthy, because that's a per-request question. Answer it per request, and a broken dependency costs you the requests that needed it. Answer it at the instance level, and it costs you everything.

Get the distinction right and the same blip stays contained: only the endpoints that needed the broken dependency fail, the rest keep serving. No fleet-wide rotation, no autoscaler digging the hole deeper, no recovery fought by your own automation. The machinery built to save you finally does it.

Background workers run into the same correlated-failure pattern from another direction: an HTTP health endpoint can happily report a live process while the work loop itself is dead. That's its own post.

The Session ID That Wouldn't Stop Changing

Michał Iżewski — Tue, 07 Jul 2026 06:48:12 +0000

I was implementing a feature where the session container would track a lastActivity timestamp, updated on every authenticated request. Standard stuff. I wrote it, tested it locally with curl, and noticed something odd: I kept getting a new Set-Cookie header value on every response. Not occasionally. On every single one. A week later I was sending a pull request to mezzio/mezzio-session-cache.

The Setup: Two Backends, One Session

Our system had a constraint: two backend applications, written in different languages, sharing a single user session. One was the main PHP/Mezzio app. The other was a service in a different stack that needed to read from, and update the lastActivity timestamp on, the same session container.

There are a few ways to make polyglot session sharing work. We landed on a shared cache backend (Redis) with a well-defined session structure. Both apps could read and write through their own libraries, as long as they agreed on the storage format and the cookie name. The session ID was the contract.

That contract is the part that quietly broke.

A Missing Escape Hatch

My first instinct was the usual list of suspects. Was something calling regenerateId() in a middleware I didn't know about? Was there a logout being triggered somehow? Was a misconfigured cache layer evicting and recreating sessions?

After a bit of digging through the call stack, I ended up in the library itself. And there it was: CacheSessionPersistence was regenerating the session ID whenever the session data changed. Not on login. Not on privilege escalation. On every write.

That's when the real question hit me: why on earth would a library do that by default?

Reading Code Before Changing It

When you find behavior that surprises you in someone else's code, the wrong move is to immediately label it broken. The right move is to assume the maintainers had a reason, and find out what it was.

The reason, in this case, is session fixation.

Session fixation is a class of attack where an attacker tricks a victim into using a session ID the attacker already knows. If the session ID never changes after authentication or privilege changes, the attacker can hijack that session and act as the victim. The standard defense is to regenerate the session ID at any point where the trust level of the session changes: at minimum on login, ideally at any meaningful state transition.

Mezzio's library went further: it regenerated on every data change. Defense-in-depth taken to its logical extreme. The cost (a few extra cookie writes) is invisible to most apps. The benefit (making session fixation nearly impossible to exploit through normal usage) is real. For a single-application setup, you'd never notice the trade-off. For anything distributed, it changes the session ID from a stable contract into an ephemeral token that rotates with every write.

For our architecture, that was catastrophic. Both backends depended on the session ID remaining stable. The moment our PHP app updated a lastActivity timestamp, the library regenerated the ID on the spot. The other backend, still holding the previous ID, was now pointing at a session that no longer existed. A classic stale reference problem, except the staleness wasn't caused by TTL or eviction. It was caused by the library deliberately rotating identifiers under us. The contract broke.

I wasn't alone in hitting this. Months before my PR, an issue described the same underlying problem from a different angle: users losing their sessions when a response failed to reach the client, because the old session ID had already been destroyed the moment the session was persisted. Two production scenarios, one missing escape hatch.

The Decision: Where Does the Fix Belong?

Three paths on the table:

Patch around it in our app. Avoid writing to the session in certain code paths, or build a parallel storage mechanism that bypasses the library. Every developer on the team would need to remember the rule. Invisible discipline. The worst kind of debt.
Fork the library. Maintain our own version with regeneration disabled. Now I own a fork of a security-sensitive library. Every upstream release becomes a merge conflict review. Every security advisory becomes a question of "does that affect our fork too?"
Contribute upstream. Add an opt-in to disable auto-regeneration, keep the secure default, let maintainers decide if it's worth merging. Rejection risk is real. Review iterations are real. But the change either lands and stops being my problem, or it doesn't and I'm back to one of the other two paths anyway.

It wasn't a hard call. Patching around it costs the team forever. Forking costs me forever. Contributing upstream costs me a week, with non-zero rejection risk. The expected value of the week was lower than the certain cost of the other two options, by a wide margin.

What tipped it was the security angle. A fork of a session library is the kind of thing that would haunt me in three years when someone reports a CVE and I have to figure out whether our fork is affected, whether the upstream patch applies cleanly, and whether I still remember why we forked in the first place. Patching around the library has the same problem in a different shape: someone new joins the team, doesn't know we patched around the library, writes to the session in what looks like a perfectly reasonable place, and session sync breaks in production. It's happened to me before, on the receiving end. I wasn't going to set that up for someone else.

Only upstream made sense long-term.

The Cost: A week of focused work, plus review iterations spread over the week. Real rejection risk.

The Win: A library-level escape hatch maintained by the project, not by us. Zero ongoing maintenance cost.

Writing the Change

The shape of the contribution was deliberately small:

A new auto_regenerate config option, defaulting to true (no behavior change for existing users).
A constructor argument on CacheSessionPersistence that threaded the option through.
The factory updated to read the option from config.
Tests covering both the default behavior and the new opt-out path.
Psalm baseline updated to reflect the new types.

End result, from a user's perspective, looks like this:

// config/autoload/session.global.php
return [
    'mezzio-session-cache' => [
        'cookie_name'     => 'sid',
        'auto_regenerate' => false, // opt out of regeneration on every write
    ],
];

Config-level rather than a runtime toggle, because flipping session ID regeneration mid-request is the kind of thing that gets you into trouble you didn't ask for.

That last point about Psalm matters. When you write code in someone else's project, you're not just meeting a functional bar, you're meeting their quality bar. Mezzio uses Psalm at a strict level. The PR wasn't done when the feature worked; it was done when every static analysis check passed at the level the maintainers expected.

The Review Process Was the Real Test

The PR went through a proper review with two maintainers from the project. The feedback was sharp and useful:

Tests were requested specifically for the false case. Fair. I'd tested the default but not the new branch as thoroughly as I should have.
A type-casting suggestion on the factory: (bool) $autoRegenerate to remove a Psalm baseline entry and prevent a class of common errors.
One small inline tweak suggested via GitHub's review UI.

The standard was what stood out. Every comment was the kind of thing a careful engineer would say in your own team's review. No gatekeeping, no theater. Just "this needs to be a little more robust, here's why."

I addressed every point, pushed updated commits, and the PR was merged into the 1.13.x branch. The feature shipped in version 1.13.0.

Looking Back

The bug wasn't in our code. The fix didn't belong in our code either. The work was upstream, so I went upstream.

Two things stuck with me from this PR. First: when a library has a default that surprises you, that default is part of the API: not a bug, not laziness, but a deliberate choice the maintainers made for users who haven't thought about the edge case. Working with that choice rather than against it is the difference between contributing and forking.

Second: turning off auto_regenerate doesn't remove the session fixation defense. It moves it, from the library into the application. That's a fair trade when you've already got the rest of the controls in place (regeneration on login, on privilege escalation, on logout). It's a security regression when you don't.

The rest is the general pattern for working with other people's code: read more than you write, take time to understand the maintainers' position, don't fork what you can fix upstream. This PR just made them concrete.

The PR discussed here is mezzio/mezzio-session-cache#51, merged into the 1.13.x branch in May 2024. The auto_regenerate option shipped in version 1.13.0 and remains in the library today. A follow-up contribution from the same project, #52, implementing InitializePersistenceIdInterface, shipped in 1.14.0 a month later. The library has continued on its 1.x line; the latest release at the time of writing is 1.17.0.

How I Cut Our CI/CD Pipeline from 35 to 5 Minutes

Michał Iżewski — Mon, 01 Jun 2026 21:48:55 +0000

You push a critical hotfix, switch branches—and your CI is still running 30 minutes later. In my current project, our backend pipeline ran across a massive monorepo (~7,000 unit, ~600 integration, ~150 API E2E tests, ~200MB vendor) — and it bloated to a painful 35+ minutes.

It was killing productivity. While the team was frustrated, no one had the time to dive into the runner infrastructure. My roots are in Linux sysadmin work, and I’ve always carried that 'tinkerer' DNA into my software engineering. I don't stop at "it works" — I want to understand the full execution path and how it interacts with the OS and the hardware.

Driven by that urge to look under the hood, I decided against throwing more AWS resources at the problem. Instead, I dug into what was actually slowing things down. This isn’t a full how-to. It’s a practical look at what actually moved the needle — and what didn’t — when we cut our pipeline down to ~5 minutes.

1. Bypassing AWS EBS Limits with RAM Disks (`tmpfs`)

Initially, I suspected our test databases were the main I/O bottleneck. But a quick look at our EC2 metrics revealed the truth: the CPU was barely breaking a sweat, while disk I/O was completely maxed out. The real killer? Logs and temporary file processing. We deliberately run our CI tests in debug mode so developers get a full stack trace instantly upon failure.

Writing massive debug logs and processing large volumes of temporary files on EBS under concurrent load was grinding the runners to a halt.

The fix: I mounted both the MySQL data directories and our application's temporary storage directly to RAM disks. To squeeze every last drop of performance, I also bypassed MySQL's data safety mechanisms (since we don't care about data loss if a CI container crashes) and added a size limit to avoid OOM kills by the kernel.

# docker-compose.ci.yml
services:
  database:
    image: mysql:8.0
    command: --innodb_flush_log_at_trx_commit=0 --sync_binlog=0
    tmpfs:
      - /var/lib/mysql:rw,noexec,nosuid,size=1G
  app:
    image: my-app:test
    tmpfs:
      - /app/var/logs:rw,size=500M
      - /app/var/cache:rw,size=500M

Caveat: This works perfectly as long as your runners are not oversubscribed. tmpfs shifts the bottleneck from I/O to memory capacity. We did hit OOM issues early on when running too many concurrent jobs — proper memory limits per container turned out to be mandatory. Otherwise, you’re just trading slow pipelines for unstable, randomly crashing ones.

What surprised me the most was how skewed the gains were — tmpfs alone accounted for the majority of the speedup. This wasn’t an incremental improvement — it was the turning point that made the rest of the optimizations actually matter.

The Trade-off:

Lost: Durability (which we don't care about in CI).
Gained: Massive I/O speed for logging and DB operations.

2. The Artifact Trade-Off and S3 Caching

GitLab’s default GIT_STRATEGY: clone forces every concurrent job to clone the entire repository. In a massive polyglot monorepo — where pulling the code means downloading other unrelated microservices, frontend apps, and heavy assets just to run backend tests — this is brutal.

I refactored the pipeline so only the first setup job fetches the code (using GIT_DEPTH: 1). Here is where I made a conscious architectural decision: I configured the job to install PHP dependencies and package the exact state (vendor/) into a GitLab artifact.

Wait, isn't passing vendor as an artifact an anti-pattern? In massive Node/PHP projects, yes. The artifact can weigh 1GB and zipping/unzipping it takes longer than a fresh install. However, in our specific context, our vendor footprint was manageable. This is a classic case where not going strictly by the book is the better choice. Building a full Docker image just for CI dependencies would have added build time and maintenance overhead without solving the real bottleneck. Combined with migrating our GitLab cache and artifacts to a dedicated AWS S3 bucket, pulling the vendor artifact with high bandwidth and low latency was significantly faster than hitting external package registries across 10 concurrent jobs.

Pro-tip: The Authoritative Classmap. Instead of a standard install, I combined dependency resolution with autoloader optimization. By using --classmap-authoritative, I forced Composer to generate a static class map and stop any further filesystem lookups for missing classes. In a large-scale application, this eliminates thousands of redundant I/O operations during test execution.

# .gitlab-ci.yml
setup_and_build:
  stage: build
  variables:
    GIT_DEPTH: 1
  cache:
    key:
      files:
        - composer.lock
    paths:
      - vendor/
  script:
    # One command to rule them all: install + authoritative classmap
    - composer install --prefer-dist --no-progress --classmap-authoritative
  artifacts:
    paths:
      - vendor/
    expire_in: 1 hour

Downstream jobs (like parallel test suites or static analysis) no longer touch Git or external registries. They just download the optimized artifact from the build job and execute immediately. In practice, this removed an entire class of network-bound variability from the pipeline.

The Trade-off:

Lost: Clean, from-scratch dependency isolation per job.
Gained: Bypassing network bottlenecks and redundant Git/Composer operations.

3. The Parallelization Trap (And Why Paratest Failed Us)

A common piece of advice for slow PHP pipelines is: "Just use Paratest to run it in parallel!" We tried. We shaved off a few seconds from our unit tests, but integration tests crashed and burned.

Why? Because parallel execution exposes the sins of legacy test architecture. Parallelization isn’t a performance optimization—it’s an architectural test. Our integration tests lacked proper state isolation and collided over database records. True parallelization requires architectural changes: wrapping tests in database transactions that rollback automatically, or dynamically provisioning isolated database schemas per test thread. If your tests rely on shared state, parallelization doesn’t make them faster — it just makes them fail faster.

4. Honorable Mention: Watch Your Docker Limits

As we sped things up and ran more tests concurrently using our RAM disks, we hit a bizarre wall. It manifested as intermittent EMFILE ("Too many open files") errors during peak parallel load.

Our Docker daemon on the self-hosted AWS runners was still using the default file descriptor limit (ulimit -n 1024). Instead of waiting for an external DevOps team to manually edit the daemon.json on the EC2 instances, I implemented the fix directly via Infrastructure as Code in our compose file.

# docker-compose.ci.yml
services:
  app:
    # ... image and tmpfs config ...
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

This was a good reminder: once you remove one bottleneck, the next one is usually just beneath it — often at the OS level.

5. OPcache in CLI and the JIT Dilemma

Usually, OPcache is something you tune for your web servers and completely ignore in CLI environments. But since we already had our blazing-fast tmpfs RAM disks set up, I decided to run an experiment.

We run three different test suites sequentially within a single CI job. I configured PHP to dump its OPcache directly into the RAM disk (opcache.file_cache=/tmp/.ci-opcache) and enabled it for CLI. The result? The first suite primed the cache, giving the second and third suites a significant cold-start boost.

But here is where it gets interesting: the JIT trade-off.

Naturally, I tried enabling PHP 8's JIT compiler. In our environment, this significantly reduced the benefit of the OPcache file cache. While standard opcodes can still be persisted to the file cache, PHP's JIT-compiled machine code cannot. To be strict with facts: the JIT buffer lives entirely in shared memory and is tightly coupled to the runtime environment. Unlike opcodes, the actual machine code is not dumped to the OPcache file cache, so it has to be regenerated on each run.

Since CLI runs spawn separate processes, the file cache becomes the only practical way to reuse compiled opcodes across test processes. JIT effectively neutralized this advantage — the machine code still has to be regenerated in every process. Our integration and API E2E tests are heavily I/O-bound rather than CPU-bound, so the instant cold start from the RAM-backed file cache delivered a much bigger win than JIT ever could.

However, you can't just turn on OPcache in a massive project and call it a day. You have to fine-tune it, keeping in mind that with a RAM disk, you are paying with RAM twice (once for the active OPcache memory, and once for the storage in /tmp/.ci-opcache). Here is our configuration. (Note: I'm showing this in php.ini format for readability, but in our pipeline, we just passed these as -d flags directly to the PHP CLI command).

# Custom php.ini for test jobs
opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=256
opcache.interned_strings_buffer=64
opcache.max_accelerated_files=60000
opcache.validate_timestamps=0
opcache.enable_file_override=1
opcache.save_comments=1
opcache.file_update_protection=0
opcache.jit=off
opcache.jit_buffer_size=0

Why these specific values?

Capacity (max_accelerated_files=60000 & interned_strings_buffer=64): Big projects mean tons of files, especially in vendor/. If you have 40k files, you have 40k namespaces and class names. You need enough buffer space to hold that string map. PHP internally adjusts this value to a nearby prime number, so instead of chasing an exact value, I opted for a safe upper bound.
Memory (memory_consumption=256): I didn't just pull this number out of thin air. I calculated it by checking the actual size of the /tmp/.ci-opcache dump for our project, adding the buffer size, and leaving a safe headroom. Don't just set this to some massive number randomly, or you will exhaust your runner's memory fast.
Killing I/O (validate_timestamps=0): This is crucial. In CI, your code is idempotent. It doesn't change during the run. Disabling timestamp validation stops PHP from wasting I/O to check if files were modified.
Instant caching (file_update_protection=0): This disables the safety delay for caching new files. We want our fresh CI files cached instantly without any grace periods - this is just a precaution, even though files extracted from the artifact are likely older than the default 2-second threshold.

💡 The "Sanity Check" Moment: The Relative Path Trap During the final rollout, we hit a few No such file or directory errors. My first thought? "It must be the OPcache configuration." The reality? The optimization simply exposed a technical debt in our test bootstraps. A recent directory shift combined with hardcoded relative paths (../../../../) meant our tests were looking for files in the system root.

The lesson: High-performance optimizations like enable_file_override=1 and classmap-authoritative require strict path discipline — ideally absolute paths. Fixing our paths resolved the issue in our case, but it’s worth noting that enable_file_override allows OPcache to short-circuit filesystem lookups and rely entirely on its internal cache. This behavior can conflict with tools like Composer, which expect standard checks (like file_exists()) to reflect the real filesystem state, not OPcache’s internal lookup table. If you run into unexplained filesystem issues, this setting should be one of the first things to revisit.

Side note on static analysis: I also tested OPcache and JIT on our PHPStan jobs. The gains were basically within the margin of error. PHPStan 2.x is already incredibly well-optimized, and since we reuse the PHPStan result cache via GitLab artifacts anyway, it was already flying. A good reminder that not every tool needs low-level engine tweaks if it already does its own caching well.

The Trade-off:

Lost: JIT CPU optimizations.
Gained: Instant cold-starts across multiple test suites via a RAM-backed file cache.

6. Removing Cruft & The Infamous `sleep(1)`

Lastly, I audited the code itself. We removed operations that only made sense for local development (like seeding default developer accounts).

More importantly, I found a hardcoded sleep(1) in a PHP class - a legacy band-aid for a concurrency issue. In production, a 1-second delay in a queue-consuming worker might be invisible. But in the test suite, running across 60 iterations with mocked data? That's 60 seconds of pure, wasted waiting.

Conclusion

Cutting a pipeline from 35+ to around 5 minutes rarely comes from a single silver bullet. It’s about combining infrastructure pragmatism (tmpfs, smart caching, bypassing OS limits) with a deep audit of what your application is actually doing under the hood.

But the real business value? Throughput and infrastructure costs. Since our GitLab runners are self-hosted on AWS EC2, we pay for instance uptime, not serverless compute seconds.

Metric	Before	After
Pipeline Time	35 min	5 min
Pipelines / Hour (Per Node)	1.7	12.0
Node Capacity Increase	-	~700%

This allowed us to drastically scale down the total number of EC2 instances required to handle the engineering team's daily load, completely eliminating CI queue times during peak hours.

The biggest takeaway? CI performance problems are rarely about compute — they’re about eliminating unnecessary I/O, redundant work, and the hidden bottlenecks in your stack.

The final result? A lower AWS bill, a snappy 5-minute feedback loop, and a much happier engineering team.

DEV Community: Michał Iżewski

The Health Check That Watched the Wrong Thing

Where the HTTP reflex comes from

The failure the green endpoint can't see

Proof of life isn't proof of work

The signal has to come from the work

A heartbeat the loop writes, a watchdog that reads it

Why put HTTP on a worker at all

Long-running processes don't get an honest signal for free

The Health Check That Caused the Outage

The assumption every orchestrator makes

Liveness vs readiness: the same mistake, two outcomes

The same trap on EC2 + ASG + ALB

The second incident: when more instances made it worse

ECS sits in the middle

Kubernetes splits the one signal into three

Readiness is about useful work

When the platform gives you one signal

Correlated failure is the pattern

The Session ID That Wouldn't Stop Changing

The Setup: Two Backends, One Session

A Missing Escape Hatch

Reading Code Before Changing It

The Decision: Where Does the Fix Belong?

Writing the Change

The Review Process Was the Real Test

Looking Back

How I Cut Our CI/CD Pipeline from 35 to 5 Minutes

1. Bypassing AWS EBS Limits with RAM Disks (tmpfs)

2. The Artifact Trade-Off and S3 Caching

3. The Parallelization Trap (And Why Paratest Failed Us)

4. Honorable Mention: Watch Your Docker Limits

5. OPcache in CLI and the JIT Dilemma

6. Removing Cruft & The Infamous sleep(1)

Conclusion

1. Bypassing AWS EBS Limits with RAM Disks (`tmpfs`)

6. Removing Cruft & The Infamous `sleep(1)`