DEV Community: Michał Iżewski

The Health Check That Caused the Outage

Michał Iżewski — Tue, 21 Jul 2026 19:43:44 +0000

Your database dropped out for a few minutes. Your health check turned that into a full outage.

That's not a hypothetical situation I made up for effect. It happened to us twice, in two different shapes. Here's the first.

We had a health check that reached DocumentDB. One day a Security Group rollout didn't go to plan, and the app could no longer reach DocumentDB. It sat behind maybe a tenth of our endpoints, so the damage should've been small.

It wasn't. The processes never crashed, the app was alive on every instance. But the health check couldn't reach DocumentDB, so it flagged every instance as unhealthy at the same moment, and the platform started cycling the whole HTTP tier out, faster than the replacements could come back. The workers kept running with nothing to do, because the thing that fed them had gone dark. Took us a while to trace it back to one Security Group change. A dependency that maybe a tenth of the product needed had taken down all of it.

Putting that database check in the health endpoint feels responsible, and that's why so many teams do it.

The assumption every orchestrator makes

Start with what a health check is for. The load balancer, the autoscaling group, the Kubernetes control plane: all of them run the same loop. Find the unhealthy instance, take it out of rotation, put a healthy one in. Route around the broken thing.

That loop has a hidden assumption baked in. It assumes failures are independent. And it works because when one instance breaks, the others are fine, so there's somewhere to send traffic and something to replace the dead node with.

A deep health check breaks that assumption. The moment your check queries a dependency that every instance shares, you've tied the health of all of them to one external thing. When that thing goes down, every instance fails its check in the same second. The machinery built to route around one broken instance now has nothing healthy to route to. It was never designed for correlated failure, and you created correlated failure on purpose.

That's the core of it. The rest of this post is what it looks like on EC2, ECS and EKS, and the way out, which turns out to be one principle applied a little differently each time.

Liveness vs readiness: the same mistake, two outcomes

Liveness and readiness are the two jobs a health check can have: is the process alive, and should it get traffic right now. Kubernetes splits them into separate probes; most other setups blur them into one.

Where you wire the deep check decides how badly it hurts.

On readiness, all instances go NotReady together when the dependency blips. The load balancer sees zero healthy targets. Now even requests that never touched the database get nothing back, because there's no instance left in rotation to serve them. A local problem becomes a total one.

On liveness, it's worse. The orchestrator doesn't only stop sending traffic. It decides the app is dead and starts killing and restarting processes, because a dependency is slow. So at the exact moment the system is fragile, you add cold starts, fresh connection storms, and more pressure on the dependency that was already struggling. You're feeding the fire you were trying to put out.

Liveness answers one question: is this process alive and able to serve a trivial request. It doesn't touch business dependencies. If the process is alive, liveness keeps succeeding and the orchestrator leaves it running. Whether traffic is routed to it is a separate, readiness concern. Endpoints that actually need the database return 503 from the app's error handler, per request, when the database is down. That 503 comes from your code, not from the infrastructure pulling the whole instance out of service.

The same trap on EC2 + ASG + ALB

Most writing on this assumes Kubernetes. The trap is the same on plain EC2, slower and more expensive, and both of our incidents ran there: behind an ALB, with an Auto Scaling Group.

The setup is ordinary. The ALB target group runs a health check against an endpoint. The ASG has health check type = ELB, so it replaces any instance the load balancer marks unhealthy. Reasonable, until the checked endpoint reaches a shared dependency. Then one downstream failure fails every target at once, the ASG marks the whole fleet unhealthy, and it does what you configured: terminate and replace.

Here's where it turns into a loop that's hard to break from the inside. When the ASG launches a replacement, CodeDeploy runs the deployment lifecycle on that fresh instance before it enters service, validation hook included. If that validation hook also reaches the dependency, it fails too, because the dependency is still down. So the replacement never passes validation and never joins the fleet. The old instances are cycled out by the health check, and the new ones can't get in past the deploy gate. The fleet can't even rebuild itself with the artifact it already has, because the thing failing is the dependency check, not the code. That's the rotation the DocumentDB outage put us in. You don't recover until the dependency does.

An ASG doesn't understand any of this. It reasons about a single number: desired capacity minus current capacity. It sees a missing instance and launches one, with no idea that every instance is failing for the same reason. So it keeps throwing fresh machines at a wall, each one hitting it in the same spot.

One AWS detail matters here, because the earlier framing is too simple for the ALB specifically. An ALB doesn't black-hole traffic the moment every target is unhealthy. It fails open: with no healthy target in a group, it routes to all of them anyway, on the assumption that the check is the thing at fault. So on EC2 the lost capacity isn't the ALB refusing to route. It's the ASG terminating the instances the ALB flagged, and the failed replacements never coming up behind them.

The shallow version is undramatic. The ALB check hits a lightweight endpoint that touches no external dependencies, the ASG replaces an instance only when the shallow check says that instance itself is unhealthy, and the maintenance policy launches a replacement before terminating the old instance, so capacity never dips.

# ALB target group: shallow check, nothing downstream behind it
resource "aws_lb_target_group" "app" {
  health_check {
    path    = "/health"   # process-alive only, never the database
    matcher = "200"
  }
  deregistration_delay = 30   # drain in-flight requests on the way out
}

# ASG: replace when the shallow check fails, launch before terminating
resource "aws_autoscaling_group" "app" {
  health_check_type = "ELB"

  instance_maintenance_policy {
    min_healthy_percentage = 100
    max_healthy_percentage = 110
  }
}

None of it helps if the endpoint behind path = "/health" reaches a shared dependency.

The second incident: when more instances made it worse

Underneath, this outage and the DocumentDB one were the same: a connectivity problem that automation escalated into a fleet-wide one, both rooted in infra config. What made this one harder to climb out of was an autoscaler that took the cascade and pulled it tighter. It's the same machinery as the earlier replacement loop, pointed the other way: instead of swapping instances out, it kept piling new ones on.

We'd been scaling up to chew through a big backlog, and the capacity got ahead of what the database could take. As I remember it, enough workers came up to use up the database's connection limit, and RDS stopped accepting new connections. From there it was a feedback loop.

You might expect a stalled database to leave the app idle, blocked on I/O, with the CPU relaxed. It went the other way, and the scaling rule fired on CPU. I was on the sidelines for this one, so I'm reconstructing the compute side, but the shape is familiar: the box wasn't waiting quietly, it was working hard at failing. Workers piled up blocked on connections that never came, every failed request threw and was logged, and a log-shipping agent churned through the flood. None of it is useful work, but all of it burns CPU, and to a scaling rule a pinned CPU looks like demand.

So it scaled out, the worst thing it could do. Every new instance opened its own connections on boot, so the moment a slot freed up a rising instance grabbed it before anything stable could. The fleet climbed to its ceiling and kept rotating, and none of it helped, because the bottleneck was the database, not the compute.

The fix had to go where the bottleneck was, at the dependency: more connections, a bigger instance, the usual capacity work. A connection proxy would've softened it too, by pooling connections instead of letting every instance open its own, but softening is not removing. A proxy still fronts a finite database, and not every dependency has one. The cleaner answer is to stop generating the pile-on.

What stuck with me was the shape of it. More instances is the standard answer to load. Here, more instances kicked us in the face. The autoscaler assumed every new instance would help. But every new instance made the starvation worse.

ECS sits in the middle

ECS lands between the manual EC2 setup and full Kubernetes, and the trap shows up there too, under different names.

ECS gives you two health signals. The container health check, the HEALTHCHECK in your task definition, behaves like liveness: fail it and ECS replaces the task. The target group health check, run by the ALB, behaves like readiness: fail it and the task is deregistered from traffic, and replaced. Put a dependency check in either one and you get the same cascade as everywhere else, with ECS rescheduling the whole set of tasks against a dependency that's still down.

The one ECS-specific knob here is healthCheckGracePeriodSeconds, which gives a freshly started task a window before the ALB health check can mark it unhealthy. Notice what it does and doesn't cover: it buys time for a slow boot, nothing more. It says nothing about whether a downstream is healthy. Those are two different problems, and ECS has no separate signal for the second one, so the workaround later in this post applies here too.

Kubernetes splits the one signal into three

I'm still getting to know Kubernetes. I come from the EC2 and ASG side, where you wire rotation and replacement together by hand. What struck me while learning EKS is that the platform separates three questions EC2 forces you to answer with a single signal: is this alive, should it get traffic, is it still starting.

Each question maps to a different action. Kubernetes restarts a container that fails liveness. It holds traffic off a pod that hasn't passed readiness. It waits through a slow boot with the startup probe instead of killing the pod for being slow. A rolling update never sends traffic to a pod until it reports ready.

It's easy to overstate the startup probe. It covers slow startup and nothing else. None of the three probes was built to express dependency health. Each answers one thing about the instance itself: liveness, the process is alive; readiness, this pod can take traffic; startup, the boot has finished. People do wire a downstream check into readiness, and sometimes that's right, but it's something you add on top. What the platform gives you by default is those three concerns kept separate. That's the real difference from EC2, where there's one signal, so everything gets crammed into it, which is also why the deep-check mistake is so easy to make there.

Readiness is about useful work

Readiness has a real job, and it's not checking dependencies. It gates traffic during rolling deploys and graceful drain. A pod starts, the liveness check returns 200 right away because the process is up, but the caches are cold and the connection pools are empty, so the first requests would be slow or fail. Readiness holds traffic off until the pod is warmed up, and during shutdown it lets you drain in-flight requests before the process exits. It's about this instance's own state, not the health of anything downstream.

Which is why the common correction misses. "Don't check the database in liveness, check it in readiness instead" just moves the deep check. On a shared dependency the correlated failure comes along for the ride: the dependency blips, every instance reports not-ready at once, and the load balancer is back to zero healthy targets, with all the operational mess that brings.

There's at least one shape where a readiness check on a dependency is the right call. Picture a small, single-purpose service: it takes a message and writes it to a queue, and that's all it does. If the queue is gone, the service has no useful work left. It can do nothing except return 503, and every one of those failed requests is another connection thrown at a dependency that's trying to recover. A healthy-but-useless instance hammering an unhealthy dependency can be the very thing that keeps it from coming back. Here, dropping the instance from rotation simply helps. There's no partial availability to protect, because every request needs the one thing that's down, and pulling out of routing takes pressure off the dependency while it recovers.

The distinction is footprint. A readiness check on a dependency is right when that dependency is the service's whole reason to exist. It becomes the outage from the start of this post the moment the dependency sits behind only part of a larger app and gets treated as a prerequisite for all of it. So readiness answers "can I do useful work," not "is my shared downstream alive."

When the platform gives you one signal

What do you do when you can't express "alive but degraded" at the instance level, because there's one health signal and it's binary. Maybe you're on EC2 with a single ALB check, maybe your platform doesn't separate the probes the way EKS does. The move is to push the dependency awareness down one level, from the instance to the request.

Keep /health shallow. It returns 200 if the process can serve a trivial request, and that signal alone decides rotation and replacement. Then let each endpoint decide for itself. An endpoint that needs the database tries the database and returns 503 if it's down. An endpoint that doesn't need the database keeps returning 200 and keeps serving. The blip degrades only the endpoints that depend on the broken thing. The rest stay up.

This has a name worth using: partial availability. The goal is to keep serving the requests that don't need the dependency, not to pretend it's healthy. If 20% of your requests depend on the failed component, fail those 20% and keep the other 80% working. A shallow check plus per-request errors gets you that. A deep check gets you zero percent, because it takes the whole fleet down over a dependency that most requests never touch.

One honest limit of the per-request approach: it still reaches for the dependency on every request. Each call tries, waits, and fails, so a struggling dependency keeps getting poked while it recovers. The usual tool for cutting that off is the circuit breaker: past a failure threshold it stops trying, returns the error at once, and probes in the background until the dependency is back. That's a post in itself, so I'll leave it here as a name to look up.

The real fix is keeping shared dependencies out of the signal that drives replacement and routing. A shallow /health is usually the simplest way to do that. There's a blunter fallback when you can't change the check itself: set health check type = EC2, so the ASG replaces only on instance-level failure and a failing ALB check can no longer trigger termination. It breaks the churn loop, but treat it as a last resort. The ELB check is what catches a process that's running but wedged: a deadlocked app, a server returning stale 500s, something alive but no longer answering. Drop to type = EC2 and you lose that, and have to rebuild it with a watchdog of your own.

Correlated failure is the pattern

Most of us design the happy path well. The question that gets asked too rarely is "what happens when a dependency gets slow or goes away." A health check is one of the few places where that question has a default answer already baked in, and the default that feels most responsible, prove the whole stack works on every check, is the one that converts a small dependency blip into a system-wide outage.

The pattern runs through every layer. Health checks, load balancing, auto-replacement, auto-scaling: all of it assumes failures are roughly independent, and that adding or replacing capacity helps. The machinery routes around the broken instance because the other instances are fine. Tie every instance to the same shared dependency and that assumption is gone. One slow downstream takes the whole fleet together, and the system built to save you either has nothing left to route to, or it scales straight into the bottleneck and makes recovery impossible.

The checks earn their keep by staying simple. Liveness: is this process alive. Readiness: can this instance do useful work yet. Neither asks whether a shared downstream is healthy, because that's a per-request question. Answer it per request, and a broken dependency costs you the requests that needed it. Answer it at the instance level, and it costs you everything.

Get the distinction right and the same blip stays contained: only the endpoints that needed the broken dependency fail, the rest keep serving. No fleet-wide rotation, no autoscaler digging the hole deeper, no recovery fought by your own automation. The machinery built to save you finally does it.

Background workers run into the same correlated-failure pattern from another direction: an HTTP health endpoint can happily report a live process while the work loop itself is dead. That's its own post.

The Session ID That Wouldn't Stop Changing

Michał Iżewski — Tue, 07 Jul 2026 06:48:12 +0000

I was implementing a feature where the session container would track a lastActivity timestamp, updated on every authenticated request. Standard stuff. I wrote it, tested it locally with curl, and noticed something odd: I kept getting a new Set-Cookie header value on every response. Not occasionally. On every single one. A week later I was sending a pull request to mezzio/mezzio-session-cache.

The Setup: Two Backends, One Session

Our system had a constraint: two backend applications, written in different languages, sharing a single user session. One was the main PHP/Mezzio app. The other was a service in a different stack that needed to read from, and update the lastActivity timestamp on, the same session container.

There are a few ways to make polyglot session sharing work. We landed on a shared cache backend (Redis) with a well-defined session structure. Both apps could read and write through their own libraries, as long as they agreed on the storage format and the cookie name. The session ID was the contract.

That contract is the part that quietly broke.

A Missing Escape Hatch

My first instinct was the usual list of suspects. Was something calling regenerateId() in a middleware I didn't know about? Was there a logout being triggered somehow? Was a misconfigured cache layer evicting and recreating sessions?

After a bit of digging through the call stack, I ended up in the library itself. And there it was: CacheSessionPersistence was regenerating the session ID whenever the session data changed. Not on login. Not on privilege escalation. On every write.

That's when the real question hit me: why on earth would a library do that by default?

Reading Code Before Changing It

When you find behavior that surprises you in someone else's code, the wrong move is to immediately label it broken. The right move is to assume the maintainers had a reason, and find out what it was.

The reason, in this case, is session fixation.

Session fixation is a class of attack where an attacker tricks a victim into using a session ID the attacker already knows. If the session ID never changes after authentication or privilege changes, the attacker can hijack that session and act as the victim. The standard defense is to regenerate the session ID at any point where the trust level of the session changes: at minimum on login, ideally at any meaningful state transition.

Mezzio's library went further: it regenerated on every data change. Defense-in-depth taken to its logical extreme. The cost (a few extra cookie writes) is invisible to most apps. The benefit (making session fixation nearly impossible to exploit through normal usage) is real. For a single-application setup, you'd never notice the trade-off. For anything distributed, it changes the session ID from a stable contract into an ephemeral token that rotates with every write.

For our architecture, that was catastrophic. Both backends depended on the session ID remaining stable. The moment our PHP app updated a lastActivity timestamp, the library regenerated the ID on the spot. The other backend, still holding the previous ID, was now pointing at a session that no longer existed. A classic stale reference problem, except the staleness wasn't caused by TTL or eviction. It was caused by the library deliberately rotating identifiers under us. The contract broke.

I wasn't alone in hitting this. Months before my PR, an issue described the same underlying problem from a different angle: users losing their sessions when a response failed to reach the client, because the old session ID had already been destroyed the moment the session was persisted. Two production scenarios, one missing escape hatch.

The Decision: Where Does the Fix Belong?

Three paths on the table:

Patch around it in our app. Avoid writing to the session in certain code paths, or build a parallel storage mechanism that bypasses the library. Every developer on the team would need to remember the rule. Invisible discipline. The worst kind of debt.
Fork the library. Maintain our own version with regeneration disabled. Now I own a fork of a security-sensitive library. Every upstream release becomes a merge conflict review. Every security advisory becomes a question of "does that affect our fork too?"
Contribute upstream. Add an opt-in to disable auto-regeneration, keep the secure default, let maintainers decide if it's worth merging. Rejection risk is real. Review iterations are real. But the change either lands and stops being my problem, or it doesn't and I'm back to one of the other two paths anyway.

It wasn't a hard call. Patching around it costs the team forever. Forking costs me forever. Contributing upstream costs me a week, with non-zero rejection risk. The expected value of the week was lower than the certain cost of the other two options, by a wide margin.

What tipped it was the security angle. A fork of a session library is the kind of thing that would haunt me in three years when someone reports a CVE and I have to figure out whether our fork is affected, whether the upstream patch applies cleanly, and whether I still remember why we forked in the first place. Patching around the library has the same problem in a different shape: someone new joins the team, doesn't know we patched around the library, writes to the session in what looks like a perfectly reasonable place, and session sync breaks in production. It's happened to me before, on the receiving end. I wasn't going to set that up for someone else.

Only upstream made sense long-term.

The Cost: A week of focused work, plus review iterations spread over the week. Real rejection risk.

The Win: A library-level escape hatch maintained by the project, not by us. Zero ongoing maintenance cost.

Writing the Change

The shape of the contribution was deliberately small:

A new auto_regenerate config option, defaulting to true (no behavior change for existing users).
A constructor argument on CacheSessionPersistence that threaded the option through.
The factory updated to read the option from config.
Tests covering both the default behavior and the new opt-out path.
Psalm baseline updated to reflect the new types.

End result, from a user's perspective, looks like this:

// config/autoload/session.global.php
return [
    'mezzio-session-cache' => [
        'cookie_name'     => 'sid',
        'auto_regenerate' => false, // opt out of regeneration on every write
    ],
];

Config-level rather than a runtime toggle, because flipping session ID regeneration mid-request is the kind of thing that gets you into trouble you didn't ask for.

That last point about Psalm matters. When you write code in someone else's project, you're not just meeting a functional bar, you're meeting their quality bar. Mezzio uses Psalm at a strict level. The PR wasn't done when the feature worked; it was done when every static analysis check passed at the level the maintainers expected.

The Review Process Was the Real Test

The PR went through a proper review with two maintainers from the project. The feedback was sharp and useful:

Tests were requested specifically for the false case. Fair. I'd tested the default but not the new branch as thoroughly as I should have.
A type-casting suggestion on the factory: (bool) $autoRegenerate to remove a Psalm baseline entry and prevent a class of common errors.
One small inline tweak suggested via GitHub's review UI.

The standard was what stood out. Every comment was the kind of thing a careful engineer would say in your own team's review. No gatekeeping, no theater. Just "this needs to be a little more robust, here's why."

I addressed every point, pushed updated commits, and the PR was merged into the 1.13.x branch. The feature shipped in version 1.13.0.

Looking Back

The bug wasn't in our code. The fix didn't belong in our code either. The work was upstream, so I went upstream.

Two things stuck with me from this PR. First: when a library has a default that surprises you, that default is part of the API: not a bug, not laziness, but a deliberate choice the maintainers made for users who haven't thought about the edge case. Working with that choice rather than against it is the difference between contributing and forking.

Second: turning off auto_regenerate doesn't remove the session fixation defense. It moves it, from the library into the application. That's a fair trade when you've already got the rest of the controls in place (regeneration on login, on privilege escalation, on logout). It's a security regression when you don't.

The rest is the general pattern for working with other people's code: read more than you write, take time to understand the maintainers' position, don't fork what you can fix upstream. This PR just made them concrete.

The PR discussed here is mezzio/mezzio-session-cache#51, merged into the 1.13.x branch in May 2024. The auto_regenerate option shipped in version 1.13.0 and remains in the library today. A follow-up contribution from the same project, #52, implementing InitializePersistenceIdInterface, shipped in 1.14.0 a month later. The library has continued on its 1.x line; the latest release at the time of writing is 1.17.0.

How I Cut Our CI/CD Pipeline from 35 to 5 Minutes

Michał Iżewski — Mon, 01 Jun 2026 21:48:55 +0000

You push a critical hotfix, switch branches—and your CI is still running 30 minutes later. In my current project, our backend pipeline ran across a massive monorepo (~7,000 unit, ~600 integration, ~150 API E2E tests, ~200MB vendor) — and it bloated to a painful 35+ minutes.

It was killing productivity. While the team was frustrated, no one had the time to dive into the runner infrastructure. My roots are in Linux sysadmin work, and I’ve always carried that 'tinkerer' DNA into my software engineering. I don't stop at "it works" — I want to understand the full execution path and how it interacts with the OS and the hardware.

Driven by that urge to look under the hood, I decided against throwing more AWS resources at the problem. Instead, I dug into what was actually slowing things down. This isn’t a full how-to. It’s a practical look at what actually moved the needle — and what didn’t — when we cut our pipeline down to ~5 minutes.

1. Bypassing AWS EBS Limits with RAM Disks (`tmpfs`)

Initially, I suspected our test databases were the main I/O bottleneck. But a quick look at our EC2 metrics revealed the truth: the CPU was barely breaking a sweat, while disk I/O was completely maxed out. The real killer? Logs and temporary file processing. We deliberately run our CI tests in debug mode so developers get a full stack trace instantly upon failure.

Writing massive debug logs and processing large volumes of temporary files on EBS under concurrent load was grinding the runners to a halt.

The fix: I mounted both the MySQL data directories and our application's temporary storage directly to RAM disks. To squeeze every last drop of performance, I also bypassed MySQL's data safety mechanisms (since we don't care about data loss if a CI container crashes) and added a size limit to avoid OOM kills by the kernel.

# docker-compose.ci.yml
services:
  database:
    image: mysql:8.0
    command: --innodb_flush_log_at_trx_commit=0 --sync_binlog=0
    tmpfs:
      - /var/lib/mysql:rw,noexec,nosuid,size=1G
  app:
    image: my-app:test
    tmpfs:
      - /app/var/logs:rw,size=500M
      - /app/var/cache:rw,size=500M

Caveat: This works perfectly as long as your runners are not oversubscribed. tmpfs shifts the bottleneck from I/O to memory capacity. We did hit OOM issues early on when running too many concurrent jobs — proper memory limits per container turned out to be mandatory. Otherwise, you’re just trading slow pipelines for unstable, randomly crashing ones.

What surprised me the most was how skewed the gains were — tmpfs alone accounted for the majority of the speedup. This wasn’t an incremental improvement — it was the turning point that made the rest of the optimizations actually matter.

The Trade-off:

Lost: Durability (which we don't care about in CI).
Gained: Massive I/O speed for logging and DB operations.

2. The Artifact Trade-Off and S3 Caching

GitLab’s default GIT_STRATEGY: clone forces every concurrent job to clone the entire repository. In a massive polyglot monorepo — where pulling the code means downloading other unrelated microservices, frontend apps, and heavy assets just to run backend tests — this is brutal.

I refactored the pipeline so only the first setup job fetches the code (using GIT_DEPTH: 1). Here is where I made a conscious architectural decision: I configured the job to install PHP dependencies and package the exact state (vendor/) into a GitLab artifact.

Wait, isn't passing vendor as an artifact an anti-pattern? In massive Node/PHP projects, yes. The artifact can weigh 1GB and zipping/unzipping it takes longer than a fresh install. However, in our specific context, our vendor footprint was manageable. This is a classic case where not going strictly by the book is the better choice. Building a full Docker image just for CI dependencies would have added build time and maintenance overhead without solving the real bottleneck. Combined with migrating our GitLab cache and artifacts to a dedicated AWS S3 bucket, pulling the vendor artifact with high bandwidth and low latency was significantly faster than hitting external package registries across 10 concurrent jobs.

Pro-tip: The Authoritative Classmap. Instead of a standard install, I combined dependency resolution with autoloader optimization. By using --classmap-authoritative, I forced Composer to generate a static class map and stop any further filesystem lookups for missing classes. In a large-scale application, this eliminates thousands of redundant I/O operations during test execution.

# .gitlab-ci.yml
setup_and_build:
  stage: build
  variables:
    GIT_DEPTH: 1
  cache:
    key:
      files:
        - composer.lock
    paths:
      - vendor/
  script:
    # One command to rule them all: install + authoritative classmap
    - composer install --prefer-dist --no-progress --classmap-authoritative
  artifacts:
    paths:
      - vendor/
    expire_in: 1 hour

Downstream jobs (like parallel test suites or static analysis) no longer touch Git or external registries. They just download the optimized artifact from the build job and execute immediately. In practice, this removed an entire class of network-bound variability from the pipeline.

The Trade-off:

Lost: Clean, from-scratch dependency isolation per job.
Gained: Bypassing network bottlenecks and redundant Git/Composer operations.

3. The Parallelization Trap (And Why Paratest Failed Us)

A common piece of advice for slow PHP pipelines is: "Just use Paratest to run it in parallel!" We tried. We shaved off a few seconds from our unit tests, but integration tests crashed and burned.

Why? Because parallel execution exposes the sins of legacy test architecture. Parallelization isn’t a performance optimization—it’s an architectural test. Our integration tests lacked proper state isolation and collided over database records. True parallelization requires architectural changes: wrapping tests in database transactions that rollback automatically, or dynamically provisioning isolated database schemas per test thread. If your tests rely on shared state, parallelization doesn’t make them faster — it just makes them fail faster.

4. Honorable Mention: Watch Your Docker Limits

As we sped things up and ran more tests concurrently using our RAM disks, we hit a bizarre wall. It manifested as intermittent EMFILE ("Too many open files") errors during peak parallel load.

Our Docker daemon on the self-hosted AWS runners was still using the default file descriptor limit (ulimit -n 1024). Instead of waiting for an external DevOps team to manually edit the daemon.json on the EC2 instances, I implemented the fix directly via Infrastructure as Code in our compose file.

# docker-compose.ci.yml
services:
  app:
    # ... image and tmpfs config ...
    ulimits:
      nofile:
        soft: 65536
        hard: 65536

This was a good reminder: once you remove one bottleneck, the next one is usually just beneath it — often at the OS level.

5. OPcache in CLI and the JIT Dilemma

Usually, OPcache is something you tune for your web servers and completely ignore in CLI environments. But since we already had our blazing-fast tmpfs RAM disks set up, I decided to run an experiment.

We run three different test suites sequentially within a single CI job. I configured PHP to dump its OPcache directly into the RAM disk (opcache.file_cache=/tmp/.ci-opcache) and enabled it for CLI. The result? The first suite primed the cache, giving the second and third suites a significant cold-start boost.

But here is where it gets interesting: the JIT trade-off.

Naturally, I tried enabling PHP 8's JIT compiler. In our environment, this significantly reduced the benefit of the OPcache file cache. While standard opcodes can still be persisted to the file cache, PHP's JIT-compiled machine code cannot. To be strict with facts: the JIT buffer lives entirely in shared memory and is tightly coupled to the runtime environment. Unlike opcodes, the actual machine code is not dumped to the OPcache file cache, so it has to be regenerated on each run.

Since CLI runs spawn separate processes, the file cache becomes the only practical way to reuse compiled opcodes across test processes. JIT effectively neutralized this advantage — the machine code still has to be regenerated in every process. Our integration and API E2E tests are heavily I/O-bound rather than CPU-bound, so the instant cold start from the RAM-backed file cache delivered a much bigger win than JIT ever could.

However, you can't just turn on OPcache in a massive project and call it a day. You have to fine-tune it, keeping in mind that with a RAM disk, you are paying with RAM twice (once for the active OPcache memory, and once for the storage in /tmp/.ci-opcache). Here is our configuration. (Note: I'm showing this in php.ini format for readability, but in our pipeline, we just passed these as -d flags directly to the PHP CLI command).

# Custom php.ini for test jobs
opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=256
opcache.interned_strings_buffer=64
opcache.max_accelerated_files=60000
opcache.validate_timestamps=0
opcache.enable_file_override=1
opcache.save_comments=1
opcache.file_update_protection=0
opcache.jit=off
opcache.jit_buffer_size=0

Why these specific values?

Capacity (max_accelerated_files=60000 & interned_strings_buffer=64): Big projects mean tons of files, especially in vendor/. If you have 40k files, you have 40k namespaces and class names. You need enough buffer space to hold that string map. PHP internally adjusts this value to a nearby prime number, so instead of chasing an exact value, I opted for a safe upper bound.
Memory (memory_consumption=256): I didn't just pull this number out of thin air. I calculated it by checking the actual size of the /tmp/.ci-opcache dump for our project, adding the buffer size, and leaving a safe headroom. Don't just set this to some massive number randomly, or you will exhaust your runner's memory fast.
Killing I/O (validate_timestamps=0): This is crucial. In CI, your code is idempotent. It doesn't change during the run. Disabling timestamp validation stops PHP from wasting I/O to check if files were modified.
Instant caching (file_update_protection=0): This disables the safety delay for caching new files. We want our fresh CI files cached instantly without any grace periods - this is just a precaution, even though files extracted from the artifact are likely older than the default 2-second threshold.

💡 The "Sanity Check" Moment: The Relative Path Trap During the final rollout, we hit a few No such file or directory errors. My first thought? "It must be the OPcache configuration." The reality? The optimization simply exposed a technical debt in our test bootstraps. A recent directory shift combined with hardcoded relative paths (../../../../) meant our tests were looking for files in the system root.

The lesson: High-performance optimizations like enable_file_override=1 and classmap-authoritative require strict path discipline — ideally absolute paths. Fixing our paths resolved the issue in our case, but it’s worth noting that enable_file_override allows OPcache to short-circuit filesystem lookups and rely entirely on its internal cache. This behavior can conflict with tools like Composer, which expect standard checks (like file_exists()) to reflect the real filesystem state, not OPcache’s internal lookup table. If you run into unexplained filesystem issues, this setting should be one of the first things to revisit.

Side note on static analysis: I also tested OPcache and JIT on our PHPStan jobs. The gains were basically within the margin of error. PHPStan 2.x is already incredibly well-optimized, and since we reuse the PHPStan result cache via GitLab artifacts anyway, it was already flying. A good reminder that not every tool needs low-level engine tweaks if it already does its own caching well.

The Trade-off:

Lost: JIT CPU optimizations.
Gained: Instant cold-starts across multiple test suites via a RAM-backed file cache.

6. Removing Cruft & The Infamous `sleep(1)`

Lastly, I audited the code itself. We removed operations that only made sense for local development (like seeding default developer accounts).

More importantly, I found a hardcoded sleep(1) in a PHP class - a legacy band-aid for a concurrency issue. In production, a 1-second delay in a queue-consuming worker might be invisible. But in the test suite, running across 60 iterations with mocked data? That's 60 seconds of pure, wasted waiting.

Conclusion

Cutting a pipeline from 35+ to around 5 minutes rarely comes from a single silver bullet. It’s about combining infrastructure pragmatism (tmpfs, smart caching, bypassing OS limits) with a deep audit of what your application is actually doing under the hood.

But the real business value? Throughput and infrastructure costs. Since our GitLab runners are self-hosted on AWS EC2, we pay for instance uptime, not serverless compute seconds.

Metric	Before	After
Pipeline Time	35 min	5 min
Pipelines / Hour (Per Node)	1.7	12.0
Node Capacity Increase	-	~700%

This allowed us to drastically scale down the total number of EC2 instances required to handle the engineering team's daily load, completely eliminating CI queue times during peak hours.

The biggest takeaway? CI performance problems are rarely about compute — they’re about eliminating unnecessary I/O, redundant work, and the hidden bottlenecks in your stack.

The final result? A lower AWS bill, a snappy 5-minute feedback loop, and a much happier engineering team.

DEV Community: Michał Iżewski

The Health Check That Caused the Outage

The assumption every orchestrator makes

Liveness vs readiness: the same mistake, two outcomes

The same trap on EC2 + ASG + ALB

The second incident: when more instances made it worse

ECS sits in the middle

Kubernetes splits the one signal into three

Readiness is about useful work

When the platform gives you one signal

Correlated failure is the pattern

The Session ID That Wouldn't Stop Changing

The Setup: Two Backends, One Session

A Missing Escape Hatch

Reading Code Before Changing It

The Decision: Where Does the Fix Belong?

Writing the Change

The Review Process Was the Real Test

Looking Back

How I Cut Our CI/CD Pipeline from 35 to 5 Minutes

1. Bypassing AWS EBS Limits with RAM Disks (tmpfs)

2. The Artifact Trade-Off and S3 Caching

3. The Parallelization Trap (And Why Paratest Failed Us)

4. Honorable Mention: Watch Your Docker Limits

5. OPcache in CLI and the JIT Dilemma

6. Removing Cruft & The Infamous sleep(1)

Conclusion

1. Bypassing AWS EBS Limits with RAM Disks (`tmpfs`)

6. Removing Cruft & The Infamous `sleep(1)`