Your CI Pipeline Is Making GitHub's Outages Worse. Here's How to Stop.

Kenneth Onyenuloya — Wed, 19 Aug 2026 19:41:54 +0000

On August 17, 2026, GitHub went down for roughly 7.5 hours. Partway through the incident, GitHub's engineers made a decision that tells you everything about how outages actually spiral: they disabled authentication-token retries. Not added more capacity. Not scaled up. They turned retries off, because the retries themselves were adding load to a system that was already struggling to stay up.

That's the part of the story most postmortems gloss over, and it's worth sitting with. An outage doesn't just happen to a service — it gets amplified by everyone downstream of it, reacting the same way at the same time. Thousands of pipelines, all failing, all retrying, all adding more load to the exact thing that's already drowning.

Your CI pipeline was almost certainly one of them.

The mechanic: why retries make things worse, not better

Retry logic exists because most failures are transient — a dropped packet, a momentary timeout, a brief 503. Retry once or twice, it usually clears up, nobody notices. That's the theory, and for isolated failures it works.

It stops working the moment the failure isn't isolated. If an upstream service is genuinely degraded, every client hitting it fails, and every client's retry logic kicks in at roughly the same time. Instead of one request, the struggling service now gets three, five, ten times the load — from retries alone. This is the "thundering herd" or "retry storm" pattern, and it's a well-known failure amplifier in distributed systems. GitHub's own mitigation on Aug 17 was a live demonstration of it.

The fix that's been known in distributed systems for over a decade is the circuit breaker pattern (Martin Fowler wrote the canonical description of it): stop calling a service that's clearly down, wait, then cautiously test if it's recovered before resuming full traffic. It's the software equivalent of the fuse box in your house — when something's wrong, cut the circuit rather than let it keep drawing power and making things worse.

CI pipelines mostly don't do this. They retry hard, then give up, then the next run does the exact same thing from scratch — with zero memory of the fact that this dependency has been failing for the last two hours.

What's on the Marketplace today — and the actual gap

I went looking for a GitHub Action that implements a real circuit breaker before building one. There's no shortage of retry actions — nick-fields/retry and Wandalen/wretry.action are both solid, and some already support jittered backoff delays. If your problem is "this one flaky step needs a couple of retries," those are the right tools.

But none of them are circuit breakers, because none of them remember anything past the current job. Every run starts from zero. If a dependency has been down for two hours, run #47 retries into the same wall that runs #1 through #46 already hit, wastes its full retry budget, times out slowly, and fails — and so does #48. There's a GitHub community feature request asking for exactly this ("auto-pause a workflow that's failed 5 times in a row instead of burning through runner minutes") that's been sitting unfulfilled. That's the gap.

Retry-Storm Circuit Breaker

I built a GitHub Action that closes that gap: a circuit breaker that remembers failures across workflow runs, not just within one job.

- name: Call flaky payments API
  id: breaker
  uses: kennethonyenuloya/retry-storm-breaker@v1
  with:
    key: payments-api
    run: curl -sf https://payments.example.com/health
    failure-threshold: 5
    window-size: 10
    open-duration-seconds: 300
    half-open-max-calls: 1

It's a standard three-state breaker — Closed, Open, Half-Open:

Closed: normal operation, calls go through, failures are tracked in a rolling window.
Open: after enough failures in that window, it stops making real calls entirely for a cooldown period and fails fast instead — no more slow timeouts, no more piling on.
Half-Open: after cooldown, it lets through a small number of probe calls. Succeed, and it closes back up. Fail, and it goes straight back to Open, cooldown reset.

The interesting engineering problem — the reason this doesn't already exist as a trivial wrapper — is state persistence. GitHub's Actions cache API only lets you write a given key once; a second save to the same key returns a 409 conflict. That's fine for caching node_modules, useless for anything you need to update every run.

The workaround: save state under a timestamp-suffixed key every time (cb-state-payments-api-2026-08-19T02:14:00Z), and restore using a prefix match instead of an exact key. GitHub's cache restore already falls back to the most recently created entry matching a prefix, so this gives mutable, "last write wins" state on top of an API that's fundamentally immutable — using only the default GITHUB_TOKEN cache permissions. No external database, no secrets to manage, no extra scopes to grant.

What it deliberately doesn't do

Worth being precise, because I don't want this to overclaim: this does not fix GitHub's outages. Nothing running in your own workflow can. What it fixes is your side of the interaction — it stops your pipeline from being one more source of load piling onto a struggling dependency, and it stops your own CI minutes from being burned re-discovering the same outage every single run.

It also doesn't ship Slack, Teams, or Discord notifications, on purpose. That space is already well served by mature, dedicated actions, and bundling notification logic here would mean taking on webhook and secret-handling liability for no real benefit over composing with what already exists. Instead, it exposes a circuit-state output you branch on:

- name: Notify Slack when breaker trips
  if: steps.breaker.outputs.circuit-state == 'open'
  uses: rtCamp/action-slack-notify@v2
  env:
    SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
    SLACK_MESSAGE: "Circuit breaker tripped: payments-api is down."

Swap in whatever notifier your team already trusts. The action's job is producing a clean signal, not owning your alerting stack.

It's also scoped to a single step, not a whole workflow — auto-pausing an entire pipeline is a bigger blast-radius decision that needs its own permissions and its own safety guardrails, and is a legitimate future direction rather than something bolted onto v1.

What this would have looked like on Aug 17

It's easy to wave at "reduces load, saves CI minutes" without showing what that means. So walk through it concretely.

Without the breaker, every run touching something behind GitHub's auth layer during those 7.5 hours would independently retry a few times, time out slowly, and fail. Then the next commit triggers another run. Same thing, again — for the full duration of the outage, every single run rediscovers the outage from scratch and pays the full cost of finding out: slow timeouts, burned CI minutes, and a PR list full of red checks with no signal of "this is a known outage" versus "your code broke something."

With the breaker wired around that step:

The first few runs still fail. No circuit breaker has a crystal ball — it needs to actually see failures accumulate (say, 5 failures in a rolling window of 10) before it has evidence this is sustained trouble, not a one-off blip.
Once it trips, every subsequent run during the outage fails instantly, with an explicit message: Circuit breaker OPEN — skipping execution, cooldown in progress. A 2-second fail instead of a slow multi-minute timeout, repeated across however many runs your team would have triggered over 7.5 hours.
If you branch the circuit-state output into a Slack notification, the trip fires once — not once per failed run. The team finds out from a single clear message instead of a wall of unexplained red X's.
The next time a run happens to execute that step — triggered by a push, a scheduled cron, or a manual re-run, same as any other workflow trigger — the breaker checks whether the cooldown has elapsed and, if so, quietly probes once. Succeed, and it closes and resumes normal behavior for future runs. The breaker doesn't create runs or re-trigger anything itself; it just makes the right call, automatically, within whatever run happens to occur next — so nobody has to remember "GitHub's back up, go re-run that failed job."

What it would not have done: made your pipeline succeed during the outage. If a step genuinely depends on GitHub's API being up, no circuit breaker changes that — GitHub being down is still GitHub being down. What changes is the cost and clarity of that downtime: fast, obvious, single-notification failures instead of slow, repeated, confusing ones.

Observability, opt-in

If you want to see breaker trips on a dashboard rather than just in Actions logs, set an OTLP endpoint and it'll emit a span per invocation (ci.step.circuit_breaker, with state-transition attributes) to whatever collector you already run. Left empty by default — no network calls happen unless you explicitly turn it on.

Try it

kennethonyenuloya/retry-storm-breaker@v1

Marketplace listing: github.com/marketplace/actions/retry-storm-circuit-breaker

If your pipeline calls anything outside your own infrastructure — a third-party API, a package registry, an internal service that occasionally has a bad day — this is worth two minutes to wire in before the next outage, not during it.

DEV Community: Kenneth Onyenuloya