DEV Community

RobustTrueTry
RobustTrueTry

Posted on

When GitHub Goes Down, Your CI Shouldn't Go With It

GitHub publishes monthly availability reports, and every report lists incidents that degraded real services. If your pipeline clones, fetches, or calls GitHub blindly, each of those incidents becomes your outage too.

By the end of this article you'll be able to poll the status API, fail fast with a clear message, and design jobs that survive short API brownouts.

The real failure mode

Most teams treat GitHub as a single dependency. A checkout fails, the job fails, the whole run fails. But the status API, git operations, and the REST API fail independently. An incident in one surface doesn't have to stall every job.

The escape route has three parts:

  • Detect the incident before wasting runner minutes
  • Distinguish "GitHub is broken" from "our workflow is broken"
  • Reduce how many steps touch the network at all

Query the status API first

GitHub exposes a machine-readable status feed that returns per-component health. Check it before running expensive setup.

curl -sf https://www.githubstatus.com/api/v2/status.json \
  | python3 -c "
import json, sys
s = json.load(sys.stdin)['status']
print(s['indicator'])
"
Enter fullscreen mode Exit fullscreen mode

The indicator field is one of none, minor, major, or critical. Treat anything above none as a signal to change behavior, not necessarily to abort.

A small gate step in GitHub Actions makes this decision explicit:

jobs:
  health-gate:
    runs-on: ubuntu-latest
    outputs:
      degraded: ${{ steps.check.outputs.degraded }}
    steps:
      - id: check
        run: |
          indicator=$(curl -sf https://www.githubstatus.com/api/v2/status.json \
            | jq -r '.status.indicator')
          if [ "$indicator" = "none" ]; then
            echo "degraded=false" >> "$GITHUB_OUTPUT"
          else
            echo "degraded=true" >> "$GITHUB_OUTPUT"
          fi
Enter fullscreen mode Exit fullscreen mode

Downstream jobs read needs.health-gate.outputs.degraded and choose a fast, cheap path instead of retrying into a wall.

Decide what to do, per job

Skipping everything is rarely right. Deploys touching GitHub Packages should pause. Tests that only need cached dependencies can still run.

Job type During a major incident Why
Unit tests with cached deps Run No GitHub calls at runtime
Clone-heavy integration tests Defer Checkout is the failing surface
Deploys that publish packages Block Partial publishes are worse than none
Docs builds Skip Low value during an incident

The point is not the specific rows. The point is that you made the decision once, in code, instead of paging someone to decide it at 3 a.m.

Cut GitHub out of the hot path

The strongest fix is reducing dependence in the first place.

  • Pin and cache dependencies with actions/cache and a lockfile so test jobs never hit a registry on a warm run
  • Use actions/checkout with fetch-depth: 1 unless you genuinely need history
  • Mirror critical container images to a registry you control, such as GHCR plus a fallback mirror
  • Avoid calling the REST API in workflow steps when a local git command answers the same question

Each of these shrinks the set of jobs that an API incident can touch.

Retry the right way

When you must call the GitHub API, retry only transient failures and respect the headers the API already gives you. The REST API returns Retry-After on rate-limit responses; honoring it beats inventing your own backoff curve.

import time
import requests


def gh_get(url, token, max_tries=4):
    headers = {
        "Authorization": f"Bearer {token}",
        "Accept": "application/vnd.github+json",
    }
    for attempt in range(max_tries):
        resp = requests.get(url, headers=headers, timeout=10)
        if resp.status_code == 403 and "Retry-After" in resp.headers:
            time.sleep(int(resp.headers["Retry-After"]))
            continue
        if resp.status_code in (502, 503) and attempt < max_tries - 1:
            time.sleep(2 ** attempt)
            continue
        resp.raise_for_status()
        return resp.json()
    raise RuntimeError(f"GitHub API unavailable after {max_tries} tries: {url}")
Enter fullscreen mode Exit fullscreen mode

Retries on 502 and 503 cover the transient degradation that availability reports describe. A 404 should fail immediately; retrying it just hides a real bug in your URL.

What still goes wrong

The status API itself can lag behind reality, and component-level status can look green while one region struggles. So keep a second signal: if more than a few consecutive jobs fail at checkout across unrelated repositories, suspect the platform, not your code.

Also watch out for over-broad gates. If your gate skips every job on a minor indicator, a routine degraded-performance notice silently stops your release train. Tie the response level to job type, as in the table above.

Key takeaways

  • GitHub's availability reports are a reminder that the platform fails in parts; your pipeline should too.
  • Check the status API in a gate job and branch on the indicator value instead of letting failures surface as opaque checkout errors.
  • Retry only transient errors (502, 503, rate limits with Retry-After) and fail fast on everything else.
  • Cache dependencies, shallow-clone, and mirror critical artifacts so fewer jobs depend on GitHub being healthy at run time.
  • Rehearse the degraded path once a quarter; a gate you have never run during a real incident is untested code.

Source

GitHub availability report: October 2026 documents the incidents that motivated this; I added the status-API gate, retry policy, and pipeline design patterns the report leaves out.

Support this work

These write-ups are researched and published with no paywall, sponsor, or tracking. If one saved you an afternoon, a small tip keeps them coming.

USDT, USDC or USDD ยท TRC-20 (Tron)

TFTNsfyomKrnUutRjBTGVULp19ByW29KbY
Enter fullscreen mode Exit fullscreen mode

Top comments (0)