Give Your CI Pipeline an Error Budget Before Adding AI Triage

#devops #ci #observability #ai

An AI-generated explanation cannot rescue an unhealthy CI pipeline. If jobs wait unpredictably, runners disappear, retries hide flakes, or logs lack revision metadata, the model is summarizing unreliable evidence.

Start with four service-level indicators:

queue_seconds = job_started_at - job_queued_at
execution_seconds = job_finished_at - job_started_at
infra_failure_rate = infrastructure_failures / completed_jobs
retry_recovery_rate = passed_after_retry / retried_jobs

Split failures into code, test_flake, infrastructure, dependency, policy, and unknown. Do not let unknown become permanent.

A practical objective

For a protected-branch workflow, one starting target could be:

95% of jobs begin within 120 seconds;
95% finish within 15 minutes;
fewer than 1% end in infrastructure failure;
fewer than 2% pass only after an unchanged retry;
every failure includes repository, revision, workflow version, runner image, attempt, and artifact links.

These are example thresholds, not universal standards. Derive yours from developer wait tolerance, release risk, runner capacity, and current distributions.

If a week has 1,000 protected-branch runs and the infrastructure-failure objective allows 10, the eleventh failure should change priorities. Pause runner-image changes, investigate the dominant signature, or reduce concurrency until recovery.

Instrument queue and execution separately. Scaling executors cannot fix serialized dependency downloads; caching cannot fix insufficient runner capacity. Record retries as new attempts linked to the original job, never overwrite the initial failure.

Before AI triage, require structured evidence:

{
  "revision": "abc123",
  "workflow_revision": "def456",
  "runner_image": "ubuntu-24.04@sha256:...",
  "attempt": 2,
  "previous_attempt": 1,
  "exit_code": 1,
  "failure_class": "test_flake"
}

Then an AI summary can cite stable logs and metadata. Evaluate it on classification accuracy, missing-evidence rate, analyst correction time, and dangerous false reassurance—not prose quality.

The public MonkeyCode repository describes AI tasks, development environments, and automated PR/MR review. CI evidence can feed that category of workflow, but this article reports no MonkeyCode deployment measurement or integration test.