09:17:49 UTC: a provider incident starts. The operational question is not “is the status page red?” It is “which measured threshold changes routing, admission, and customer communication in our service?”
The evidence timeline is bounded. OpenAI's first incident record reports mitigation monitoring at 10:02:52 and resolution at 11:08:36 on July 25, 2026. A second record starts at 11:35:24. Research observed it as identified, with elevated errors and mitigation underway, while overall OpenAI status was Partial System Degradation. This does not establish root cause, global scope, affected-user count, or final restoration for the later incident.
Define the SLI at your boundary
Measure accepted task outcomes, not raw provider HTTP success:
availability SLI = valid, non-duplicate outcomes before deadline
/ eligible submitted tasks
Exclude user cancellations and declared maintenance only through a documented policy. Count schema-invalid, semantically rejected, and late outputs as unavailable for this SLI. Keep provider transport errors as a diagnostic dimension rather than the numerator.
A proposed objective might be 99.5% over 28 days for eligible low-authority tasks, with a separate five-minute burn alert. That threshold is an example, not a measured recommendation. Select it from user tolerance and error budget economics.
Incident runbook
# unexecuted example
trigger:
local_valid_outcome_ratio_5m_below: 0.90
minimum_eligible_tasks: 30
actions:
- freeze automatic retries
- mark unresolved attempts unknown
- admit only read_only task classes
- compare local telemetry with official status
fallback_gate:
require_semantic_fixture_pass: true
prohibit_external_actions: true
recovery:
canary_count: 5
observation_minutes: 15
require_no_unknown_attempts: true
Triage: separate DNS, TLS, authentication, rate limit, timeout, provider 5xx, invalid output, and local queue saturation. Record task_id, attempt_id, provider, route, deadline, response class, validation result, and status-page observation time.
Contain: stop retry amplification, cap queue age, and preserve unknown attempts. If work cannot meet its deadline, reject it explicitly rather than letting the queue manufacture latency.
Degrade: allow a manual path or an alternate provider only for preclassified work. Multi-provider fallback has semantic risks because models differ in instructions, tools, structured output, and side effects. A transport canary cannot certify task equivalence.
Recover: use low-authority canaries, then gradually restore admission. The provider's resolved label is evidence, but your SLI and unresolved-attempt ledger decide when your service is healthy. Rollback by closing new admission and returning to the prior route; do not delete incident evidence.
Drill before an outage
Inject an adapter timeout after submission, a malformed successful response, and a delayed completion after fallback begins. Expected—not observed—results are unknown, validation rejection, and exactly one accepted winner. Clean up synthetic tasks and verify dashboards do not mix them into production denominators.
Evaluate another operating path carefully
The overseas MonkeyCode Try Online currently displays “Start free.” Its README describes managed server-side cloud environments supporting build, test, and preview with integrated models. It is currently free to start; exact model/server quotas, regions, and uptime or SLA commitments may change and require confirmation in the console.
The official MonkeyCode repository is AGPL-3.0 open source. On main commit 18baaf54937a65a7d47f1f9d83dd808777aa6cea, the README describes built-in development environment, model, task, and requirement management. An SRE evaluation can use source inspection and self-hosting as an exit path while using overseas hosting for a limited trial. That topology is not automatically outage-proof, and hosted MonkeyCode reliability was not tested.
I would add both options to the runbook as evaluated routes only after collecting their own SLIs. Provider diversity without independent failure domains, telemetry, and semantic gates is a diagram, not resilience.
Disclosure: I'm a MonkeyCode user sharing my own experience, not affiliated with the project.
AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.
Top comments (0)