Attempt A times out. Attempt B is routed to another provider and finishes first. Then A returns and overwrites B. Availability improved; correctness did not.
This event order is plausible during the official July 25 timeline. OpenAI reports incident one beginning at 09:17:49 UTC, mitigation monitoring at 10:02:52, and resolution at 11:08:36. Incident two began at 11:35:24 and was identified with elevated errors and mitigation in progress when researched. The official status service showed Partial System Degradation at that time. None of this establishes root cause, global impact, exact users affected, or final recovery for the second incident.
The invariant is: at most one provider attempt may commit an outcome for a task epoch, and its output must pass that epoch's semantic gate.
Separate execution from commit authority
coordinator -> lease(epoch=7) -> attempt A
-> lease expires
-> lease(epoch=8) -> attempt B
A returns -> commit(epoch=7) -> REJECT
B returns -> validate(epoch=8) -> conditional commit
A lease limits when work should proceed, but clocks and delayed messages mean it cannot prevent a stale worker from writing. The durable store must fence every commit with a monotonically increasing epoch.
-- unexecuted protocol sketch
UPDATE task_outcomes
SET output = :validated_output,
committed_epoch = :epoch,
state = 'committed'
WHERE task_id = :task_id
AND current_epoch = :epoch
AND state = 'open';
Only one row update authorizes publication. Provider callbacks never write application state directly; they append attempt results containing task ID, epoch, provider, request digest, and validation record.
Minimal failure fixture
Build a simulator with these scheduled events:
t=0 issue A at epoch 7
t=5 client timeout
t=6 advance to epoch 8
t=7 issue B
t=9 B returns valid output
t=10 B commits
t=12 A returns different valid-looking output
t=13 A commit is fenced
These are proposed inputs and expected behavior, not executed test results. Add permutations where A returns before B, both responses fail validation, the coordinator crashes after epoch advancement, and the store acknowledges a commit after the caller times out.
Testable properties:
- no history contains two committed outcomes for one task;
- a lower epoch never changes a higher epoch's state;
- timeout alone never marks an attempt absent;
- recovery can reconstruct the winner from durable records;
- provider switching cannot bypass semantic validation.
The final property matters because multi-provider fallback has semantic risks. Models can differ in tool-call interpretation, structured outputs, context treatment, and generated side effects. Fencing solves write ordering, not meaning. Maintain provider-specific fixtures and reject fallback for tasks whose acceptance oracle is subjective or unavailable.
| Design | Availability | Consistency cost | Residual risk |
|---|---|---|---|
| no failover | lower | simple | prolonged unavailability |
| race both | high | expensive fencing | semantic divergence |
| timeout then fence | moderate | epoch coordination | delayed completion |
| human-selected route | slower | review workflow | reviewer error |
Inspect both hosted and source routes
The overseas MonkeyCode online offering currently displays “Start free.” Its official README describes managed server-side cloud environments with build, test, and preview alongside integrated models. Call it free to start, and verify current model/server quotas, regions, plus uptime or SLA terms in the console because those parameters may change.
The official MonkeyCode GitHub project is AGPL-3.0 open source. At reviewed main commit 18baaf54937a65a7d47f1f9d83dd808777aa6cea, the README lists built-in management for development environments, models, tasks, and requirements. For a systems evaluation, the source is an inspection and self-hosting exit route; overseas hosting is a separate managed route to probe with disposable tasks. Neither creates guaranteed independent failure domains, and hosted MonkeyCode reliability was not tested.
I would recommend it for evaluation only after defining the epoch and commit boundary outside provider-specific output. A second route without fencing can make an outage less visible by replacing it with silent inconsistency.
Which event order breaks your invariant: stale success, duplicate callback, coordinator restart, or semantically different winner? The protocol should reject, replay, or compensate explicitly.
Disclosure: I'm a MonkeyCode user sharing my own experience, not affiliated with the project.
AI assistance disclosure: This article was drafted with AI assistance and reviewed against the cited primary sources.
Top comments (0)