DEV Community

Robin
Robin

Posted on

Make Long-Running Agent Events Monotonic Before Making Them Real-Time

#architecture #ai #distributedsystems #reliability

Real-time delivery does not guarantee a correct timeline. WebSockets reconnect. Brokers redeliver. Workers race with cancellation. A client may receive event 42, then 40, then a snapshot generated after 45.

For long-running agent tasks, define convergence before optimizing animation latency.

Give each operation an ordered log 

type OperationEvent = {
  operationId: string;
  sequence: number;
  eventId: string;
  type: "accepted" | "started" | "progress" | "needs_input" | "completed" | "failed" | "cancelled";
  occurredAt: string;
  payload: unknown;
};
Enter fullscreen mode Exit fullscreen mode

The producer allocates a strictly increasing sequence per operation in the same durable transaction that appends the event. eventId deduplicates delivery; sequence detects order and gaps. Wall-clock timestamps are for display and diagnosis, not ordering.

Consumer rules:

sequence <= applied_sequence   -> ignore duplicate or stale event
sequence == applied_sequence+1 -> apply and advance
sequence > applied_sequence+1  -> buffer briefly and request replay
Enter fullscreen mode Exit fullscreen mode

Terminal states need an invariant: after completed, failed, or cancelled, later non-audit state transitions are rejected. Cancellation is a request until the authoritative terminal event is committed.

Reconnect with snapshots plus replay

Return a snapshot containing operationId, materialized state, and throughSequence. Then replay events beginning at throughSequence + 1. The server must create the snapshot from a consistent log position; otherwise the client can miss the gap between two independent reads.

Compact old progress events only after a snapshot is durable and consumers no longer need them for audit. Retain terminal and authority-changing events according to policy.

Inject duplicate delivery, reordering, a missing sequence, reconnect during snapshot, worker death after a side effect but before event commit, cancellation racing with completion, and two clients reconnecting at different offsets.

Validate these properties:

  • every client eventually reaches the authoritative terminal state;
  • each side effect has an idempotency key independent of delivery attempts;
  • no client applies the same event twice;
  • gaps become observable rather than silently skipped;
  • late progress cannot reopen a terminal task.

The public MonkeyCode repository describes long-running AI tasks, managed environments, and synchronized PC/mobile workflows. Those capabilities make ordered recovery a relevant evaluation question, but this article does not claim that MonkeyCode uses this schema or report a reliability test.

Disclosure: I contribute to the MonkeyCode project. Product context is based on public documentation; the protocol is an independent reference design.

Real-time is a transport property. Correct recovery is a system property. Build the second one first.

Top comments (0)