An agent session stops updating at 14:07. Did the model stall, the browser stream disconnect, the managed environment fail, or the task finish while the client missed the final event?
Without session-level evidence, those incidents look identical.
Here is an OpenTelemetry-inspired acceptance protocol for MonkeyCode SaaS. It does not claim that MonkeyCode exports OpenTelemetry data.
Define one traceable task
Use a disposable repository:
Change
/healthfrom{"status":"up"}to{"status":"ok"}, update its test, run that test, and report changed files.
evaluation_id: "mc-obs-001"
base_commit: "<full SHA>"
requirement_hash: "<SHA-256>"
expected_files: ["src/health.*", "test/health.*"]
timeout_seconds: 600
Ask whether UI, API, logs, or support evidence can reconstruct this logical trace:
agent.task
├── requirement.accept
├── environment.prepare
├── model.request
├── workspace.command
├── artifact.persist
└── task.complete
For every available event, retain task/session identity, timestamp, status, attempt number, base commit, and artifact reference. Do not request chain-of-thought or secret-bearing payloads.
Drill the streaming boundary
During the task, disconnect the browser for 30 seconds. After reconnecting, answer:
- Is it still the same task?
- Can missed, replayed, or duplicated events be distinguished?
- Is terminal state unambiguous?
- Does the patch agree with that state and base commit?
Then run a controlled failure using a nonexistent test target. Expected outcome: visible failure, no false success.
terminal_state: "failed"
failed_stage: "workspace.command"
error_class: "test_target_missing"
artifact_created: false
retry_attempts: "<count or unknown>"
Write unknown when evidence is unavailable. Unknown is a finding, not an invitation to guess.
Adoption gates
| Gate | Pass condition |
|---|---|
| Correlation | Task, session, base, artifact connect |
| Timeline | Major transitions are ordered |
| Reconnect | Current state is unambiguous |
| Failure | Failed stage and useful error visible |
| Retry | Attempts are distinguishable |
| Hygiene | No credentials in evidence |
| Retention | Evidence lifetime is known |
MonkeyCode's README documents an online platform with managed environments and model/task/requirement management, making these operational questions relevant. It does not document or promise the fields above.
Sources: MonkeyCode repository and SaaS.
Limitations: this does not measure model quality, SLOs, throughput, cost, or security posture.
Disclosure: I'm a MonkeyCode user sharing my own experience, not affiliated with the project. This is one of several independently useful technical articles published by accounts managed by the same operator; it is not an independent endorsement.
Which missing field would make you reject a SaaS agent session as operationally unverifiable?
Top comments (0)