How to Black-Box Test MonkeyCode Task Reliability

A long-running AI development task is a distributed workflow even when the interface looks like one conversation.

The client can disconnect. A worker can die. A model request can time out after producing a response. A repository write can succeed before its acknowledgment is lost. Cancellation can race with completion. Reliability depends on how the system resolves those events, not on whether the happy-path demo finishes.

This article develops a black-box test plan for MonkeyCode, whose public repository describes AI task management and managed development environments. The source reference is commit 1ac778f. The plan has not been executed against a production deployment, so it defines claims that could be measured rather than reporting results.

Define a black-box adapter

Do not couple the test model to undocumented endpoints. Create an adapter around the supported user or API surface:

submit(logical_request_id, repository_revision, task_spec) -> task_id
observe(task_id, after_cursor) -> ordered_events
answer(task_id, question_id, value, expected_version)
cancel(task_id, expected_version)
artifacts(task_id) -> patch, checks, preview metadata

The adapter may drive an official API, a test-only interface, or browser automation. Its job is to normalize observations without pretending to know the internal queue or database schema.

Every test event should include:

test_run_id, logical_request_id, task_id, event_id, task_version,
event_type, observed_at, actor, repository_revision, artifact_digest

If the public surface does not expose an event ID or version, record that limitation. A test cannot assert ordered recovery from evidence the system never reveals.

Write invariants before workloads

An invariant must remain true across retries and failures.

I1: one logical submission has one accepted identity

Retrying a timed-out submit with the same logical request ID must return the existing task or an explicit duplicate relationship. It must not silently create two workers that can both write a patch.

I2: terminal state does not move backward

Once the authoritative task is succeeded, failed, or canceled, a delayed event must not make it running again.

I3: success has evidence

A task cannot be considered successful until required artifacts exist and their verification result is attached. “The model stopped” is not task success.

I4: cancellation is a protocol

The client must distinguish cancel requested from canceled. If completion wins the race, the final state and artifacts must explain that outcome.

I5: repository effects are attributable

Every patch or repository write must map to one task identity, source revision, and actor policy. A retry must not duplicate a non-idempotent effect.

I6: observation cannot create work

Refreshing, polling, reconnecting, or opening the same task from a second device must not resubmit it.

Control the workload

Use fixtures with deterministic verification even if model output varies.

Fixture	Task	Verifier
Small API	add division-by-zero validation	named unit test passes; bounded file set
Frontend form	add required-field feedback	DOM test plus accessibility assertion
Dependency update	update one pinned library	lockfile change plus full test command
Broken build	repair one known configuration error	build exits 0; expected file changes

Pin the repository revision, toolchain, dependency cache policy, model route, task text, maximum duration, and acceptance command. The model response may differ, but the outcome remains machine-checkable.

Inject failures at state boundaries

The most valuable failures occur after a side effect but before its acknowledgment.

Fault	Injection point	Required observation
Client disconnect	after submit response is lost	reconnect finds one task without resubmission
Duplicate submit	same logical ID in parallel	one accepted task or explicit deduplication result
Worker termination	during clone, edit, test, and artifact upload	bounded recovery or explicit terminal failure
Model timeout	before and after response receipt	retry policy visible; no duplicated repository effect
Delayed event	deliver older running event after terminal state	version does not regress
Cancel race	cancel while final verification completes	one explainable terminal result
Storage interruption	while persisting event or artifact metadata	no success without durable evidence
Credential expiry	before repository read or write	bounded failure; no partial unauthorized retry

Use a disposable deployment or test environment. Killing shared production workers to prove a point is not a reliability test plan.

Measure ambiguity as a first-class failure

Traditional uptime can hide the worst task outcome: neither the user nor the operator knows whether work happened.

Track:

task_completion_rate = verified terminal tasks / accepted tasks
duplicate_effect_rate = duplicated repository effects / logical tasks
orphan_rate = non-terminal tasks with no live owner beyond deadline / accepted tasks
ambiguous_outcome_rate = tasks whose effect and final state disagree / accepted tasks
recovery_time = authoritative recovery event time - injected fault time
cancel_latency = confirmed terminal event time - cancel acceptance time

Report distributions, not only averages. A p95 recovery time and the raw worst cases matter more than a fast median when a developer waits for a lost task.

Separate availability from correctness

A system can remain available while returning stale or contradictory task state. Score each run on four axes:

Axis	Question
Availability	Could the user submit, observe, and act?
Safety	Were duplicate or unauthorized side effects prevented?
Liveness	Did the task reach a bounded terminal or recovery state?
Explainability	Could the user and operator reconstruct the outcome from evidence?

Do not combine the axes into one “reliability score” until the raw failure categories are visible.

What this plan can establish

Executed against a versioned deployment, this plan can provide evidence about observed task semantics under declared faults. It cannot prove all internal failure modes, security isolation, model quality across arbitrary tasks, or behavior under a topology that was not tested.

The official MonkeyCode README documents the product category and deployment options needed to make the plan relevant. It does not establish that each proposed adapter field or invariant is already exposed. Missing observability is itself a test finding, but it should not be rewritten as an implementation claim.

Disclosure: I contribute to the MonkeyCode project. This is a proposed black-box evaluation design based on public documentation, not an independent reliability benchmark.

Teams interested in executing the matrix can join the MonkeyCode Discord to clarify supported test surfaces and deployment topology. The team can also confirm current availability, eligibility, and limits for free model credits used during an evaluation.

The strongest reliability claim is not “our agent usually finishes.” It is a versioned test result showing what happens when acknowledgments disappear, workers die, events arrive late, and cancellation loses a race.