A cure is not a cure until SigNoz proves it

In my Tool Sepsis test, an agent can keep serving work while one of its tools fails half the time. The health endpoint stays green. Behind it, retries grow, latency stretches, and tokens pay for calls that never had a chance.

I built Agency Immune around that gap. It watches a LangGraph workload in SigNoz, diagnoses one of six named failure modes, checks a small policy file, applies an allowed response, then runs the detection query again. I only call the incident healed when the measured signal recovers.

That last query became the part of the system I care about most.

The loop I ended up with

The first version of the idea fit in a line:

DETECT -> DIAGNOSE -> DECIDE -> HEAL -> VERIFY -> REPORT

The real system needed sharper boundaries. SigNoz owns the evidence. The Chief, a diagnostic agent connected to the SigNoz MCP server, can read that evidence. A deterministic policy owns permission. The field-agent control API owns the action. SigNoz gets the final vote through a second measurement.

The workload exports OTLP/gRPC traces, metrics, and logs using one resource identity. SigNoz alert webhooks start the supervisor. During diagnosis, the Chief can make at most six read-only MCP tool calls. Approved cures return through an authenticated API; the supervisor never mounts the Docker socket.

I kept the recovery engine headless. The HQ interface can disappear and the alert-to-verification path still runs.

Instrumenting the organism

The test workload has an orchestrator, searcher, critic, and summarizer. Each case produces nested agent, LLM, and tool spans. I added metrics for case duration, retries, token use, estimated cost, tool errors, and tool latency. Python logs use the active trace context, which lets me move from a slow span to the relevant exception in SigNoz.

The initialization is ordinary OpenTelemetry code. The useful decision was giving every signal the same resource:

resource = Resource.create({
    "service.name": "field-agents",
    "service.version": os.environ.get("SERVICE_VERSION", "0.1.0"),
    "deployment.environment": os.environ.get(
        "DEPLOYMENT_ENVIRONMENT", "production"
    ),
})

LLM spans carry gen_ai.request.model, input tokens, and output tokens. Tool spans carry agent.name, tool.name, duration, status, and retry links. I use hq.* attributes only for data that does not have an appropriate standard field.

One small readiness rule saved me from fooling myself: HQ shows 3 SIGNALS LIVE only when the trace, metric, and log queries all return data. A reachable SigNoz endpoint with an empty result is still unready.

Reproducing Tool Sepsis

make chaos-sepsis recreates the field-agent workload with FAIL_RATE=0.5. This is fault injection in the running service. Browser fixtures play no part in that path.

The hq-tool-sepsis alert uses two trace queries grouped by tool.name:

A  = count(tool.exec where has_error = true)
B  = count(tool.exec)
F1 = (A / B) * 100

The rule fires when F1 exceeds 30 percent over five minutes. Its webhook carries the tool_sepsis disease label into the supervisor.

The Chief then queries SigNoz for bounded evidence and returns strict JSON. It may recommend circuit_break_tool, prompt_rollback, or page_human. The recommendation itself has no authority. This policy does:

mode: dry-run
maxHealsPerHour: 3
confidenceThreshold: 0.6

diseases:
  tool_sepsis:
    actions: [circuit_break_tool, prompt_rollback, page_human]

actions:
  circuit_break_tool: { revertAfterMs: 600000 }

Dry-run is the default. Live actions need a disease/action match, enough confidence, room in the hourly budget, and valid credentials. The circuit reverts after ten minutes. Any rejected decision pages a person.

What the system can recognize

I resisted adding a general “fix whatever is wrong” tool. Agency Immune has six known diseases with named queries and short response lists.

The six alerts are provisioned as typed SigNoz v2 rules with Terraform. Four dashboards cover the agent fleet, token economics, the recovery loop, and signal integrity. HTTP Query Builder v5 payloads live in one TypeScript package with snapshot tests.

That shared query package prevents an awkward class of bugs. Persisted dashboard panel JSON and the /api/v5/query_range request body look related, but they are different contracts. The dashboards keep their own schema; runtime code imports tested builders.

The bug hiding in absence detection

Five diseases improve when a measured value drops. Telemetry Necrosis behaves in the opposite direction. It fires when no field-agents spans arrive for three minutes, so recovery means the span count rises above zero.

I made the inversion visible in code:

if (disease === "telemetry_necrosis") {
  outcome = after >= threshold ? "healed" : "failed";
} else if (after < threshold) {
  outcome = "healed";
} else if (after < before * 0.7) {
  outcome = "improved";
} else {
  outcome = "failed";
}

A generic “lower is healthier” helper would have marked a dead telemetry stream as recovered. This four-line branch is more valuable than making the abstraction look tidy.

How VERIFY works

An accepted control request proves that the control API answered. It says nothing about the agent workload.

Agency Immune stores a baseline before the cure. After the action, it waits 90 seconds and rebuilds the query for that disease. The second SigNoz window receives one of the verdicts shown below.

healed means the value crossed the healthy threshold. improved means it dropped by at least 30 percent and remains outside the threshold. Anything else fails. Telemetry Necrosis uses the inverted comparison described above.

The supervisor allows one more verification attempt. A second failure escalates. Every action also emits an immune.heal span, so the cure, its incident, and the recovery window can share one traceable story.

Running it on a clean checkout

Foundry installs SigNoz and its MCP server from the committed casting.yaml and casting.yaml.lock files:

foundryctl cast -f casting.yaml
cp .env.example .env
docker compose -f docker-compose.hq.yaml up -d --build

export TF_VAR_signoz_endpoint=http://localhost:8080
export TF_VAR_signoz_api_key="$SIGNOZ_API_KEY"
terraform -chdir=infra/signoz init
terraform -chdir=infra/signoz apply

I keep control, webhook, operator, and WebSocket credentials separate. The browser receives normalized telemetry and never receives the SigNoz API key.

The current checkout passes 92 unit tests across the query package, supervisor, scale advisor, Python agents, and HQ. The test suite covers Query Builder payloads, policy decisions, authentication, verification math, restart-safe case records, tool failures, and UI transforms. A live SigNoz run remains a separate acceptance gate because unit tests cannot prove that telemetry was ingested.

The six injected failures are known scenarios. Thresholds are tuned for a demonstration. A novel failure goes to a human. I have a time-to-answer study protocol in the repository, but I have left its results blank until I can record both arms on the same SigNoz instance.

The same rule applies to autoscaling. The scale advisor implements the KEDA external-scaler contract and its decision logic is tested. I will describe live scaling only after I have captured KEDA polling, the scale event, cooldown, and the SigNoz source series from a Kubernetes run.

The lesson I am taking forward is narrow: remediation needs an independent success condition. For Agency Immune, that condition is the same SigNoz query that identified the disease. If the number stays sick, the case stays open.