DEV Community

Cover image for Faultline Follow-Up: From Agent Investigation to Controlled Recovery
Harish Kotra (he/him)
Harish Kotra (he/him)

Posted on

Faultline Follow-Up: From Agent Investigation to Controlled Recovery

#signoz #ai #programming #dailybuild2026

Faultline began as a focused answer to an operator question: which agent run went wrong, what did it decide, and where is the SigNoz evidence?

That is useful, but investigation alone leaves the operator with a browser full of evidence and no controlled way to act. The follow-up release adds Recovery Studio: select the failed or risky agent, change the smallest useful inputs, rerun only the affected part of the pipeline, and compare the new trace with the original.

The important constraint is that this is not a second observability dashboard. Self-hosted SigNoz remains the system of record for traces, logs, metrics, and alerts. Faultline is the operator workflow around a specific agent run.

The Recovery Loop

The Recovery Loop

An operator can alter three fields:

  • The task prompt for the selected agent.
  • The OpenAI-compatible model identifier.
  • The token budget for the recovery.

The system prompt and upstream artifacts remain immutable. That makes the recovery comparable: the change is explicit rather than a hidden rewrite of the original workload.

Scope Is the Product Decision

For a pipeline such as:

Planner -> Architecture -> Design system -> Primary screen -> Expo preview
Enter fullscreen mode Exit fullscreen mode

a recovery beginning at Architecture does not rerun Planner. Faultline marks Planner as reused, preserves its artifact as read-only context, and executes Architecture through Preview on a new trace.

The runner creates a child run with explicit lineage:

type RecoveryConfig = {
  taskPrompt: string;
  model: string;
  budget: number;
};

type Run = {
  parentRunId?: string;
  recoveryAgentId?: string;
  recoveryNumber?: number;
  sourceTraceId?: string;
  recoveryConfig?: RecoveryConfig;
  reusedAgentIds?: string[];
};
Enter fullscreen mode Exit fullscreen mode

This produces three useful guarantees:

  1. The source run is never mutated.
  2. The operator does not pay to regenerate known-good upstream work.
  3. The recovery trace has a clear parent relationship for investigation in SigNoz.

SigNoz Correlation Without Duplicating SigNoz

Each recovery emits a separate agent.run trace and adds lineage attributes:

await inSpan("agent.run", {
  "faultline.run_id": run.id,
  "faultline.parent_run_id": run.parentRunId,
  "faultline.recovery_agent_id": run.recoveryAgentId,
  "faultline.recovery_number": run.recoveryNumber,
  "faultline.source_trace_id": run.sourceTraceId,
}, runRecovery);
Enter fullscreen mode Exit fullscreen mode

Faultline continues to log prompts, outputs, retries, artifacts, and failures through OTLP. The console renders three compact evidence capsules and native SigNoz links instead of copying its query builder or generic dashboards.

The operator gets a focused before/after delta:

  • status and failure reason
  • duration, tokens, and retries
  • artifact and model output state
  • direct source and recovery trace links
  • source and recovery Expo preview links when available

Budget and Provider Guardrails

Recovery is intentionally more controlled than the initial demo scenario. Before each recovery-stage request, Faultline checks the remaining recovery budget and caps max_tokens accordingly. If no budget remains, it stops before starting another downstream agent.

Provider failures are classified without another model call:

function classifyIncident(error?: string) {
  const message = (error || "").toLowerCase();
  if (message.includes("429") || message.includes("rate limit")) {
    return "provider_rate_limit";
  }
  if (message.includes("timeout")) return "timeout";
  if (message.includes("budget")) return "budget_exhausted";
  if (message.includes("invalid model")) return "invalid_model_response";
  return "unknown";
}
Enter fullscreen mode Exit fullscreen mode

This matters in practice. During development, OpenRouter returned a 429 after the free-model daily allowance was exhausted. Faultline now records the safe provider reason and retry guidance in the agent dossier and correlated logs rather than collapsing it into an unhelpful LLM 429.

Durable, Dependency-Free History

Faultline is still optimized for a local, single-operator workflow. It now persists the latest 100 runs in .faultline/runs.json, which is ignored by Git. The journal is written atomically using a temporary file followed by rename, then loaded when the runner starts.

await mkdir(dirname(journalPath), { recursive: true });
await writeFile(temporary, snapshot, "utf8");
await rename(temporary, journalPath);
Enter fullscreen mode Exit fullscreen mode

This is deliberately not presented as a multi-user database. It is enough to reopen an investigation after restarting the local stack and to preserve source/recovery relationships during a demo.

API Surface 

POST /api/runs
POST /api/runs/:runId/recoveries
GET  /api/runs
GET  /api/runs/:runId
GET  /api/runs/:runId/agents/:agentId
GET  /api/runs/:runId/comparison
GET  /api/preview/:runId
Enter fullscreen mode Exit fullscreen mode

Example recovery request:

{
  "agentId": "architecture",
  "taskPrompt": "Design a pragmatic Expo architecture. Keep it concise.",
  "model": "cohere/north-mini-code:free",
  "budget": 1000
}
Enter fullscreen mode Exit fullscreen mode

Testing the Recovery Story

The project validates recovery primitives with the built-in Node test runner, so no dependency was added for this release:

npm run build
npm test
Enter fullscreen mode Exit fullscreen mode

The tests cover upstream artifact reuse, incident classification, stable fingerprints, and recovery-budget boundaries. The full local demo is:

npm run signoz:up
npm run dev
Enter fullscreen mode Exit fullscreen mode

Then run an injected incident in the console, select Architecture, use Start recovery, and compare the source and recovery traces.

What Changed in the Product

Faultline is no longer only a flight recorder. It now closes the loop between evidence and action:

Observe -> Explain -> Adjust -> Rerun -> Compare -> Escalate to SigNoz
Enter fullscreen mode Exit fullscreen mode

That is the line between a good demo of telemetry and a useful operator tool.

Code & more: https://www.dailybuild.xyz/project/202-faultline

Top comments (0)