Faultline: Building An Agent Run Investigation Console With Self-Hosted SigNoz

Agent systems fail differently from traditional applications. A request can be syntactically valid, one agent can retry successfully, and the visible product can still be too expensive or subtly wrong. The important question is rarely “what is the service error rate?” It is “what happened in this run, at this stage, and what should I inspect next?”

Faultline is a small operator console built around that question. It runs a real OpenAI-compatible mobile-app generation workflow, renders the resulting product as an Expo app, and uses self-hosted SigNoz as the canonical telemetry backend.

The Product Boundary

Faultline is not a generic dashboard. SigNoz already has a stronger dashboard, trace explorer, log explorer, and alerting surface. Recreating those inside a demo would add noise and weaken the integration.

Instead, Faultline owns the agent-specific investigation flow:

1. Find a run worth inspecting.
2. Follow its causal path.
3. Select the problematic agent.
4. Read the actual prompt, output, retry context, and artifact.
5. Escalate into the exact SigNoz trace, logs, or alert when raw evidence is needed.

The Runner: A Deliberately Small Workflow

The runner uses a fixed workflow: Planner, Architecture, Design system, and Primary screen. A run is inserted into memory immediately, then executed in the background. This gives the console a useful live state instead of a spinner that resolves only at the end.

const run: Run = {
  id,
  scenario,
  status: 'running',
  agents: pipeline.map(([id, name, artifact]) => ({
    id,
    name,
    artifact,
    status: 'queued',
    events: [],
  })),
};

runs.unshift(run);
res.status(201).json(run);
void executeRun(run);

When an agent begins, Faultline records a typed execution event. The UI can therefore distinguish “waiting for the previous stage,” “workspace started,” “retrying,” “model response received,” and “artifact written.”

Full Agent Context Without a Second Telemetry Product

The most useful debugging context is not a token chart. It is the exact task prompt, system instruction, response, and decisions that led to the current artifact.

Faultline keeps this operational dossier with the run and emits equivalent structured OTLP logs to SigNoz. The console uses the dossier for immediate live rendering; SigNoz remains authoritative for durable telemetry investigation.

logEvent(taskPrompt, {
  'faultline.event': 'model.request',
  'faultline.run_id': run.id,
  'faultline.agent_id': agent,
  'faultline.prompt_type': 'user',
});

logEvent(llm.value.decision, {
  'faultline.event': 'model.response',
  'faultline.run_id': run.id,
  'faultline.agent_id': agent,
  'faultline.output_type': 'decision',
});

The selected-agent endpoint returns a normalized shape to the console. It deliberately exposes no SigNoz query language or generic dashboard controls.

GET /api/runs/:runId/agents/:agentId

{
  agent: {
    systemPrompt,
    taskPrompt,
    output,
    artifact,
    events,
    traceId,
    spanId
  },
  evidence: [traceCapsule, logCapsule, riskCapsule]
}

Three Evidence Capsules

The console caps SigNoz integration at three human-readable signals:

Capsule	Question answered	Native escalation
Trace	Is execution correlated and complete?	Exact trace
Logs	Did SigNoz capture prompt, output, retry, and artifact events?	Correlated log search
Risk	Is there a retry, failure, or budget breach?	Alert or dashboard

This constraint is the important product decision. The console remains calm and task-specific, while SigNoz stays visible as a real observability system instead of a logo on a mock screen.

OpenTelemetry Design

The root span is agent.run. Child spans cover each agent, LLM request, retry, artifact write, and preview publication. Shared attributes make the traces and logs navigable across every signal:

await inSpan('agent.architecture', {
  'faultline.run_id': run.id,
  'faultline.agent_id': 'architecture',
  'faultline.scenario': run.scenario,
}, async () => {
  // workspace, retry, model call, artifact write
});

The injected incident has two real effects:

1. Architecture receives an explicit timeout event and retries before making a real model request.
2. The run’s token budget is intentionally lower than observed model usage, producing a budget-breach metric and risk state.

That makes the demo causal rather than theatrical: the operator sees the same retry and budget condition in Faultline and in SigNoz.

From Agent Output To A Working Mobile Preview

An artifact list is not an app preview. Faultline now generates a constrained MobilePlan from the real run. The plan contains product copy, a balance or headline metric, categories, activity, and an insight. Expo renders that plan into an interactive mobile template with Home, Activity, Insights, and Profile views.

type MobilePlan = {
  appName: string;
  balance: string;
  categories: Array<{ name: string; amount: string; color: string }>;
  activity: Array<{ title: string; meta: string; amount: string }>;
  insightTitle: string;
};

This is intentionally constrained generation. It reliably demonstrates a real mobile experience without allowing arbitrary, unsafe model-produced code to execute in the local preview server.

Why Expo Needed Its Own Runtime Path

Faultline sits inside a broader local workspace. Expo SDK 52 expects React 18 and Node 20. Resolving React 19 from the enclosing workspace created incompatible React elements and a blank preview.

The fix has three parts:

• Pin React 18.3.1, React DOM 18.3.1, and React Native 0.76.9 across Faultline.
• Keep a preview-local Metro configuration.
• Start Expo through a script that finds the installed Node 20 nvm runtime, even when the shell defaults to a newer Node release.

const runtime = join(versionsDirectory, node20, 'bin', 'node');
const child = spawn(runtime, [expoCli, 'start', '--web'], { stdio: 'inherit' });

Where To Take It Next

The next logical implementation steps are durable run metadata, redaction rules for production prompts, a configurable pipeline graph, native SigNoz dashboard provisioning, and browser-level regression tests. The core lesson is already useful: agent observability is strongest when the operator can start with an intelligible causal story and only then descend into raw telemetry.

Code & more: https://www.dailybuild.xyz/project/201-faultline