Ashiha Mahesh Kumar

Posted on Jul 26

Nights Watch: Guarding AI Agents Beyond the Wall

#agents #ai #signoz #automation

Nights Watch: Using SigNoz as a Runtime Control Plane for AI Agent Recovery

A technical write-up of a hackathon build for WeMakeDevs' "Agents of SigNoz." This post covers the architecture, the OpenTelemetry instrumentation, and two annotated trace walkthroughs — one clean run and one with deliberately injected drift — from github.com/ashihams/Nights-Watch.

1. Problem statement

Most AI agent safety discussion centers on adversarial failure: prompt injection, jailbreaks, tool poisoning. There is a second, quieter failure mode that gets far less attention: an agent that is not attacked at all, and simply drifts. Given a bounded task — "find and book a flight from SFO to LAX under $400" — an agent can, without any external interference, select a $1,200 option, add an unrequested hotel booking, or otherwise take an action that no longer matches what was approved. Nothing throws an exception. No policy is technically "broken" in the sense of hitting an error path. The system just quietly does more, or something different, than it was asked to do, and by the time this is noticed, an irreversible action (a non-refundable booking, in this case) may already have executed.

Nights Watch is a runtime resilience system built to catch this specific failure class. It does five things:

Tracks an agent's actual execution against its originally approved plan, step by step.
Scores each step against policy rules (0–100; higher means more severe deviation).
Produces a plain-language explanation of why a violation occurred, at the moment it's detected.
Pauses execution before an out-of-scope or irreversible action fires.
Rolls back to the last known-good checkpoint, re-plans, and resumes — either automatically or pending human approval, depending on severity.

The architectural claim this post is built around, and the reason it's relevant to a SigNoz-focused write-up specifically: the Policy Engine queries SigNoz synchronously, on the critical execution path, to help decide what happens next. SigNoz is not a passive sink for logs reviewed after the fact — a live query against it is part of the decision loop. That is a materially different integration pattern than "instrument the app, look at dashboards later," and it's the part of the build I think is worth documenting in detail.

2. System architecture

Nights Watch has four components, deliberately kept in a strict role separation:

Component	Role	Owns state?
n8n	Executor. Runs the actual agent workflow (search → select → confirm → book) as a visual workflow. Reports each step's result to the backend over webhook.	No
Nights Watch Backend (TypeScript, Fastify)	The decision layer. Contains the Policy Engine, Checkpoint Manager, and Recovery Engine.	Yes — execution state lives here
SigNoz	Observability backend. Ingests OpenTelemetry traces/metrics/logs from the backend; exposes a Query API and an MCP server for querying that data.	No — by design
React Dashboard	Live view over WebSocket: policy score timeline, checkpoint markers, explanation feed, recovery metrics, human-approval controls.	No

        React + TypeScript + Tailwind
           (Nights Watch Dashboard)
                     │
            REST  /  WebSocket
                     │
      Nights Watch Backend (TypeScript)
   ┌─────────────────┼──────────────────┐
   │                 │                  │
   ▼                 ▼                  ▼
Policy Engine   Checkpoint Manager   Recovery Engine
   │                 │                  │
   │        (owns execution state,      │
   │         backed by local store)     │
   │                 │                  │
   └─────────────────┼──────────────────┘
                     │
            OpenTelemetry SDK
                     │
                  SigNoz
                     ▲
                     │
          n8n Workflow Engine

n8n's role is intentionally narrow: it executes steps and reports outcomes. It makes no policy decisions and holds no execution state. That separation matters for the same reason it matters in any control-plane/data-plane split — the component that acts should not also be the component that judges, or you lose the ability to independently audit and override.

2.1 The data-source classification rule

The single design decision this whole project hangs on is a strict split between two categories of data, based on latency tolerance:

	Live context (blocks execution)	Observability context (informs, doesn't block)
Examples	current step, current checkpoint, current tool call, current spend	prior policy violations this run, recovery metrics, historical traces, budget trend
Source of truth	the Checkpoint Manager's own local store	SigNoz, via Query API or MCP
Latency tolerance	none — the agent loop is blocked on this	high — SigNoz can be slower without breaking anything

The rule that follows from this table: if a rollback decision ever depends on the result of a SigNoz query, that's a bug. Rollback state is reconstructed from the Checkpoint Manager's own store — never from SigNoz — precisely because a query against an external HTTP service is a latency and availability dependency you don't want in a path that's supposed to fire in milliseconds. SigNoz is where the Policy Engine goes to ask decision-support questions ("has this run already had a medium-severity violation?"), not decision-critical questions ("what state do I roll back to?").

This is also why the project uses two separate access patterns into SigNoz rather than one:

Direct Query API — used by the Policy Engine, on the critical path, for fast, typed, deterministic lookups. No LLM indirection here, because the Policy Engine runs on every single step and can't absorb the added latency or non-determinism an LLM-driven query would introduce.
SigNoz MCP server — used by the Explanation Layer, which only fires occasionally (on a violation). SigNoz's MCP server (signoz.io/docs/ai/signoz-mcp-server) exposes tools for querying metrics, traces, logs, alerts, and dashboards through natural language, with the MCP client (an LLM tool-calling loop, in this case) deciding what to ask for rather than the query being pre-specified. That's the right shape for "explain why this happened" — an LLM deciding what context is relevant — and the wrong shape for "should I roll back right now."

3. Instrumentation: what actually gets sent to SigNoz

The backend uses the OpenTelemetry SDK for Node.js. Every run produces one root span, run.execute, of kind Internal, with resource attributes standard to any Node.js OTel setup — service.name: "nights-watch", telemetry.sdk.name: "opentelemetry", telemetry.sdk.version: "1.30.1", process.runtime.name: "nodejs", process.runtime.version: "24.5.0" — plus process-level detail like process.command, process.owner, and host.name, which is standard OpenTelemetry Resource semantic convention data rather than anything custom.

Underneath the root span sit child spans for each checkpoint boundary: checkpoint.created, policy.evaluation, executor.step, and (on a recovery run) checkpoint.restored. A clean run produces 14 spans; a run that hits one recovery cycle produces 19.

The interesting instrumentation decision is how the content — the plan, the policy score, the recovery outcome — gets attached. Early versions of the instrumentation put this data directly on span attributes, which made the flame graph difficult to read and mixed "facts about the operation as a whole" with "things that happened at a specific moment." OpenTelemetry's own guidance on this draws exactly that distinction: attributes describe the operation as a whole and are best used for data known when the span starts, while an event should be used when the data represents a distinct, timestamped occurrence within the span's lifetime — something that could happen zero or more times (opentelemetry.io/docs/specs/semconv/general/events; opentelemetry.io/docs/concepts/signals/traces). A policy evaluation, a recovery start, and a run completion are all exactly that: discrete, timestamped things that happen during the life of a run.execute span, not properties of the span as a whole. So each is recorded as a span event on the root span, not a top-level attribute:

plan.ready — fired once, carries the full generated plan as a JSON attribute
run.recovery_started — fired when a violation triggers rollback; carries policy.score, severity, and the step that failed
run.resumed — fired after successful rollback; carries checkpoint.id, completedSteps, durationMs, and recovery.action
checkpoint.pre_irreversible — fired immediately before an irreversible step (e.g. book) executes
run.completed — fired at the end; carries policy.lastScore, recovery.count, the list of checkpoints, and steps

One caveat worth being upfront about: OpenTelemetry has an in-progress proposal to deprecate the span-events API in favor of log-based events correlated to the active span context (opentelemetry.io/blog/2026/deprecating-span-events). Existing data and span-event views are expected to keep working, and this is a forward-looking migration rather than something that affects a project built today — but it's a reasonable thing to track if you're instrumenting a similar system now and expect it to still be running in a year or two.

The plan.ready event's JSON payload is the single most load-bearing piece of data in the whole system, because it's what the Policy Engine diffs every actual step against:

{
  "goal": "Find and book a flight from SFO to LAX under $400",
  "maxBudget": 400,
  "currency": "USD",
  "origin": "SFO",
  "destination": "LAX",
  "steps": [
    { "id": "search", "order": 0, "description": "Search flights within budget",
      "expectedTool": "search_flights", "constraints": { "maxPrice": 400 } },
    { "id": "select", "order": 1, "description": "Select an in-budget flight option",
      "expectedTool": "select_flight", "constraints": { "maxPrice": 400 } },
    { "id": "confirm", "order": 2, "description": "Confirm passenger and itinerary details",
      "expectedTool": "confirm_details", "constraints": {} },
    { "id": "book", "order": 3, "description": "Book the confirmed flight",
      "expectedTool": "book_flight", "constraints": { "irreversible": true },
      "expectedTargets": ["flights"] }
  ]
}

Every field here is queryable structured data, not free text — which is what lets the Policy Engine's evaluation be a deterministic rule check (does expectedTool for this step match the tool actually invoked; does the actual value respect the step's constraints) rather than something fuzzier.

4. Trace walkthrough — clean run

The demo scenario is fixed: find and book a flight from SFO to LAX, budget capped at $400. On a clean run, the dashboard's Policy Score Timeline — driven by data streamed out of SigNoz over WebSocket — stays at 0 across all four checkpoints.

The corresponding SigNoz trace (run.execute, 4.2 s, 14 spans, 0 errors) shows the expected repeating pattern in the waterfall: checkpoint.created → executor.step → policy.evaluation, once per plan step.

Two details from the span's Overview tab are worth calling out specifically because they're standard OTel resource attributes doing real diagnostic work, not custom instrumentation: process.command (D:\project\SignozHack\backend\src\index.ts) and process.command_args, which together let you confirm — from inside the trace, without touching the terminal — exactly which build and entry point produced this run. That's the kind of detail that matters when you're debugging "why did staging behave differently from local" and don't want to rely on memory of what was deployed when.

By the closing run.completed event:

policy.lastScore: 0, recovery.count: 0 — the run never needed the Recovery Engine, because the Policy Engine never scored a step above the pause threshold. This is the boring, expected outcome. Section 5 breaks it on purpose.

5. Trace walkthrough — drift injection and recovery

The Control Room dashboard has a second entry point for exactly this: Start Run (Inject Drift), which perturbs the "select" step so it resolves to a $1,200 fare instead of staying under the $400 cap — a direct simulation of the silent-overreach scenario from Section 1.

The moment select_flight is invoked at $1,200, the Policy Engine's rule evaluation fires before the workflow is allowed to proceed to confirm. This is the live-query path from Section 2.1 doing its job: the backend is blocked, mid-execution, on this evaluation.

Note the three severity thresholds visible on the Policy Score Timeline's y-axis in these screenshots: Pause at 40, Rollback at 70, Hard-stop at 90. This run's violation scored 40 — a medium-severity pause that surfaces for human approval rather than triggering an automatic hard rollback, which is the intended behavior for a violation that's clearly wrong but not yet irreversible.

The corresponding SigNoz span event, run.recovery_started, records the identical numbers the dashboard is rendering:

policy.score: "40", severity: "medium", step: "select". This one-to-one correspondence between the dashboard and the raw trace data was a deliberate verification point during the build — the dashboard reads these values live off SigNoz via WebSocket, so there's no separate in-memory state that could silently drift out of sync with what's actually recorded in the trace.

Choosing Reject & Recover triggers the Recovery Engine, which reconstructs state from the Checkpoint Manager's own store (not from SigNoz — see Section 2.1) and re-plans from the last safe checkpoint:

recovery.action: "rollback_replan", completed in 13 milliseconds. That number is the direct payoff of the architectural split: because rollback reads from a local store rather than round-tripping to an external HTTP API, its speed is bounded by local I/O, not network latency or SigNoz query load. Once resumed, the agent re-executes select correctly, then confirm and book. The dashboard's Recovery Health panel updates accordingly:

And the full Policy Score Timeline for the run shows the shape of the whole incident in one chart — a spike to 40 at the second select attempt, then back to 0 for confirm and book:

The final run.completed event confirms the run finished with one recovery cycle and the original goal still met:

recovery.count: 1. The agent hit a real, scored violation and did not need a full restart or an unrecoverable failure state to get back on track.

6. Discussion

Three things stood out as genuinely non-obvious while building this, beyond what the architecture diagram alone communicates:

Deciding what SigNoz is and isn't allowed to gate was the actual hard design problem. It's tempting, especially with an MCP server available, to let a live query decide "should this rollback happen." An early prototype did something close to this, and the added latency plus the non-determinism of an LLM-mediated query made the pause-and-recover loop noticeably slower and occasionally inconsistent in repeated testing. The fix wasn't a performance optimization — it was a re-scoping of which questions SigNoz is allowed to answer on the critical path at all (Section 2.1's table).

Span events, not span attributes, are what kept the trace view legible. Once the plan JSON, policy scores, and recovery outcomes moved off top-level span attributes and onto discrete events on the root span, the flame graph stopped needing to be scrolled through wide attribute dumps to find the one relevant number for a given moment in the run.

A 13 ms and a 2 ms recovery time are not incidental — they're the metric that actually validates the architecture. If rollback took seconds instead of milliseconds, the "resilience" claim in the project's name would be weaker in practice than in the pitch. The number in the trace is the honest test of whether the live/observability split held up under an actual injected failure, not just in the design doc.

7. Summary

Nights Watch treats SigNoz as an active input to a runtime decision loop, not a passive record of what already happened: a fast, typed Query API path for the Policy Engine's every-step decisions, and the MCP server's natural-language querying for the occasional, non-blocking "explain this" case. The two traces above — one clean, one with an induced $1,200 drift and a 13 ms rollback — are the unedited output of that split working as designed.

Architecture and full workflow diagrams for the n8n executor layer are in progress and will follow in an update to this post.

Repository: github.com/ashihams/Nights-Watch
Event: WeMakeDevs — Agents of SigNoz Hackathon
References: OpenTelemetry — Traces · OpenTelemetry — Semantic Conventions for Events · SigNoz MCP Server docs