From a Live Voice Prompt to a Safe Canvas Draft: Observing Sidechalk End to End with SigNoZ

Project: Sidechalk on GitHub

Hackathon: WeMakeDevs Agents of SigNoZ

Track: Track 01 — AI & Agent Observability

Video: Watch the Sidechalk x SigNoZ demo

In my final live demo, I asked Sidechalk to inspect an inclined-plane diagram, reason about its missing force, and add one yellow friction arrow as a draft. The application saw the authorized canvas, kept the first two answers conversational, and created a visible proposal only after the third instruction. The learner could inspect, accept, or reject it; an accepted change could still be undone.

The SigNoZ command center captured the same run. It showed token series for both gpt-realtime-2.1 and gpt-5.6-terra, structured-provider duration, complete agent-turn duration, estimated model cost, one fallback event, and background-worker activity. A second provisioned dashboard covers proposal validation, decisions, safety blocks, and authoritative realtime delivery.

Getting there exposed a real blind spot. In an earlier run, the learner-facing workflow succeeded while most AI panels stayed empty and only worker telemetry appeared. The problem was not SigNoZ or metric-export delay: I had instrumented the typed assistant route, but the live WebRTC sideband followed a different lifecycle.

Fixing that gap forced me to answer a harder question than “Did the model respond?”: Can I observe the complete agent path that the learner actually used?

What Sidechalk is trying to make safe

Sidechalk is a collaborative living notebook. A learner can draw on an Excalidraw canvas, work in a document, import media, and speak or type to an assistant. The assistant can inspect only the authorized Sidechalk page and propose arrows, handwriting, circles, movement, comments, citations, replacement, or recoverable deletion.

Its central invariant is draft first. Model output cannot directly mutate the canonical Yjs document. AI marks stay visually distinct until the learner explicitly accepts them, and accepted work has a deterministic inverse for undo.

For the demo I use an incomplete free-body diagram: a block on an incline with mg, N, and a velocity arrow down the slope, but no friction force. The conversation is deliberately split into three turns:

1. “What can you see on the board? Do not change anything yet.”
2. “The velocity arrow shows the block moving down the slope. In which direction should friction act? Do not draw yet.”
3. “Add one yellow arrow up the incline and write friction beside it. Change nothing else, and keep it as a draft.”

The first two turns test fresh visual context, conversation memory, and physics reasoning without authorizing a surface operation. Only the third turn authorizes a draft.

I arrived at this scenario after another real limitation: an imported PNG is one flattened canvas element. Asking the agent to delete one arrow baked into that image is not the same as deleting an Excalidraw vector element. Instead of hiding that mismatch, I changed the task to a useful additive correction that the surface contract can perform honestly.

Before the correction: the diagram shows the block moving down the slope, but the friction force is missing.

After the voice instruction: Sidechalk draws friction up the incline as a private proposal; the learner still controls accept or reject.

One editing turn crosses more than one model system

The system is a TypeScript monorepo with Next.js and Excalidraw on the web, Fastify for authoritative APIs, Supabase for authentication and durable state, Yjs/Hocuspocus for collaboration, Redis/BullMQ for distributed work, and a constrained Python STEM verifier. OpenAI handles live voice and multimodal proposals; provider adapters also exist for Google and Anthropic.

Next.js / Excalidraw
        |
        v
Fastify API ----> authorized surface context
        |-------> OpenAI Realtime voice response
        |-------> structured proposal model
        |-------> verifier / citation checks
        |-------> proposal validation and persistence
        |-------> explicit accept, reject, or undo
        v
Yjs + Hocuspocus ----> collaborators

Instrumented services --OTLP/HTTP--> self-hosted SigNoZ
SigNoZ MCP --read only-----> Sidechalk Incident Copilot

The Node API, realtime, and worker services export traces, metrics, and selected logs. The Python verifier exports traces, while the Next.js service uses @vercel/otel. Automatic instrumentation captured HTTP, fetch, and supported library activity, but it could not name the product boundaries I cared about. I added manual spans at each trust boundary:

Boundary	Spans	What it answers
Learner turn	sidechalk.ai.turn	Did the complete typed or voice/proposal turn finish?
Authorized context	sidechalk.ai.context.build	Did the model receive the current permitted surface?
Provider	sidechalk.gen_ai.invoke	Which model call was slow, cancelled, or failed?
Verification	sidechalk.verifier.check, sidechalk.citation.verify	Did deterministic or source checks run?
Draft safety	sidechalk.proposal.validate, sidechalk.proposal.persist	Was the model output valid and stored only as a draft?
Human control	sidechalk.proposal.decide, sidechalk.proposal.undo	What happened after the learner explicitly decided?
Collaboration	sidechalk.realtime.authoritative_update	Was the accepted binary update published?
Background work	sidechalk.worker.job	Did asynchronous processing continue outside the request?

That manual instrumentation follows SigNoZ's guidance for tracing business operations that framework instrumentation cannot explain. It also gives each failure a useful boundary: provider, validator, persistence, human decision, or realtime publication.

The blind spot: typed turns and live voice were different systems

My typed /turns route already wrapped the complete workflow and recorded custom metrics. This shortened excerpt keeps the real span and metric names while omitting unrelated validation and streaming code:

return withSpan("sidechalk.ai.turn", {
  "sidechalk.channel": "typed",
  "sidechalk.ai.mode": mode,
}, async () => {
  const canonical = await withSpan(
    "sidechalk.ai.context.build",
    { "sidechalk.surface.type": page.type },
    () => productionRepository.prepareTurn(request, conversationId, userId),
  );

  const response = await withSpan(
    "sidechalk.gen_ai.invoke",
    { "gen_ai.operation.name": "chat" },
    () => createProviderResponse(canonical, { adapters }),
  );

  await withSpan("sidechalk.proposal.validate", {}, validateProposal);
  await withSpan("sidechalk.proposal.persist", {}, () =>
    productionRepository.persistAssistant(response, userId, reservationId),
  );
});

Live voice does not remain inside one HTTP request. A browser sends audio through WebRTC, a server-side sideband receives provider events, the browser captures the current authorized page, and tool calls may pause while a draft is rendered and acknowledged. The original implementation persisted those events correctly, but never called recordAiTurn() and never created the equivalent root/provider spans.

The pre-fix dashboard screenshot made the gap obvious:

Before the fix: successful live-voice activity was absent from the AI panels because the Realtime sideband had not yet been instrumented.

Waiting longer could never repair an uninstrumented path.

Fixing an asynchronous Realtime trace

The fix was not to add a timer around the endpoint that triggers response.create; that request ends before the asynchronous provider lifecycle does. I had to keep a turn span alive across sideband events and create a child provider span for each Realtime response. The following excerpt is shortened, but the lifecycle and attributes are taken from the running implementation:

if (!session.activeTurn) {
  session.activeTurn = {
    startedAtMs: performance.now(),
    span: realtimeTracer.startSpan("sidechalk.ai.turn", {
      attributes: safeAttributes({
        "sidechalk.channel": "voice",
        "sidechalk.ai.mode": session.mode,
        "sidechalk.surface.type": context.board.page.type,
        "sidechalk.visual.available": context.visualAvailable,
      }),
    }),
    inputTokens: 0,
    outputTokens: 0,
    responseCount: 0,
  };
}

const parent = trace.setSpan(otelContext.active(), session.activeTurn.span);
session.activeResponse = {
  startedAtMs: performance.now(),
  span: realtimeTracer.startSpan("sidechalk.gen_ai.invoke", {
    attributes: safeAttributes({
      "gen_ai.operation.name": "realtime",
      "gen_ai.provider.name": "openai",
      "gen_ai.request.model":this.options.environment.OPENAI_REALTIME_MODEL,
    }),
  }, parent),
};

When response.done arrives, Sidechalk extracts only numeric usage metadata, ends the provider span, and either continues through a tool call or closes the learner turn:

const calls = completedFunctionCalls(event);
const usage = realtimeResponseUsage(event);

activeTurn.inputTokens += usage.inputTokens;
activeTurn.outputTokens += usage.outputTokens;
activeResponse.span.end();

if (calls.length === 0) {
  recordAiTurn({
    durationMs: performance.now() - activeTurn.startedAtMs,
    result: "success",
    provider: "openai",
    model: this.options.environment.OPENAI_REALTIME_MODEL,
    inputTokens: activeTurn.inputTokens,
    outputTokens: activeTurn.outputTokens,
  });
  activeTurn.span.end();
}

The surface-proposal tool is nested under that voice turn. Its structured model call now records provider/model attributes, token usage, estimated cost when the model exists in the checked-in price table, proposal validation, and persistence. Cancelled speech, provider errors, and socket failure close spans explicitly instead of leaking unfinished traces.

This was the lesson I could not get from an installation guide: instrument the lifecycle, not the endpoint that starts it.

The next fresh voice proposal produced the signals that were missing before:

This is not an empty dashboard: the selected window contains fallback, cost, worker, provider-duration, token, and complete agent-turn signals. The separate metric-histogram P95 panel still says No Data in this capture, while the trace-derived provider and agent-turn P95 panels are populated. I left that visible rather than presenting a cleaner but misleading screenshot.

The trace connects the live voice turn to proposal generation and exposes the failed provider request instead of hiding it behind the successful overall workflow.

Metrics and dashboards built around trust

The Node SDK exports traces, metrics, and selected correlated logs over OTLP/HTTP. Metrics use a 15-second periodic reader. I record:

• AI turn volume and end-to-end latency
• provider invocation duration and fallback
• input/output tokens and estimated model cost
• proposal operation, validation, decision, and undo outcomes
• verifier and citation outcomes
• queue wait and worker activity
• canonical-safety invariant violations

Two idempotently provisioned dashboards contain eight panels each:

• Sidechalk AI Command Center covers volume, P95 latency, provider duration, fallback, tokens, estimated cost, worker activity, and complete agent-turn duration.
• Sidechalk Trust and Proposal Safety covers proposal operations/outcomes, verifier and citation activity, validation, invariant blocks, authoritative realtime delivery, and errors.

Four enabled rules watch provider reliability, learner-facing latency, model budget, and canonical safety. A tested local webhook accepts alert envelopes. I do not claim a production firing-and-recovery canary; this article describes my verified local stack.

Proposal creation, human decisions, validation, and shared delivery are monitored as separate trust stages.

“No Data” has three different meanings

This project produced three superficially similar empty states:

1. Expected absence: no fallback or invariant violation occurred. Empty is healthy evidence.
2. Missing instrumentation: in the pre-fix run, worker data arrived while the voice AI path stayed empty. The application path was invisible until I instrumented its sideband lifecycle.
3. Broken query infrastructure: a P95 query failed because my generated ClickHouse configuration did not load histogramQuantile correctly.

The third case was unusually specific. The generated executable-function configuration referenced the wrong filename pattern, and the UDF's quantile parameter used the wrong type. I corrected the local generated files, restarted only ClickHouse, and verified the function directly:

docker exec signoz-telemetrystore-clickhouse-0-0 \
  clickhouse-client --query \
  "select name from system.functions where name = 'histogramQuantile'"

The distinction matters. I should not manufacture a fallback event to make a panel attractive, but I also should not call an uninstrumented voice route healthy.

Reproducing my local path

I used Windows 11, Docker Desktop, WSL2, Node 24, and Foundry v0.2.14. Foundry generated the self-hosted SigNoZ stack from the checked-in casting.yaml:

foundryctl gauge -f casting.yaml
foundryctl cast -f casting.yaml

The standard container services use SigNoZ UI port 8080 and MCP port 8000. Those ports were already occupied on my machine, so my current host mappings are 18080 for the UI and 18000 for MCP. OTLP/HTTP remains 4318, and the Sidechalk verifier uses 8002. Check your own port owners instead of copying my remap blindly.

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
OTEL_ENVIRONMENT=development
OTEL_LOGS_ENABLED=true
OTEL_CAPTURE_AI_CONTENT=false
OTEL_HASH_SALT=<private-random-value-at-least-16-characters>

SIGNOZ_UI_URL=http://localhost:18080
SIGNOZ_URL=http://localhost:18080
SIGNOZ_MCP_URL=http://127.0.0.1:18000/mcp
SIGNOZ_API_KEY=<service-account-key>
SIGNOZ_ALERT_CHANNEL=Local Webhook Receiver
OBSERVABILITY_ADMIN_USER_IDS=<my-Supabase-user-UUID>

From the repository root:

pnpm ports:preflight
docker compose up -d redis verifier
pnpm dev

pnpm signoz:provision --dry-run
pnpm signoz:provision
pnpm signoz:verify

The verifier checks the expected dashboard names, Query Builder version, panel IDs and counts, enabled alert routing, webhook-channel identity, and three representative telemetry queries. It deliberately failed when sidechalk.ai.turn.duration.bucket had last been seen four hours earlier. That failure was useful: provisioning existed, but fresh traffic had not proved the path.

After a new agent turn, this filter locates the workflow:

service.name = 'sidechalk-api' AND name = 'sidechalk.ai.turn'

Incident investigation without write access

Sidechalk also includes an Incident Copilot backed by the official SigNoZ MCP server. The MCP server can expose metrics, traces, logs, alerts, dashboards, and services; Sidechalk deliberately allowlists only read operations.

Each investigation is limited to four calls, 15 seconds per call, 45 seconds overall, and a maximum six-hour time window. Evidence cards retain the SigNoZ tool, time range, service, and deep link. Create, update, delete, and unrestricted execution tools are excluded.

A development-only Fault Lab can inject bounded provider delay/failure, verifier contradiction, or worker delay. Faults are labeled sidechalk.demo.fault.*, stay in memory, expire automatically, and are forbidden in production. Synthetic evidence must announce that it is synthetic.

The copilot queries read-only SigNoZ evidence and refuses to invent a root cause when no correlated fault or error is found.

Observability without copying the notebook

The most important telemetry decision was what not to attach to the AI-specific spans, metrics, and logs. Sidechalk's manual instrumentation excludes prompts, responses, handwriting, page images, audio, transcripts, authorization data, and raw user/board IDs. Manually recorded error spans keep only an error class and a fixed content-free message. IDs intentionally used for AI-workflow correlation are HMAC-hashed before export.

const forbiddenKey =
  /(?:prompt|response|content|body|text|audio|transcript|document|authorization|cookie|secret|api.?key|access.?token|email|user.?name)/iu;

export function safeAttributes(attributes: SafeAttributes = {}): Attributes {
  const result: Attributes = {};
  for (const [key, value] of Object.entries(attributes)) {
    if (value === undefined || forbiddenKey.test(key)) continue;
    result[key] = typeof value === "string" ? value.slice(0, 240) : value;
  }
  return result;
}

Those custom AI signals contain operation names, duration, provider/model, numeric token totals, proposal category/result, and safe correlations—not the learner's notebook content. I still treat generic auto-instrumentation as a separate privacy surface and avoid placing sensitive values in URLs or captured headers.

What I would tell myself before building this

1. Trace the path users actually take. A beautifully instrumented typed route did nothing for my live-voice demonstration.
2. Instrument product invariants. HTTP success cannot prove that an AI draft remained non-canonical.
3. Async agents outlive request spans. WebRTC and sideband tools need explicit context propagation and lifecycle closure.
4. Treat No Data as a diagnosis, not a cosmetic problem. Expected absence, missing telemetry, and query failure are different states.
5. Verify provisioning with live queries. A successful create/update response does not prove that panels can read real signals.
6. Keep investigation agents least-privileged. Evidence access does not require configuration write access.
7. Filter before export. Redaction after learner content reaches the backend is already too late.

Closing the loop

Sidechalk's promise is not merely that an AI can answer a physics question. It is that the agent can see authorized work, reason across a conversation, propose a spatial correction, wait for explicit approval, publish exactly one canonical update, and undo it safely.

SigNoZ made that promise testable. The temporary empty state showed where my first implementation was blind; the populated command center now shows that the real live-voice path is observable.

The AI can work on the page, but it cannot silently control the page—and now the path enforcing that rule is observable too.

Links

• Sidechalk source code
• Video demonstration
• Agents of SigNoZ hackathon
• Hackathon blog-writing guide
• SigNoZ Node.js instrumentation guide
• SigNoZ manual Node.js instrumentation
• OpenTelemetry JavaScript documentation
• SigNoZ MCP server