Building the SRE Console for AI Agents: Real-Time Observability Down to the Row

Most AI monitoring tools show you latency graphs and token counts. But when your agent calls a database and returns garbage, those metrics won't save you. Here's how to build an operator console—rooted in SRE discipline—that surfaces actual database rows in real time for true AI observability.

The Operator Console Gap in AI Agent Monitoring

Every SRE team worth their salt has an operator console. It's the screen that shows you exactly what your system is doing, not just how fast it's doing it. When a user reports a problem at 2 AM, you don't stare at a p99 latency graph—you watch the actual requests flowing through your system, inspect the payloads, and trace the failure to its source.

AI agent monitoring, despite billions in investment, barely reaches this bar. Most agent observability platforms offer you high-level metrics: total invocations, average response time, token throughput. These are vanity metrics. They tell you that your agent ran. They don't tell you whether it ran correctly.

Consider a concrete scenario. Your AI agent receives a user query like "Show me all customers who signed up in the last 30 days and haven't made a purchase." The agent decomposes this into steps: generate a SQL query, execute it against PostgreSQL, interpret the results, and format a response. Your dashboard shows green. 400ms response time. 1,200 tokens generated. But the actual SQL query the agent wrote was SELECT * FROM users WHERE created_at > NOW() - INTERVAL '30 days'—completely missing the purchase filter. The agent confidently returned 847 customers when it should have returned 23.

No latency spike will tell you this. No token count will flag it. You need to see the actual data flowing through each stage of the agent's execution. You need row-level visibility.

SRE Fundamentals That Apply Directly to Agent Orchestration

SRE practice has codified three principles that translate perfectly to AI observability, yet almost no agent monitoring platform implements them:

1. Log Aggregation with Structured Context. In traditional SRE, every log line carries a trace ID, a span ID, and structured metadata. In AI agent monitoring, this means every database query the agent generates should be captured alongside the original user intent, the agent's reasoning, and the returned rows. Not just "query executed successfully"—the actual query text and result set.

2. Distributed Tracing Across Heterogeneous Systems. Your AI agent isn't a monolith. It calls LLMs, databases, APIs, vector stores, and tools. An SRE-style trace follows the request across all of these. When an agent orchestration involves a chain of five LLM calls and three database queries, you need to see each step as a span in a single trace, with the ability to drill into any span and see what data passed through it.

3. Error Budgets Applied to Agent Quality. SREs define error budgets—if your availability drops below 99.9%, you stop shipping features and focus on reliability. Agent orchestration needs an equivalent: if your agent's output accuracy drops below a defined threshold, you pause rollout and investigate. But measuring accuracy requires visibility into what the agent actually did, not just that it finished.

{
  "trace_id": "agnt_7f3a2b1c",
  "spans": [
    {
      "span_id": "parse_step",
      "type": "llm_inference",
      "model": "claude-sonnet-4-5",
      "input_tokens": 847,
      "output_tokens": 231,
      "reasoning": "User wants customers from last 30 days with zero purchases",
      "generated_tool_call": {
        "tool": "database_query",
        "query": "SELECT u.id, u.email, u.name, u.created_at FROM users u LEFT JOIN orders o ON u.id = o.user_id WHERE u.created_at > NOW() - INTERVAL '30 days' AND o.id IS NULL",
        "expected_columns": ["id", "email", "name", "created_at"]
      }
    },
    {
      "span_id": "db_exec",
      "type": "database_query",
      "rows_returned": 23,
      "execution_time_ms": 34,
      "sample_rows": [
        {"id": 10412, "email": "[e-mail]", "name": "Sarah Kim", "created_at": "2025-06-02"}
      ]
    }
  ]
}

This is what structured tracing looks like for agents. Every span captures not just timing but intent, reasoning, and actual data. When you build your operator console on top of this trace format, debugging AI becomes a matter of clicking through spans, not guessing from aggregated metrics.

Designing the Real-Time Dashboard: What an AI Operator Console Actually Needs

A real-time dashboard for AI agent monitoring should be modeled after the operator consoles used in NOCs and SRE war rooms. Here's the layout that actually works when you're debugging agent behavior in production:

The Trace Stream. The left panel shows a live feed of agent invocations, color-coded by status: green for successful completions, yellow for degraded quality (the agent completed but with low confidence or unverified output), red for failures. Each entry expands to show the full trace. This is your primary interface for agent monitoring—watching real requests flow through your system in real time.

The Data Inspector. When you click on a database span within a trace, the right panel shows the exact query that was generated, the parameters it received, and a paginated view of the result rows. This is the critical piece. You're not guessing whether the agent's SQL was correct—you're reading it. You're not assuming the data makes sense—you're seeing the actual rows. In a real-time dashboard, these rows update as queries execute.

The Quality Metrics Bar. Across the top, a thin bar shows your current agent quality metrics: output accuracy (measured by comparing agent responses against verified ground truth), tool call success rate, average response relevance score, and your current error budget remaining. These metrics update continuously. When accuracy dips below your defined threshold, the bar turns red and triggers an alert.

The Anomaly Detector. Running in the background, a statistical model watches for unusual patterns: sudden increases in NULL values returned by queries, unexpected row counts, queries that return schema mismatches, or response patterns that deviate from historical norms. When the agent starts consistently generating queries that return zero results on a dataset that should have thousands of rows, you want to know immediately—not after customers complain.

-- The anomaly detector runs this kind of query against your trace data
SELECT
  date_trunc('minute', span_timestamp) AS minute,
  COUNT(*) AS total_queries,
  SUM(CASE WHEN rows_returned = 0 THEN 1 ELSE 0 END) AS zero_row_queries,
  ROUND(
    SUM(CASE WHEN rows_returned = 0 THEN 1 ELSE 0 END)::decimal / COUNT(*) * 100,
    2
  ) AS zero_row_percentage
FROM agent_tool_spans
WHERE tool_type = 'database_query'
  AND span_timestamp > NOW() - INTERVAL '1 hour'
GROUP BY 1
HAVING SUM(CASE WHEN rows_returned = 0 THEN 1 ELSE 0 END)::decimal / COUNT(*) > 0.15
ORDER BY 1;

This query surfaces any minute where more than 15% of database calls return zero rows—a classic signal that your agent's query generation has drifted from the actual schema or data state. The real-time dashboard feeds these results directly into the anomaly panel.

Implementing Row-Level Observability for Debugging AI in Production

Row-level observability means capturing and surfacing the actual data that flows through every tool call your agent makes. This is not optional—it's the difference between "the agent seems fine" and knowing exactly what the agent did. Here's how to implement it.

Step 1: Instrument Your Tool Layer, Not Just Your Agent Layer. Most teams wrap their LLM calls with tracing but leave their tools uninstrumented. The database tool, the API tool, the search tool—these need their own spans. Wrap every tool call in a tracing context that captures inputs and outputs:

import { trace } from '@tormentnexus/sdk';

async function executeDatabaseQuery(
  query: string,
  params: any[],
  context: TraceContext
) {
  const span = trace.startSpan({
    name: 'database_query',
    parent: context.currentSpan,
    attributes: {
      'db.system': 'postgresql',
      'db.query': query,           // Capture the FULL generated query
      'db.params': params,
      'agent.step': context.stepIndex,
      'agent.intent': context.originalUserIntent  // Link back to user request
    }
  });

  const startTime = Date.now();

  try {
    const result = await pool.query(query, params);

    span.setAttributes({
      'db.rows_returned': result.rowCount,
      'db.execution_time_ms': Date.now() - startTime,
      'db.result_sample': JSON.stringify(result.rows.slice(0, 5)),  // First 5 rows
      'db.column_names': result.fields.map(f => f.name).join(','),
      'db.has_nulls': result.rows.some(row =>
        Object.values(row).some(v => v === null)
      )
    });

    // Flag potential issues
    if (result.rowCount === 0) {
      span.setAttribute('agent.quality_flag', 'zero_results');
      span.addEvent('anomaly_detected', {
        message: 'Query returned zero rows - potential agent error'
      });
    }

    span.setStatus({ code: SpanStatus.OK });
    return result;
  } catch (error) {
    span.setStatus({ code: SpanStatus.ERROR, message: error.message });
    throw error;
  } finally {
    span.end();
  }
}

Step 2: Build a Query Replay Capability. When you're debugging an AI agent's behavior, you often want to re-run the exact query it generated against your current data to see if the results have changed. Your operator console should let you click a "Replay" button on any database span that executes the query again and shows you the current result set alongside the original. This is invaluable for diagnosing time-sensitive data issues.

Step 3: Correlate Agent Decisions with Data State. The most powerful debugging AI pattern is understanding not just what data the agent saw, but how that data influenced its reasoning. Tag each span with the agent's interpretation of the data: "Found 847 customers who match the criteria" should be correlated with the actual 847-row result set. When you're in the real-time dashboard, clicking on the agent's reasoning span should show you both the text it generated and the data it was responding to, side by side.

The Agent Quality Pipeline: From Observability to Automated Guardrails

Once you have row-level observability, you can build an automated quality pipeline that catches agent errors before they reach users. This is where AI observability transitions from passive monitoring to active protection.

Schema Validation Gate. Before any database query executes, validate it against your schema registry. If the agent generates a query referencing a column that doesn't exist, or uses a table alias that's been deprecated, reject it and request regeneration. In practice, this catches 8-12% of agent-generated queries in systems with evolving schemas.

Result Contract Verification. After a query executes, verify that the result set conforms to expected patterns. If the agent's reasoning says "customers from the last 30 days" but the query returns rows with timestamps from 2023, flag it. If the result count is an order of magnitude different from what the agent's reasoning predicts, flag it. This catches the subtle correctness issues that latency and token metrics completely miss.

Ground Truth Sampling.</strong

---

Originally published at tormentnexus.site