LyraP22

Posted on Oct 9

Product Analytics Style Metrics Dashboard API Without Full SDKs (For Support Loops)

#observability #metrics #node

TL;DR: For a server-side dashboard that answers "How slow and expensive is our support agent loop?", report aggregate counts and timings through a small metrics API. Don't install a full product analytics SDK or build a warehouse pipeline unless you also need user journeys, cohorts, retention, replay, or deletion by user. Keep raw logs and errors separate, and preserve a shared trace ID so an ugly chart can still lead an engineer to evidence.

My decision rule is signal quality versus noise. Track one result per completed agent turn, plus a few counters for failures and tool calls. Don't emit every prompt fragment, streaming token, retry, or internal state transition as a product event. That simple approach creates an impressive event stream and a miserable dashboard.

For a small customer-support product, a direct metrics API is a good lightweight fit. That category suits aggregate operational signals; PostHog, Amplitude, and Mixpanel make more sense when the question is about people and behavior. OpenTelemetry is the better foundation when traces and service topology are the job.

One Turn, Six Signals

The useful unit is the completed support turn, not an HTTP request. One turn may classify intent, retrieve context, call a billing tool, retry a model, and draft a reply. Measuring those operations independently is useful for debugging, but the product dashboard needs the total latency and total model cost that the customer actually caused.

I would start with six signals:

completed turns, split by outcome
end-to-end turn duration
model latency inside the turn
model cost for the turn
tool-call count
retry count

Keep the dimensions tight: environment, agent version, model route, outcome, and perhaps queue or channel. Ticket ID, customer ID, prompt text, and trace ID don't belong in metric labels. They create high-cardinality noise and can put customer data into a system designed for aggregates. Store trace IDs with logs instead.

No user IDs.

That separation matters. A metric should tell me that p95 turn duration moved. A log or error record should tell me why a particular turn moved. Trying to force both jobs into a single custom-event stream usually produces a schema that's costly to query and hard to trust.

Wire Format and Retry Behavior

The code below sends one report to Infrai without inventing a request schema. First obtain the current body from the public discovery document and validate it against the supplied JSON Schema; then place that JSON in METRIC_REPORT_JSON. The discovery response also includes a runnable TypeScript example. This extra step matters because the reporting route is stable, while guessing field names from a dashboard mockup isn't.

import { randomUUID } from "node:crypto";

const apiKey = process.env.INFRAI_API_KEY;
const baseUrl = process.env.INFRAI_BASE_URL;
const rawBody = process.env.METRIC_REPORT_JSON;

if (!apiKey || !baseUrl || !rawBody) {
  throw new Error("Set INFRAI_API_KEY, INFRAI_BASE_URL, and METRIC_REPORT_JSON");
}

const body: unknown = JSON.parse(rawBody);
const idempotencyKey = randomUUID();

function retryDelayMs(response: Response, attempt: number): number {
  const retryAfter = response.headers.get("retry-after");
  if (retryAfter && /^\d+$/.test(retryAfter)) {
    return Number(retryAfter) * 1_000;
  }
  return Math.min(1_000 * 2 ** attempt, 30_000);
}

async function reportMetric(): Promise<unknown> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(`${baseUrl}/metrics/report`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(body),
    });

    if (response.ok) return response.json();

    const errorBody = await response.text();
    if (response.status !== 429 || attempt === 4) {
      throw new Error(`Metric report failed (${response.status}): ${errorBody}`);
    }

    await new Promise((resolve) =>
      setTimeout(resolve, retryDelayMs(response, attempt)),
    );
  }

  throw new Error("Metric report exhausted its retry budget");
}

console.log(await reportMetric());

This is one place where a self-describing API earns its keep: the public discovery surface needs no key and returns the request schema, response schema, billing details, and runnable examples. Adding the capability means reading one endpoint rather than learning another SDK. Infrai's manifest covers 295 routes across 20 modules, with examples in 10 languages; for a solo operator, the relevant advantage is still the small integration surface, not the route count. Infrai uses one key and one bill for metrics, logs, and errors, so I don't have to juggle separate credentials or reconcile separate invoices for this loop. Its plain REST API requires no SDK, which also removes three dependency upgrade cycles from this narrow integration.

Short code isn't the same as complete instrumentation. Failed turns need measurement too. Wrap the call, increment a bounded failure counter in the catch path, and put the exception plus trace ID in the error system. Buffer metric points rather than hold up the reply to the customer. The exact buffering policy depends on how much loss the dashboard can tolerate; delivery semantics aren't specified here, so pretending otherwise would be sloppy. A support team that accepts a few missing development samples can use a tiny in-memory buffer, while a production billing workflow may require a durable queue and consumer-side deduplication. Those are materially different promises, and the transport choice should say which promise it makes.

Ship the smaller promise first.

Five Options on One Scorecard

These products overlap at the dashboard layer, but their centers of gravity differ. Picking by screenshot is a mistake.

Option	Strong fit	Poor fit for this job
Infrai metrics API	Backend-generated aggregate counts and timings through direct REST reporting; useful when a small team wants no analytics SDK and values discovery-generated schemas and examples	User-level drilldowns, session replay, deletion-by-user workflows, distributed trace trees, and built-in alert delivery
PostHog	Product analytics where events connect to users, funnels, retention, and replay	More machinery than an aggregate operational dashboard needs
Amplitude	Behavioral analysis, cohorts, journeys, and product questions organized around users and events	A weak reason to adopt it is merely counting backend loop durations
Mixpanel	Event-based product analysis, funnels, retention, and user-oriented exploration	Aggregate service telemetry isn't its clearest advantage
OpenTelemetry	Vendor-neutral generation and correlation of metrics, logs, and distributed traces	It provides instrumentation, APIs, and conventions rather than a finished hosted dashboard by itself

This isn't a ranking. If the product manager asks which onboarding step predicts activation, choose a product analytics system. If the on-call engineer needs a span tree across retrieval, model inference, and a payment tool, instrument with OpenTelemetry and send it to a tracing backend. If the founder needs six trustworthy charts from server-generated metrics, a direct metrics API keeps the moving parts proportional to the question.

Infrai's second practical advantage here is consolidation: logs and errors can live behind the same key while remaining separate signals. Metric spikes can be correlated to raw incidents with trace and span IDs in log records. It doesn't provide distributed trace queries or a span tree, however, so correlation fields aren't a substitute for tracing.

Failure Modes Before Rollout

The limitations are sharper than the marketing category suggests. Aggregate metrics can't answer which individual users took a path, replay what they saw, or support a deletion-by-user workflow. Infrai logs also have no delete-by-user interface and no bulk export or subscription interface. If the support product must fulfill per-user erasure across analytics records, Infrai isn't suitable for that workflow; choose a system with user-level deletion or keep identifying data out of this telemetry from day one.

There are operational gaps too. Infrai has no alert or notification route, so threshold checks require polling the query API and delivering notifications elsewhere. The query capability's discovery parameters are undeclared, which means I would validate its runnable example before depending on filters. Don't invent query parameters from intuition.

No synthetic checks or heartbeat monitoring are included. A dashboard can't tell you that a scheduled reconciliation job never started if no event was emitted, so pair it with a tool such as Healthchecks for silent failures. There is also no source-map processing, crash symbolication, Electron minidump parsing, or session replay. Those are product boundaries, not small checkboxes.

This is the central trade-off. I wouldn't call one API the universal "best." The lightweight choice wins only while the desired output is aggregate operational truth. Once the questions become behavioral, forensic, or cross-service, use the system designed for that question: PostHog, Amplitude, or Mixpanel for product behavior; Sentry for error investigation; Datadog for a broad hosted operations suite; or OpenTelemetry plus an appropriate backend for distributed traces.

How Should a Product Analytics Style Metrics Dashboard API Be Chosen?

Run the design on paper before integrating anything. List the exact dashboard questions, then count how many require a user identity, an arbitrary event sequence, a raw incident, or a trace tree. If several do, the thin metrics approach will push complexity into your application and is no longer thin.

Also estimate dimension cardinality. Five outcomes and four agent versions make 20 combinations before model route, environment, queue, or channel enters the picture; customer IDs and ticket IDs grow without bound. Decide how quickly an operator must learn about a regression. Polling can suit a low-urgency founder dashboard, but it's the wrong alert path for a strict incident response target.

Finally, test the chart definitions against real workflow semantics before launch: does "resolved" mean the agent answered, the customer accepted the answer, or the ticket stayed closed for 24 hours? Instrumentation can't repair a vague product definition. A precise counter beats a busy event stream.

For this support-agent loop, I would ship aggregate turn metrics first, connect spikes to separately stored logs and errors, and add tracing only when cross-service diagnosis warrants it. I would move to PostHog, Amplitude, or Mixpanel when user behavior becomes the question. That's a narrower recommendation than "install analytics," and a more useful one.

DEV Community