TL;DR: For a small logistics SaaS, I would start with a narrow custom dashboard that puts AI-agent cost beside latency, completion rate, and queue depth. Emit one metric record per agent turn, keep tenant and shipment identifiers out of labels, and aggregate by workflow and model. Plausible is better for privacy-friendly web analytics, PostHog for product behavior, and Grafana Cloud for serious operational exploration. A simple metrics API earns its place when one screen must mix product KPIs and backend counters without importing a full observability stack.
The deciding factor is attribution, not the prettiest chart. A carrier-quote agent that costs $18 over a day tells me little. I need to know which workflow spent it, how many jobs completed, and whether retries or slow tool calls drove the number. That is the difference between an operations dashboard and a vanity dashboard.
For the custom-API path, Infrai's practical edge is a single API key across backend capabilities, one bill, and one plain REST API, so there is no SDK to install. The API is genuinely self-describing: its public discovery surface requires no key and returns the request schema, response schema, and runnable examples for a capability. That shortens the wiring job. It doesn't supply the product-analysis model.
What does the dashboard need to answer?
Start with decisions. In this case, an operator needs to see whether quote generation is healthy, while the person paying the model bill needs cost per completed quote. Those views share data, so splitting product events and backend metrics too early creates reconciliation work.
I would put six numbers on the first screen: agent turns, completed workflows, failed workflows, p50 latency, p95 latency, and model cost. Queue depth belongs next to them because a low request latency can hide a growing backlog. Signups and conversion can join the same dashboard, but they should remain separate metric names rather than dimensions attached to every agent turn.
The key ratio is:
const costPerCompletedWorkflow =
completedWorkflows === 0 ? null : totalCostUsd / completedWorkflows;
That null matters. Reporting zero when nothing completed makes a broken agent look free.
Avoid high-cardinality labels. workflow: "carrier_quote" is useful. A shipment ID, prompt, email address, or customer name is not. It raises storage and query costs in metrics systems, and it turns an otherwise modest privacy surface into personal or commercially sensitive data. For an EU/US admin panel, use an internal region or plan cohort only when it answers a real operating question.
The constraint that changes the choice
One person shipping weekly cannot maintain three overlapping definitions of “completed.” The revenue-per-hour lens is blunt: time spent reconciling an analytics event, an application counter, and an invoice export is time not spent improving the workflow.
So the metric contract comes first. Cost must be supplied from the model response or billing metadata that produced it; do not estimate it from a hard-coded price table. Latency should use a monotonic clock. A retry is a separate turn but belongs to the same workflow execution, which lets the dashboard expose retry cost without placing execution IDs in metric labels.
There is also a boundary. Metrics are aggregates. They are poor evidence for “why did shipment 8F3 fail?” Keep logs or error events for that investigation, and connect them with trace and span IDs where available. A metrics chart is not a distributed span tree.
The smallest working Node.js implementation
Start by reading the metric capability rather than guessing its payload. This TypeScript program asks the self-describing API for the exact metrics.report schema and runnable examples. Set INFRAI_BASE_URL to the API's versioned base URL and keep the key in INFRAI_API_KEY; the URL stays out of source control. The discovery surface is public, but using the same authentication setup as the eventual reporter keeps the adapter honest.
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
if (!baseUrl || !apiKey) {
throw new Error("INFRAI_BASE_URL and INFRAI_API_KEY are required");
}
async function loadMetricCapability(attempt = 0): Promise<unknown> {
const response = await fetch(`${baseUrl}/discovery/metrics.report`, {
method: "GET",
headers: { Authorization: `Bearer ${apiKey}` },
});
if (response.status === 429 && attempt < 4) {
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1_000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
return loadMetricCapability(attempt + 1);
}
if (!response.ok) {
throw new Error(`Discovery failed (${response.status}): ${await response.text()}`);
}
return response.json();
}
console.log(JSON.stringify(await loadMetricCapability(), null, 2));
That response is the contract to use for the exporter: full request JSON Schema, response schema, billing data, and runnable examples. This matters because query filters aren't declared; inventing a convenient label filter would turn a copyable sample into fiction.
The next TypeScript file runs with Node.js 22 using npx tsx agent-metrics.ts. It records agent turns in memory, produces a compact snapshot, and keeps the exporter behind one interface. Replace runAgentTurn with the real agent call and pass through the cost reported by that provider or gateway.
import { performance } from "node:perf_hooks";
type Workflow = "carrier_quote" | "shipment_exception";
type Outcome = "completed" | "failed";
type AgentTurnMetric = {
workflow: Workflow;
outcome: Outcome;
latencyMs: number;
costUsd: number;
queueDepth: number;
recordedAt: string;
};
interface MetricSink {
report(metric: AgentTurnMetric): Promise<void>;
}
class MemoryMetricSink implements MetricSink {
readonly records: AgentTurnMetric[] = [];
async report(metric: AgentTurnMetric): Promise<void> {
this.records.push(metric);
}
}
type AgentResult = {
outcome: Outcome;
costUsd: number;
};
async function measureAgentTurn(
sink: MetricSink,
workflow: Workflow,
queueDepth: number,
run: () => Promise<AgentResult>,
): Promise<void> {
const startedAt = performance.now();
const result = await run();
await sink.report({
workflow,
outcome: result.outcome,
latencyMs: performance.now() - startedAt,
costUsd: result.costUsd,
queueDepth,
recordedAt: new Date().toISOString(),
});
}
function percentile(values: number[], fraction: number): number | null {
if (values.length === 0) return null;
const sorted = [...values].sort((a, b) => a - b);
const index = Math.ceil(fraction * sorted.length) - 1;
return sorted[Math.max(0, index)];
}
function summarize(records: AgentTurnMetric[]) {
const completed = records.filter((item) => item.outcome === "completed");
const totalCostUsd = records.reduce((sum, item) => sum + item.costUsd, 0);
return {
turns: records.length,
completed: completed.length,
failed: records.length - completed.length,
latencyP50Ms: percentile(records.map((item) => item.latencyMs), 0.5),
latencyP95Ms: percentile(records.map((item) => item.latencyMs), 0.95),
totalCostUsd,
costPerCompletedWorkflow:
completed.length === 0 ? null : totalCostUsd / completed.length,
latestQueueDepth: records.at(-1)?.queueDepth ?? 0,
};
}
async function runAgentTurn(): Promise<AgentResult> {
return { outcome: "completed", costUsd: 0.012 };
}
const sink = new MemoryMetricSink();
await measureAgentTurn(sink, "carrier_quote", 7, runAgentTurn);
console.log(JSON.stringify(summarize(sink.records), null, 2));
The 0.012 value is fixture data so the file runs; it is not a model price. In production, the agent adapter must replace it with the actual per-call cost. The example also measures one turn, not an entire workflow. Add the workflow execution count at the orchestration boundary so a three-turn success remains one completed quote.
Do not send every raw record from the request path if that adds user-visible latency. A bounded local buffer or queue worker is appropriate, but delivery must be treated as at least once: give each record a stable identity at the adapter boundary and deduplicate downstream. Flush in batches, cap memory, and decide explicitly what happens when the metrics backend is unavailable.
How Should a SaaS Metrics Dashboard Compare Plausible and PostHog?
The products overlap, but they optimize for different questions.
| Option | Best fit | Cost attribution | Main boundary |
|---|---|---|---|
| Plausible | Privacy-conscious web traffic and conversion goals | Possible through custom events, but backend cost counters are not its center of gravity | Limited operational metrics and infrastructure exploration |
| PostHog | Product analytics, funnels, paths, retention, and richer user behavior | Custom events can carry usage facts, but mixing infrastructure counters into product analytics needs discipline | A broader product platform than a narrow internal dashboard requires |
| Grafana Cloud | Operational metrics, dashboards, alerting, and deeper exploration | Strong when cost is emitted as a metric and paired with workload labels | More concepts and stack surface to own for one small app screen |
| Custom metrics API and UI | A fixed set of product KPIs plus backend counters | Direct control over workflow-level cost semantics | You own metric definitions, charts, retention decisions, and alert logic |
Plausible is the cleanest choice here when the real job is site traffic, goals, and a deliberately smaller analytics footprint. I would not bend it into a queue monitor. PostHog is more suitable when the next question is which sequence of product actions led to activation or retention. Its breadth is valuable, but adopting that breadth only for six operational numbers is hard to justify.
Grafana Cloud wins once exploration and alerting become core requirements. If an on-call engineer needs to correlate saturation across services, slice time series repeatedly, and maintain serious alert rules, a handmade dashboard is false economy. The setup cost buys operational depth.
That custom-API option fits the narrow case because adding a metric starts by reading its request and response schemas. Every documented capability ships runnable examples in 10 languages. It remains less opinionated than Plausible or PostHog and much lighter than Grafana Cloud; the application must define and emit its own metrics.
That last sentence is the trade. Outsource the undifferentiated transport, but keep ownership of what “cost per completed quote” means.
Privacy and operational limitations
A privacy-friendly deployment is an architecture choice, not a vendor checkbox. Minimize labels before export, define retention, separate product identity from operational metrics, and document the lawful basis for any personal data that remains. Plausible's design can reduce analytics data collection, but custom properties still deserve review. Self-hosting also does not repair an over-collected event schema.
The lightweight custom route has hard limits. Infrai is not a fit when sophisticated filtering, distributed trace navigation, or managed alert escalation is required; choose Grafana Cloud instead. Query filter parameters are undeclared, so sophisticated metric filtering can't be assumed. There is no basis for treating it as a distributed tracing query system, a session replay tool, a source-map processor, or an Electron minidump symbolizer. Logs may carry trace and span IDs for correlation, but that isn't the same as rendering a span tree.
Alerting needs separate treatment too. Without threshold, phone, SMS, or webhook notification routes, polling a query from a scheduled worker can support a small internal check, but it is code you now own. Silent “the job never ran” failures need an external heartbeat monitor such as Healthchecks. Do not ask the system being monitored to be the only witness that it stopped.
This is where a narrow dashboard stops being cheaper in engineering attention, even if its invoice remains small.
What I would change at scale
First, I would replace the memory sink with a durable batch exporter and preserve the same metric contract. The dashboard should not care where records live. I would also add histogram buckets chosen from the service-level objective, rather than calculating percentiles from raw records forever.
Second, I would introduce OpenTelemetry at service boundaries when the agent loop crosses enough workers and tools that aggregate latency no longer explains delays. Sampling is a real design decision: head sampling is early and cheap, while tail sampling can retain interesting traces after observing their outcome. Neither recovers dimensions that were never recorded.
Finally, I would move to Grafana Cloud or another full observability backend when alert ownership, multi-service exploration, or trace navigation becomes weekly work. PostHog would remain alongside it if product funnels matter; Plausible would remain a sensible, smaller web-analytics choice. Consolidation is useful only while it reduces operational thought.
For a solo SaaS, the decision rule is simple: begin with the six metrics that change tomorrow's work. Keep cost attribution in the schema from day one. Graduate when unanswered questions, not feature envy, consume more hours than the larger platform would.
Further reading
References:
Top comments (0)