TL;DR: For a marketplace, choose the telemetry shape from the incident you must reconstruct, not from the prettiest chart. Keep low-cardinality operational counters for alerting, but record a separate bounded attribution event for each AI operation: tenant, workflow, model family, outcome, latency bucket, and token counts. Aggregate those events into a dashboard and retain the original evidence under an explicit policy. A StatsD-style API, Prometheus Pushgateway, Mixpanel, and Datadog cover different slices of that flow; none removes the need to define ownership, cardinality, and retention first.
The practical trade-off is detail versus cost. A counter such as checkout.agent.failed tells me that something broke. It cannot tell support which marketplace tenant paid for the failed model call, which workflow ran, or whether a retry doubled usage. Putting every customer and request ID into metric labels makes the dashboard expensive and hard to operate. I would split the stream instead: small metrics for rapid health decisions, structured attribution records for reconstruction.
That split is the decision. Everything else is plumbing.
What evidence must survive a customer incident?
Start with the questions support will ask. For a disputed marketplace order, the useful sequence is concrete: which tenant initiated the workflow, which operation ran, how many attempts occurred, which attempt failed, how long each one took, and how many input and output tokens were attributed to it. The dashboard is a projection of that evidence, not the evidence itself.
I would avoid customer email addresses, prompt text, and raw responses in this stream. They are not required for cost attribution, and collecting them expands the privacy and access-control problem. Use an internal tenant key and a generated operation ID instead. Keep the operation ID in structured logs or traces so an authorized responder can join the aggregate anomaly back to a single execution.
For user-facing performance, percentiles beat averages because averages hide the slow tail. The Core Web Vitals guidance uses the 75th percentile as its assessment point for user experience. That does not turn p75 into a universal backend objective, but it is a useful reminder: choose a percentile and population explicitly. A marketplace dashboard might show p50 and p95 latency per workflow, while a customer-experience panel uses p75 for browser signals. Mixing those populations produces a persuasive chart with no stable meaning.
Keep it bounded.
The first schema should be boring enough to inspect during an incident and strict enough to reject surprise dimensions. Imagine a support request that names a tenant and a two-hour window but has no operation ID. The responder should be able to open the daily tenant aggregate, identify the workflow and outcome that changed, narrow the evidence query to that tuple, and then retrieve the individual attempts. This route works because the aggregate and evidence record share controlled dimensions. If either side accepts arbitrary workflow names, model strings, or outcomes, the join slowly fragments into almost-matching values, and the incident becomes a cleanup exercise.
| Field | Purpose | Cardinality rule |
|---|---|---|
tenantId |
Attribute usage and incidents | Internal tenant key; event store only |
workflow |
Compare checkout, search, and support | Controlled enum |
modelFamily |
Explain latency and token shifts | Small allowlist |
outcome |
Count success, timeout, rejection, and error | Controlled enum |
attempt |
Detect retry amplification | Small integer |
inputTokens, outputTokens
|
Attribute usage | Numeric values, aggregated later |
durationMs |
Build latency distributions | Numeric value or fixed buckets |
operationId |
Join evidence during investigation | Never a metric label |
Build the attribution boundary before the dashboard
The following TypeScript example is deliberately vendor-neutral. It validates the dimensions, emits a cheap operational counter, and writes one structured record through interfaces that can be backed by the components already in the system. No transport decision leaks into application code.
type Workflow = "checkout" | "search" | "support";
type Outcome = "success" | "timeout" | "rejected" | "error";
type AiOperation = {
operationId: string;
tenantId: string;
workflow: Workflow;
modelFamily: "small" | "large" | "embedding";
outcome: Outcome;
attempt: number;
inputTokens: number;
outputTokens: number;
durationMs: number;
occurredAt: string;
};
interface Metrics {
increment(name: string, tags: Record<string, string>): void;
observe(name: string, value: number, tags: Record<string, string>): void;
}
interface EvidenceStore {
append(event: AiOperation): Promise<void>;
}
const allowedWorkflows = new Set<Workflow>(["checkout", "search", "support"]);
export async function recordAiOperation(
event: AiOperation,
metrics: Metrics,
evidence: EvidenceStore,
): Promise<void> {
if (!allowedWorkflows.has(event.workflow)) throw new Error("Unknown workflow");
if (event.attempt < 1 || event.inputTokens < 0 || event.outputTokens < 0) {
throw new Error("Invalid attribution values");
}
const metricTags = {
workflow: event.workflow,
modelFamily: event.modelFamily,
outcome: event.outcome,
};
metrics.increment("ai.operation.count", metricTags);
metrics.observe("ai.operation.duration_ms", event.durationMs, metricTags);
await evidence.append(event);
}
Notice what is absent from metricTags: tenantId and operationId. Both are valuable during an investigation, but both can create an unbounded number of metric series. The evidence sink keeps them. A scheduled aggregation can then produce daily tenant totals for dashboard queries without forcing the real-time metrics path to carry customer-level cardinality.
There is also a failure decision to make. If metrics emission fails, the customer request generally should not fail with it. If the evidence write is part of billing or a contractual audit record, losing it may be unacceptable. Put the evidence event onto a durable application-owned path, make its delivery idempotent by operationId, and decide whether the business action may complete before persistence. That policy belongs in the application design, not in a dashboard toggle.
How Should a Node.js Startup Compare StatsD Metrics for Its Dashboard?
A StatsD-style metrics API fits the low-cardinality side. The application increments counters and observes timings through a small interface, which keeps instrumentation easy to replace. Its boundary is equally important: a counter transport does not provide the customer-level evidence store, identity model, or incident join by itself.
Prometheus Pushgateway addresses a narrower lifecycle problem: metrics from short-lived service-level batch jobs that cannot be scraped while they run. Its documentation explicitly warns that it is usually a poor fit for pushing metrics from general applications, and pushed series remain exposed until they are deleted. That persistence changes the cleanup obligation. For a nightly marketplace attribution rollup, the gateway may expose the rollup job's success and duration; it should not become the ledger of per-operation customer evidence.
Mixpanel starts from event analytics and user or group segmentation. That shape can make tenant and workflow exploration natural, but the decision hinges on whether product analytics governance, identity semantics, retention, and access controls match operational incident response. It is an event projection, not automatically the authoritative record for billing disputes.
Datadog combines infrastructure telemetry, application metrics, logs, and dashboards in a managed platform. The integration can reduce the work needed to correlate operational signals, while indexed volume, custom metric cardinality, retention, and account boundaries still need explicit review. Sending every operation ID as a metric tag remains a poor schema even when a backend accepts it.
These are not four interchangeable dashboard skins. They occupy different layers: instrumentation API, batch-job metric bridge, behavioral event analysis, and managed observability suite. Compare the layer the marketplace lacks. If the missing capability is durable tenant evidence, changing the charting layer will not fill it.
Layer first. Product second.
Cost attribution deserves its own query contract. Define totals for input tokens, output tokens, attempt count, and completed operations grouped by tenant, workflow, model family, and a fixed time window. Keep currency out of the raw event. Rates change; measured usage should not. A later calculation can apply the appropriate rate table without rewriting historical telemetry.
Operate it without turning telemetry into the incident
Test the event contract like a payment boundary. A unit test should reject unknown workflows and negative token counts. An integration test should replay the same operationId and prove that durable attribution does not double-count it. Before deployment, send synthetic events for every allowed outcome and verify that the aggregate, evidence lookup, and retention job all agree.
Watch the watcher, too. Track rejected events, evidence-write lag, aggregation freshness, and the gap between emitted operations and stored operations. Those four signals reveal a broken pipeline before a customer asks why the dashboard total moved. Keep the health metrics free of tenant IDs.
Roll out one workflow first. Checkout is a sensible candidate only if its support burden makes the added evidence worthwhile; a lower-risk workflow may be better for validating the schema. During the first retention window, inspect dimension growth and query latency, then remove fields that no investigation uses. More data is not automatically more evidence.
Access should follow the purpose of each layer. Engineers need aggregate service health. Support may need a restricted lookup by tenant and time window. Finance needs stable usage exports. Few people need raw operation-level records, and dashboard links should not quietly grant that access.
The final operational check is prose because it is a chain, not a shopping list. Confirm that every business operation receives one stable ID, retries retain that ID while incrementing the attempt, metric labels come from allowlists, evidence writes are durable and idempotent, dashboard totals expose freshness, and retention deletion is exercised in a test environment. Then rehearse one incident from an aggregate spike to a single authorized evidence record. If that path cannot be completed, the dashboard is decoration.
For a small team, this design keeps the reversible choice cheap. Application code depends on two narrow interfaces, operational metrics stay bounded, and customer-level evidence has an explicit owner. Choose the evidence contract first; choose its transport and charts after the reconstruction drill works.
Top comments (0)