callumreed2198

Posted on Oct 13

Product Analytics-Style Metrics API: 4 Server-Side Dashboard Signals for Marketplaces

#metrics #analytics #api

A product analytics-style metrics API is a good lightweight base for a server-side dashboard when a marketplace needs custom counters and charts without a full analytics SDK. The page may say the AI agent loop is slow and expensive, but on-call needs more than red pixels: they need to identify which run changed, which step dominated latency, whether retries inflated cost, and which request ID connects the aggregate spike to raw evidence.

TL;DR: report four backend-generated signals for every completed loop: run count, end-to-end duration, model-call cost, and retry count. Keep stable dimensions such as workflow, outcome, model route, and region, but keep buyer or seller IDs out of metric labels. Use a lightweight metrics API when the job is aggregate incident reconstruction; choose a product analytics system when the question is about individual user journeys.

That boundary matters. A dashboard can tell an operator where to look without becoming a second customer database.

Should a server-side metrics dashboard use a product analytics API?

Work backward from the alert. The useful page is not "latency is high." It shows the affected marketplace workflow, when the change began, whether success volume moved with it, and whether retries or model-call cost rose at the same time. Those four signals let the responder distinguish a slower successful loop from a retry storm, a traffic mix change, or a drop in completed work.

For a loop that drafts a listing description, for example, graph completion count beside duration and cost. Split only on dimensions that lead to an action: workflow, outcome, model route, and deployment region are defensible. A listing ID is not. High-cardinality identifiers make poor chart dimensions and create a privacy burden; preserve a request ID in logs instead, then use it to cross the boundary from a suspicious aggregate to a single execution.

I have been paged by missed jobs and duplicate deliveries. That history creates an idempotency reflex: a counter named agent_runs must represent one logical run, not one delivery attempt. If the worker can receive a message twice, deduplicate before reporting the completion or make the reporting operation idempotent. Otherwise the dashboard will confidently describe the queue's retry behavior as customer demand.

The earlier signal should usually be a rate or distribution change, not a single slow request. Compare a current window with a known operational baseline, and require enough completed work for the ratio to mean anything. There is no universal threshold in the available evidence; traffic shape and incident tolerance decide it. A thin overnight marketplace and a busy regional launch should not share one static trigger.

Put a contract between the loop and the backend

Instrumentation should survive a provider change. The application emits a small domain contract; an adapter sends it to the chosen metrics backend. Swapping the vendor behind that capability then leaves the agent-loop code alone, while the adapter owns transport, authentication, retry policy, and provider-specific naming.

Here is a runnable Go example of that boundary. It inspects the live discovery document, finds the metrics reporting capability by its verified method and path, and prints its schema. This is the honest first step because the query filters and reporting request fields are not declared in the available material. The production adapter can then be generated from the returned schema without contaminating business logic or guessing fields.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strings"
)

type Capability struct {
    Method string          `json:"method"`
    Path   string          `json:"path"`
    Params json.RawMessage `json:"params"`
}

type Discovery struct {
    Capabilities []Capability `json:"capabilities"`
}

func main() {
    baseURL := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
    apiKey := os.Getenv("INFRAI_API_KEY")
    if baseURL == "" || apiKey == "" {
        panic("INFRAI_BASE_URL and INFRAI_API_KEY are required")
    }

    req, err := http.NewRequest(http.MethodGet, baseURL+"/v1/discovery", nil)
    if err != nil {
        panic(err)
    }
    req.Header.Set("Authorization", "Bearer "+apiKey)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        panic(err)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        panic(fmt.Sprintf("discovery failed: status=%d body=%s", resp.StatusCode, body))
    }

    var discovery Discovery
    if err := json.Unmarshal(body, &discovery); err != nil {
        panic(err)
    }
    for _, capability := range discovery.Capabilities {
        if capability.Method == http.MethodPost && capability.Path == "/v1/metrics/report" {
            fmt.Printf("method=%s path=%s params=%s\n", capability.Method, capability.Path, capability.Params)
            return
        }
    }
    panic("metrics reporting capability not found")
}

INFRAI_BASE_URL belongs in deployment configuration so this unlinked comparison does not bake a vendor URL into application code. The key stays in the environment. Discovery is public, but sending the configured bearer credential demonstrates the same client setup the reporting adapter will use; the sample checks non-2xx responses and exposes the response body instead of silently assuming success.

The adapter still needs a durable idempotency decision. A run ID should be claimed in shared storage before reporting completion, because a process-local map cannot protect against two queue consumers. The same rule applies if the transport retries after an ambiguous timeout.

Keep transport details in the adapter. One lightweight option provides direct metric reporting and querying without installing a full analytics SDK, and its broader contract covers 295 routes across 20 modules under one key. That breadth can reduce integration churn when adjacent backend capabilities change. It does not turn aggregate metrics into customer analytics, and the application should depend on a local metrics interface rather than a vendor-shaped object.

Reconstruct the incident, not just the chart

When the alert fires, start with completed-run count. If count fell, the apparent latency improvement may merely mean slow work stopped completing. If count held while duration rose, compare retries and cost over the same window. More retries with higher cost suggests repeated model work; stable retries with higher duration points elsewhere. These are investigative branches, not claims of automatic root cause.

Next, pivot to logs using the request or trace identifiers recorded by the worker. Metrics provide the shape of the incident; logs provide the sequence. The lightweight API can be paired with log or error search for this handoff, but it has no distributed trace query or span tree. Its log records may carry trace_id and span_id, so correlation is possible, yet operators who need a visual critical path should use a tracing system.

There is another gap that matters for scheduled marketplace work. No metric arrives when a job never starts. A completion counter cannot distinguish "nothing was due" from "the scheduler failed silently," and this API has no synthetic or heartbeat monitoring. Use a dead-man's-switch service such as Healthchecks for expected executions, while treating the metrics dashboard as evidence about work that actually ran.

This separation makes the runbook sharper:

Confirm expected work fired in the heartbeat system.
Check completion count, duration, cost, and retries for the affected workflow.
Select a bounded incident window and correlate request IDs into logs or errors.
Escalate to traces when the missing evidence is the cross-service critical path.

Choosing among real alternatives

The deciding question is what must happen after someone clicks a spike. The products below overlap, but they optimize for different investigations.

Option	Strong fit	Important boundary
Infrai metrics API	Backend-generated aggregate counts and timings through one REST API under one key; the application contract can stay fixed while the provider behind the capability changes	Not ideal for user-level analysis: no built-in alert or notification routes, no session replay, and no user-deletion workflow for logs. Query polling and separate alert delivery are required.
PostHog	Product analytics where events, funnels, paths, retention, and session replay belong in one investigation	More product-analytics machinery than a small server-side operational dashboard needs. Review person profiles and deletion controls as part of the data model.
Mixpanel	User-level event analysis, funnels, retention, and cohort exploration	Best when identity and customer behavior are central; it is a different abstraction from a compact set of operational counters.
Datadog	Metrics, logs, traces, monitors, and incident response in an integrated observability workflow	A broader agent and platform commitment than a direct metrics endpoint; valuable when cross-service diagnosis and managed alerting justify it.
Prometheus plus Alertmanager	Infrastructure-style time series, expressive queries, and routed alerts with control over deployment	Operators own collection, storage, cardinality discipline, availability, and upgrades unless they choose a managed service.

This is why "best API" has no useful answer without the drilldown target. The central limitation of the lightweight choice is that aggregate charts stop before user journeys, managed paging, and span trees. Choose PostHog or Mixpanel when support or product teams must follow a buyer's journey. Choose Datadog when on-call needs managed monitors and a metric-to-trace path. Choose Prometheus and Alertmanager when operational control and time-series querying outweigh running the stack. The direct API is compelling when the dashboard remains aggregate and server-side, the adapter boundary matters, and polling is acceptable.

Privacy can force the decision. The lightweight log search surface has no per-user deletion interface and no bulk export or subscription interface. If a deletion-by-user workflow is mandatory, keeping personal identifiers out of logs is the first control, but it does not manufacture a missing lifecycle operation. Select a system whose documented deletion model matches the obligation.

The alert is a product, too

Because the lightweight option has no threshold-rule, phone, SMS, or webhook notification route, alert evaluation must poll the query API and deliver notifications elsewhere. Do not improvise undocumented query filters: the discovery parameters for metrics querying are undeclared. Inspect the live capability schema before implementing the adapter, and pin tests to the fields the schema actually exposes.

Polling introduces three operational choices: evaluation interval, missing-data behavior, and deduplication. Store an alert instance key derived from rule and evaluation window so a delayed evaluator cannot page twice. Mark absent data separately from a healthy zero. Finally, make recovery explicit; an on-call channel filled with repeated "still broken" messages is not a state machine.

The false-positive cost is real. A threshold tuned to one busy region can flap in a quiet one, and a cost alert without a minimum run count can fire on one unusual loop. Every page spends attention and weakens trust in the next page. Start with a dashboard and recorded investigations, establish the baseline from your own traffic, then promote a signal to paging only when the runbook names a concrete action. Keep lower-confidence changes as tickets or dashboard annotations.

The practical decision: use aggregate metrics to get from page to incident window, durable IDs to get from the window to evidence, and a separate heartbeat to prove scheduled work ran. If any of those transitions is missing, the chart is reporting, not observability.

DEV Community