Picking a managed metrics dashboard for a small Node.js startup

#observability #metrics #monitoring #startup

TL;DR

If you're a five-person startup shipping a Node.js API and you want a metrics dashboard by Friday, send your telemetry to a managed backend and keep only the instrumentation layer inside your own repo. The alternative — standing up a time-series database, an object store for long-term blocks, and a dashboard service — puts three more components on an on-call rotation that hasn't earned its first SLO yet. Settle the wire format now and treat the backend as a config line you can change later.

I own the platform team's roadmap, which in practice means I'm the person who defends the monitoring bill in a budget review and also the person who gets paged when a disk fills at 03:00. Those two jobs pull in opposite directions, and most of the advice online is written by people who only hold one of them.

Usually the pager wins the argument.

Should a startup run its own metrics stack, or pay for a managed dashboard?

Start with capacity, because that's the step everyone skips before signing anything. A moderately instrumented Node.js API — say 40 HTTP routes, two queue workers, default runtime and event-loop metrics, one latency histogram with ten buckets — sits somewhere around 3,000 to 8,000 active series per process. Multiply by replicas. Multiply again by every environment you keep alive, including the staging cluster nobody admits to. You are at 50k active series before a single engineer has written a custom counter, and a self-hosted scraper will chew through that on a 2 GB VM without noticing. It will still be fine at 500k. Past a few million active series you're into sharding, remote storage, and a retention argument with whoever pays for object storage — that's the point where the self-hosted route stops being free and turns into a project with a headcount attached.

None of that work is hard. It's just never zero.

Dimension	Self-hosted stack	Managed metrics backend
Time to first dashboard	1–3 days	under an hour
Who owns retention	you, plus the storage bill	vendor, per plan tier
On-call surface	storage tier + query service + dashboard service	a status page and an API key
Data residency	wherever you run it, Europe or US	only the regions that vendor operates
Query portability	config and rules are yours	dialect and dashboards are theirs
Cardinality blast radius	your disk	your invoice

The row that decides it for most teams isn't cost, it's the residency line. If your customers are in Europe and your data processing agreement says telemetry stays in the region, the question stops being buy-versus-build and becomes which managed vendors actually operate a European region — several do, some only on their higher tiers, and a couple will happily let you sign before you notice the ingest endpoint is in the US. That's a procurement question. Answer it in week one, not after you've built 40 dashboards on a backend you have to leave.

The failure mode nobody budgets for: a 200 that quietly drops the series

Last year I spent an afternoon staring at a payments dashboard that had gone flat. Not spiky, not gappy — flat. The panel rendered, the axis was drawn, the line simply stopped at 14:05 and never resumed, and every push our collector made was coming back 200. The collector had accepted each batch, acknowledged it, and dropped it, because a relabel rule I'd written three weeks earlier matched a label we had just started emitting on a new checkout route, and the whole family was being filtered out before it ever left the box. Nothing logged at warning level. No alert fired, because the alert was defined on a threshold and a series that doesn't exist never crosses a threshold. We found out at 19:40 when someone in finance asked why the refund counter had read zero since lunch — roughly 5 hours of silent data loss on the one dashboard the company actually looks at. I'm still not sure why the exporter didn't shout about it; as far as I can tell it did count the drops, on an internal metric we weren't scraping, which is its own small joke.

That story is the entire argument for verification, and it's why I don't trust a green ingest status page as evidence of anything.

The Prometheus instrumentation guide is blunt about the mechanism underneath: every distinct label combination is another time series, so a user ID or a raw URL path in a label turns a tidy counter into a memory leak with a graph attached. On a self-hosted stack that mistake fills a disk. On a managed backend billed by active series it lands on an invoice instead. The failure moves; it doesn't go away, and the managed option quietly converts an availability problem into a finance problem, which is a trade-off worth making with your eyes open rather than by accident.

A thin instrumentation layer that survives swapping the dashboard vendor

The rule I hold to is that exactly one file in the application knows which backend we ship to. Everything else talks to a local registry, exposes a standard text endpoint, and stays ignorant. If the app emits standard exposition and OpenTelemetry-shaped names, a backend swap is a change to an agent config and a DNS entry, not a refactor across 60 route handlers.

// metrics.js - the only module that knows anything about a backend
const client = require('prom-client');

const registry = new client.Registry();
registry.setDefaultLabels({ service: 'checkout-api', region: process.env.REGION });
client.collectDefaultMetrics({ register: registry });

// Bounded label sets only. route is the matched pattern, never req.url.
const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request latency by matched route',
  labelNames: ['route', 'method', 'status'],
  buckets: [0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
  registers: [registry],
});

async function exposition() {
  return registry.metrics();
}

module.exports = { httpDuration, exposition, registry };

Two habits keep that honest over time. Labels come from a fixed vocabulary — the matched route pattern, never the request URL, and status as a class rather than every distinct code you'll ever return. And the series budget gets enforced by a machine in CI, because a rule that depends on reviewers remembering it during a Friday merge isn't a rule.

How I verify the dashboard is telling the truth, and how I roll it back

Verification is a build step here, not a ritual. The Go tool below scrapes one instance, counts series per metric family, and fails the pipeline when a family exceeds the budget the app team agreed to — the cardinality gets caught in CI, where it costs a rerun, instead of in production, where it costs a retention window.

// cardguard: fail the deploy when a metric family outgrows its series budget.
package main

import (
    "bufio"
    "fmt"
    "net/http"
    "os"
    "strings"
)

// Per-family budgets, not one global number: a single runaway label set
// should not hide behind an otherwise healthy total.
var budget = map[string]int{
    "http_request_duration_seconds": 900,
    "queue_job_total":               120,
}

func main() {
    resp, err := http.Get(os.Getenv("SCRAPE_TARGET"))
    if err != nil {
        fmt.Fprintln(os.Stderr, "scrape:", err)
        os.Exit(2)
    }
    defer resp.Body.Close()

    series := map[string]int{}
    sc := bufio.NewScanner(resp.Body)
    for sc.Scan() {
        line := strings.TrimSpace(sc.Text())
        if line == "" || strings.HasPrefix(line, "#") {
            continue
        }
        name := line
        if i := strings.IndexAny(line, "{ "); i > 0 {
            name = line[:i]
        }
        series[family(name)]++
    }

    over := false
    for fam, limit := range budget {
        if n := series[fam]; n > limit {
            fmt.Printf("%s: %d series, budget %d\n", fam, n, limit)
            over = true
        }
    }
    if over {
        os.Exit(1)
    }
    fmt.Printf("ok: %d families within budget\n", len(budget))
}

// family folds _bucket/_sum/_count back into one histogram family.
func family(name string) string {
    for _, suffix := range []string{"_bucket", "_sum", "_count"} {
        name = strings.TrimSuffix(name, suffix)
    }
    return name
}

The second half of verification is a heartbeat you can alert on. A tiny job bumps a counter every minute, and the alert fires on the absence of that series for ten minutes rather than on any threshold — an absence rule would have caught my flat payments panel in ten minutes instead of five hours, and it's the one alert I now write before I write any others. Pair it with an availability SLO on ingest that you measure from your own side, because a vendor's own uptime number is measured at their edge and yours is measured at yours.

Rollback needs to be boring. Keep the local exposition endpoint alive after you cut over, keep a small local scraper with 24 hours of retention running beside it for the first two weeks, and make the managed agent's destination an environment variable so backing out is a redeploy rather than an archaeology project.

The catch is that all of this assumes your questions fit the metrics model — pre-aggregated counters and histograms with bounded labels. If you need arbitrary group-bys after the fact, per-request attributes at high cardinality, or debugging by slicing on a customer ID, a metrics dashboard is not a good fit at any price and you'd be better off with a wide-event or tracing backend, accepting the ingest cost that comes with it. Same conclusion if regulation keeps telemetry inside your own network: managed backends don't support that by definition, so stick with self-hosting and staff it properly. Your mileage may vary on the middle ground — I've seen teams run a small local scraper for infrastructure and a managed backend for product metrics, and honestly that hybrid has held up better than either purist position I've argued for in the past.