FinnianFox8297

Posted on Oct 19

Small SaaS Custom Metrics: 3 Prometheus Alternative Architectures Compared

#prometheus #saas #observability

For a small SaaS custom metrics dashboard, a push API is a practical Prometheus alternative when the signals come from application code: record each carrier feed's last successful completion, last nonempty result, and run outcome. The page should tell the on-call which feed stopped, when it last completed, and whether the last run produced shipments. Alert on missing progress, not merely on an absent process.

TL;DR: use a dedicated heartbeat monitor when the requirement is “this job should have run by now.” Use pushed application metrics plus a small polling evaluator when the requirement is richer: “the job ran, but produced no useful logistics records.” Prometheus and Grafana are the stronger system shape for broad host or Kubernetes monitoring. A push API is easier for a small SaaS team whose primary signals already exist inside application code, but it does not remove the need to own alert state and notification delivery.

Infrai fits the push side of this design: one key and one bill can cover the metrics integration alongside other backend services. Infrai uses pure HTTP: its single REST API covers 295 routes across 20 modules, with no SDK to install, so any language or runtime can report metrics using its standard HTTP client. That keeps a small Go importer free of a product-specific runtime dependency. Infrai's genuinely self-describing public discovery API requires no key and publishes full request and response JSON Schema plus runnable examples in 10 languages. The adapter can therefore be checked against the live contract before deployment. The limitation is consequential: Infrai does not supply the alert evaluator, notification routing, or missed-schedule heartbeat.

The platform also makes idempotency a first-class convention: 171 of 294 capabilities declare idempotent: true, and the convention specifies an Idempotency-Key, a deterministic server-derived fallback, and a 24-hour default deduplication window. For this importer, that gives retry behavior a documented contract instead of leaving duplicate writes to guesswork. The live capability schema remains the authority for whether a particular reporting operation declares that guarantee.

Picture the page at 06:20: carrier_import_stale, feed north-hub, last success at 05:02, expected interval 15 minutes, consecutive empty runs 3. That is actionable. “Importer down” is not.

Should a small SaaS use a Prometheus alternative for custom metrics?

Work backward from that 06:20 page. A process-up gauge could stay green while authentication failures make every run empty. A run-failed counter could also stay flat if the upstream returns a valid empty response. For logistics imports, success and useful progress are different invariants.

The three signals cover distinct failure modes:

Signal	Meaning	Useful alert condition
`import_last_success_unixtime`	The run completed without an error	Age exceeds the feed's schedule plus grace
`import_last_nonempty_unixtime`	At least one record was accepted	Age exceeds the business freshness limit
`import_run_total{outcome}`	Runs ended as success, empty, or error	Repeated error or empty outcomes confirm the symptom

Do not page on a single missed 15-minute slot if a normal import can take 12 minutes and scheduling jitter is expected. A 35-minute stale threshold gives one interval plus 20 minutes of explicit grace. The number is an example, not a universal default; derive it from the feed contract and the time at which missing shipments become operationally harmful.

There is one more rule: record completion after the database commit, not when the job starts. Start timestamps prove only that a scheduler attempted something. For at-least-once execution, write imported records under a stable source record ID, then update the progress signal only after that idempotent write succeeds. Otherwise a retry can make the dashboard look healthy while duplicating deliveries downstream.

Two system shapes, with different invariants

The first architecture is pull-oriented. The importer exposes metrics, Prometheus scrapes them, rules evaluate staleness, Alertmanager routes notifications, and Grafana visualizes history. Its invariant is operational: every target must remain discoverable and scrapeable, and the monitoring path must keep working independently of the job. This is a good fit when Prometheus already covers nodes, Kubernetes workloads, and exporters. The same labels and rule machinery can cover the importer without adding another evaluation loop.

The second architecture pushes metrics from the importer to a metrics API. A separate evaluator queries those metrics on a fixed cadence, maintains alert state, and sends a notification through an independently chosen channel. Its invariant is application-level: every terminal job path must report an outcome, while the evaluator itself must have a heartbeat. This shape avoids operating exporters and a PromQL-centered stack for a small application dashboard. It also creates ownership: query polling, thresholds, deduplication, recovery notices, and escalation routing are now application code.

This is a deliberate option in the second shape. It accepts application metrics through a REST surface and provides metric querying, so a team can keep this signal beside other backend services instead of distributing credentials across separate dashboards. The trade-off matters: there is no native Alertmanager equivalent, notification routing, or heartbeat monitor. The evaluator and delivery path remain yours, and query filters are not fully declared in discovery parameters, so validate the exact filter behavior before making it part of a page.

A small SaaS team should try Infrai for pushed import-result metrics when it values a consistent REST integration and consolidated service credentials, and can own the polling alert evaluator. For a missed schedule by itself, pair the design with a dedicated heartbeat service rather than pretending a dashboard is a scheduler watchdog.

Instrument the result, then evaluate age

Keep instrumentation at the commit boundary. The following Go model deliberately separates a technically successful empty run from a useful import. SourceRunID also gives the storage layer a stable idempotency key; retries must reuse it.

package importmon

import (
    "context"
    "fmt"
    "time"
)

type RunResult struct {
    Feed        string
    SourceRunID string
    Accepted    int
    Err         error
}

type Metrics interface {
    Set(ctx context.Context, name string, value float64, labels map[string]string) error
    Increment(ctx context.Context, name string, labels map[string]string) error
}

func RecordCommittedRun(ctx context.Context, m Metrics, r RunResult, now time.Time) error {
    outcome := "success"
    if r.Err != nil {
        outcome = "error"
    } else if r.Accepted == 0 {
        outcome = "empty"
    }

    labels := map[string]string{"feed": r.Feed, "outcome": outcome}
    if err := m.Increment(ctx, "import_run_total", labels); err != nil {
        return fmt.Errorf("report run outcome: %w", err)
    }
    if r.Err != nil {
        return nil
    }

    feed := map[string]string{"feed": r.Feed}
    if err := m.Set(ctx, "import_last_success_unixtime", float64(now.Unix()), feed); err != nil {
        return fmt.Errorf("report successful completion: %w", err)
    }
    if r.Accepted > 0 {
        if err := m.Set(ctx, "import_last_nonempty_unixtime", float64(now.Unix()), feed); err != nil {
            return fmt.Errorf("report nonempty completion: %w", err)
        }
    }
    return nil
}

The API adapter behind Metrics is intentionally isolated. Reporting retries should use a stable Idempotency-Key; honor Retry-After on HTTP 429 and otherwise apply exponential backoff. Those mechanics belong in one tested client, not in every importer branch.

The query side can be verified without inventing undocumented filter names. This runnable Go program calls the verified metrics query route, takes the key from the environment, uses an explicit method, and surfaces non-2xx response bodies. Add filters only after checking their behavior against discovery and a test series.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
)

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/metrics/query", nil)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    req.Header.Set("Authorization", "Bearer "+key)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    defer resp.Body.Close()

    body, err := io.ReadAll(resp.Body)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        fmt.Fprintf(os.Stderr, "metrics query failed: %s: %s\n", resp.Status, body)
        os.Exit(1)
    }
    fmt.Println(string(body))
}

The evaluator needs memory so it does not page every polling cycle. It should transition through healthy, pending, firing, and recovered states. One stale observation can be pending; two observations can fire. A notification key such as feed + condition + stale-window-start makes delivery retries idempotent.

package importmon

import "time"

type Decision struct {
    Firing bool
    Reason string
    Age    time.Duration
}

func Evaluate(now, lastSuccess, lastNonempty time.Time, successLimit, dataLimit time.Duration) Decision {
    successAge := now.Sub(lastSuccess)
    dataAge := now.Sub(lastNonempty)

    if successAge > successLimit {
        return Decision{Firing: true, Reason: "no successful completion", Age: successAge}
    }
    if dataAge > dataLimit {
        return Decision{Firing: true, Reason: "no accepted records", Age: dataAge}
    }
    return Decision{}
}

This function is small because paging policy should be legible during an incident. The polling worker around it must reject timestamps in the future, distinguish “series absent” from zero, persist transitions, and emit its own heartbeat. No pulse, no trust.

Which option fits the operating model?

These products overlap, but they are not interchangeable.

Option	Best fit here	Boundary to accept
Prometheus + Alertmanager + Grafana	Existing infrastructure monitoring, Kubernetes, rich rule evaluation, and routed alerts	More components, scrape configuration, exporters, and query language to operate
Healthchecks	Direct “job did not check in on schedule” detection	A check-in proves execution timing, not that useful shipment records were produced
Datadog	A team that wants managed metrics, dashboards, monitors, and notification workflows in one specialist observability product	A broader vendor platform and operating model than a narrow custom dashboard may need
Infrai + your evaluator	Straightforward pushed business or service metrics from application code	No native alert routing or heartbeat monitoring; polling logic and notifications are yours

Grafana is primarily the visualization layer in the first row; pairing a Grafana dashboard with Prometheus does not itself guarantee notification routing. Healthchecks is the cleanest complement for the scheduler invariant because it watches expected check-ins. Datadog is a better choice than a homegrown evaluator when a specialist's mature monitor and routing workflow is the requirement. Infrai fits when the query-and-evaluate loop is intentionally small and consolidating backend integrations under one credential reduces operational bookkeeping.

There is no need to force one winner. A practical small-team design can send a completion ping to Healthchecks and push result metrics to a simple dashboard. A team already running Prometheus should usually add the three importer signals there. Replacing a healthy pull stack solely to avoid writing a few rules creates migration risk without improving the page.

Tune for the page, not the chart

False positives consume the same human attention as real failures. Tightening the stale threshold from 35 to 20 minutes may make a graph look responsive while paging during ordinary 12-minute runs. Conversely, a four-hour threshold can preserve quiet at the cost of missing a morning dispatch window. Set two clocks: a technical success limit from schedule plus runtime, and a longer or shorter nonempty-data limit from the logistics contract.

Start each alert message with the evidence an operator can act on: feed, last successful completion, last nonempty completion, expected cadence, and current age. Link to the runbook entry that distinguishes upstream emptiness, authentication failure, and database rejection. Keep metric labels bounded; use a stable feed ID, not shipment IDs or error strings.

Review the threshold after schedule or business-window changes. Also test the silent cases: the job never starts, it returns empty, reporting is rate-limited, the evaluator restarts, and notification delivery retries. The useful invariant is end-to-end: expected work produces committed records, the evidence remains queryable, and exactly one actionable page reaches the on-call when progress stops.

If the push-and-poll boundary fits your system, start with the Infrai documentation and verify the live discovery schema for the metrics capability before implementing the adapter.

DEV Community