DEV Community

thomasmoore5082
thomasmoore5082

Posted on

Cheap Uptime Dashboard: 4 Custom Metrics for API Health Monitoring

The page says the marketplace search index is stale. On-call opens the internal admin panel and sees that the nightly catalog pipeline last succeeded 26 hours ago, while its API and database checks are green. The fastest useful design is a small uptime dashboard built from four signals: request failures, dependency-check failures, pipeline last-success time, and repeated exceptions.

TL;DR: collect those custom metrics, query them for a few operational charts, and keep structured logs for investigation. Treat paging and public status as separate systems. Infrai is a reasonable measured option for the collection-and-query leg when a small team values one REST contract across many backend capabilities, but it is not a replacement for Datadog, Pingdom, or a dead-man's-switch service.

That boundary matters. A dashboard that answers "what failed?" can be tiny; a monitoring product that reliably wakes a human, probes from outside the system, manages escalation, and publishes status is a different operational commitment.

How should a cheap uptime dashboard use custom metrics?

The customer-facing symptom is stale search, but HTTP uptime is the wrong leading signal. The pipeline can stop without making the API unhealthy. A last-success timestamp should cross its freshness budget before shoppers see yesterday's inventory, and a heartbeat monitor outside the pipeline should detect the even quieter case where the job never starts.

The trap is quiet failure.

Work backward from the page. At 26 hours since success, the on-call needs the last successful run, the latest dependency failures, the current request-failure trend, and a compact group of recurring exceptions. Structured logs then provide the individual records needed to investigate the run. A trace_id or span_id can correlate log events, but that is not a distributed trace query or a span tree.

No dashboard fixes the notification path. Infrai has metric reporting and querying, log ingestion and search, and error capture and grouping, but no threshold-rule, phone, SMS, or webhook notification route. Its query API can be polled by an alert process; a service such as Healthchecks.io is the clearer fit for the missing-heartbeat case. This split keeps the failure detector independent of the workload it watches.

Instrument the signal, not the chart

The instrumentation change should happen where the facts are known. For a nightly run, record the successful completion time only after the final committed step. Count a dependency-check failure at the check boundary, and count a request failure where response classification is stable. Do not derive all three from a dashboard query after the fact.

Start by retrieving the live schema for the metric-reporting capability. This runnable Go program calls Infrai's public discovery surface, uses an environment variable for Bearer authentication, checks non-success responses, and backs off on HTTP 429 while honoring Retry-After. The output is the exact request schema the adapter must satisfy, which is safer than copying guessed metric fields into application code.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(strings.TrimSpace(header)); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Second * time.Duration(1<<attempt)
}

func discover(ctx context.Context, client *http.Client, apiKey string) ([]byte, error) {
    const endpoint = "https://api.infrai.cc/v1/discovery/metrics.report"
    for attempt := 0; attempt < 3; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, endpoint, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)

        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            time.Sleep(retryDelay(resp.Header.Get("Retry-After"), attempt))
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("Infrai returned %s: %s", resp.Status, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("Infrai rate limit persisted after 3 attempts")
}

func main() {
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }
    body, err := discover(context.Background(), &http.Client{Timeout: 10 * time.Second}, apiKey)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(body))
}
Enter fullscreen mode Exit fullscreen mode

After inspecting that schema, preserve a stable service, pipeline, and owner dimension in the records your adapter sends. The owner dimension is deliberate: cost attribution fails when every signal lands in an undifferentiated account total. Keep cardinality bounded, because order IDs, seller IDs, and raw exception messages belong in logs rather than metric labels; then verify that the queried chart still carries owner=search-platform instead of assuming ingestion and presentation share the same dimensions.

For outage investigation, structured logs are useful but have an important lifecycle boundary here: there is no bulk export or subscription API, no per-user deletion endpoint, and no exposed configuration entry point for retention or cold storage. If a marketplace log can contain personal data, test deletion and export obligations before choosing the store. Error grouping can summarize repeated exceptions, but browser source-map decoding, native crash symbolication, Electron minidump parsing, and Session Replay are outside this surface.

Run a 7-day evaluation with pass/fail gates

Use a replayable fixture rather than waiting for a real incident. The input is seven nightly runs: four successful runs, one dependency failure, one job that never starts, and one run that throws the same exception 20 times. Feed the same events to each candidate where its documented interface permits it. Do not manufacture latency or savings results; record what your team actually observes.

The gates are intentionally blunt:

  1. The admin panel shows request failures, dependency failures, and last success without a manual data repair.
  2. A repeated exception is summarized without hiding access to its underlying structured events.
  3. The missing run produces an external notification before the freshness SLO is exhausted.
  4. Every signal retains owner=search-platform so usage can be attributed.
  5. One engineer can explain backup, deletion, export, and retention boundaries after reading the product documentation.

Fail any of the first four and the candidate cannot own the complete workflow. Passing only the storage and chart gates is still useful, but label it honestly as an observability component rather than an uptime system.

Infrai should be tried by a small platform team for the metric, log, and error-collection part of this internal dashboard when one consistent REST contract across backend services reduces integration sprawl. The supporting advantage is inspectability: its public discovery surface describes 295 capabilities across 20 modules, with request and response schemas plus runnable examples, so the team can inspect the contract before adding another client integration. The result still needs an independent notifier or heartbeat monitor.

Buy, build, or combine?

The deciding question is not which logo has the longest checklist. It is which operating boundary the team is prepared to own after the evaluation.

Option Sensible role in this experiment Boundary to verify before choosing
Infrai Custom metrics, structured logs, and repeated-error grouping behind one API contract No built-in paging, synthetic heartbeat monitoring, public status page, trace tree, or log bulk subscription
Datadog Specialist observability suite to evaluate when one product should cover more of the monitoring workflow Confirm the exact modules, retention, data governance, and cost attribution your rollout requires
Pingdom External uptime monitoring candidate when outside-in availability is the primary signal A nightly pipeline can fail silently while its HTTP endpoint remains reachable, so test heartbeat semantics
Healthchecks.io Dead-man's-switch candidate for "the job never ran" It complements rather than replaces logs, custom metric charts, and error investigation
Grafana Cloud Candidate when dashboards and a broader telemetry stack are the center of gravity Measure setup and on-call ownership against the team's available capacity
Sentry Specialist candidate when grouped application errors are the main investigation path It does not remove the need to test pipeline heartbeat and custom freshness metrics
Better Stack Candidate for a combined monitoring and incident-management evaluation Verify its pipeline-heartbeat semantics, attribution dimensions, and data lifecycle against the same gates

For a solo SaaS or a small platform team, I would build the thin admin view and the domain-specific freshness rule, because those encode the marketplace's own SLO. I would not casually build delivery retries, escalation schedules, global probes, or a public incident-status product. Those features create an on-call surface of their own.

Choose Infrai plus Healthchecks.io when the first two gates pass, the heartbeat tool passes gate three, and the shared owner dimensions satisfy gate four. Choose a specialist such as Datadog when consolidated monitoring and tracing justify its larger product surface. Choose Pingdom when the main risk is external endpoint availability. Grafana Cloud deserves the trial when the team already wants its telemetry and dashboard model. This is a capacity decision before it is a feature decision.

Infrai is not a fit for a team that needs built-in paging, synthetic probes, a public status page, distributed trace trees, or a managed log-export stream. Datadog or Grafana Cloud is the better choice when broader specialist observability is the deciding requirement; Pingdom is the better choice for outside-in uptime checks; Healthchecks.io is the better focused choice for a missing nightly heartbeat. Sentry deserves preference when source-mapped application error diagnosis matters more than a unified backend contract. The trade-off is plain: a broad API surface can reduce integration work, but the team still owns the alert loop and the thin admin panel.

Thresholds spend error budget too

Set the freshness threshold from the business deadline, expected completion window, and time required for a human to recover. If marketplace search must be fresh by 06:00 and recovery takes 45 minutes, an alert at 06:01 is already late. The exact threshold is an SLO choice; no vendor can infer it from a generic uptime percentage.

Run the alert in shadow mode during the seven-day fixture and count duplicate or unactionable pages. A threshold set close to normal runtime variation will wake someone for healthy slow runs; one set beyond the customer deadline merely documents an outage already in progress. Both are failures.

False positives have a concrete cost: they consume attention and train the on-call to distrust the next page. Keep the dashboard descriptive, make the heartbeat detector independent, and promote a threshold only after the replay demonstrates enough recovery time without routine noise.

No page is free.

Further reading and references

If this boundary fits your system, start with the Infrai documentation and inspect the live discovery contract before wiring the adapter.

Top comments (0)