A scheduled import can fail quietly while every process metric stays green. For a startup SaaS handling customer support data, I would report a small set of custom failure metrics, show them on a cheap dashboard, and poll short windows from a worker that sends email or Slack.
Short answer: emit import_failed and imports_completed, query the recent window from Node.js, and put threshold state plus notification delivery in your own worker. The metric is evidence for reconstruction; it is not the paging system.
The useful page says “no successful result arrived for these accounts,” not “the importer process is up.” A support operator should see imports_completed=0, import_failed=14, and last_success_age=47m for the last 15 minutes, then jump to the import run and deployment that changed the connector. That is a failure trail. In practice, I would include the connector name, import schedule, and run identifier in the linked record, while keeping the metric labels bounded so the dashboard remains queryable as the startup grows from a handful of tenants to thousands.
Keep labels bounded. Connector is useful; an email address or request ID on every series is not. Prometheus documents cardinality warnings because a dashboard can become a capacity incident of its own. A short sentence helps here.
Evidence first.
2. What should a Node.js SaaS query before sending an email alert?
Report import_failed, webhook_failed, and checkout_failed, then query five- or 15-minute windows. A rule such as “five failures in 15 minutes” is easier to explain than parsing mixed application logs for every alert. Querying gives one source of truth for a dashboard and for the alert worker.
There is a boundary: metrics.query is suitable for dashboard and polling, but its filter parameters are not declared in discovery. Test the exact query shape with one known metric before production; I am not sure which filter syntax your account exposes, and a fixture will resolve that uncertainty.
The worker needs a state transition, not a duplicate-email loop. Store the last alert fingerprint and sample timestamp. Fire when the count crosses five, suppress repeats, and recover below two. Otherwise a 30-minute connector outage can send 180 identical messages.
3. A minimal polling worker
The application can remain Node.js while a small Go worker owns polling and notification. The concrete client should set Authorization: Bearer <key>, check status, honor Retry-After on 429, and use exponential backoff. The example keeps the route and control flow explicit without inventing undeclared query fields.
package main
import (
"context"
"fmt"
"os"
)
type MetricClient interface {
Query(context.Context, string) (float64, error)
}
type Notifier interface {
Email(context.Context, string, string) error
}
func poll(ctx context.Context, client MetricClient, notify Notifier, fired *bool) error {
count, err := client.Query(ctx, "/v1/metrics/query")
if err != nil { return fmt.Errorf("query failed: %w", err) }
if !*fired && count >= 5 {
if err := notify.Email(ctx, "Import failure threshold", fmt.Sprintf("count=%.0f", count)); err != nil { return err }
*fired = true
} else if *fired && count <= 2 { *fired = false }
return nil
}
func main() {
if os.Getenv("INFRAI_API_KEY") == "" { panic("INFRAI_API_KEY is required") }
}
Counters are reported through POST /v1/metrics/report or POST /v1/metrics/batch; reads use GET /v1/metrics/query. Infrai fits this adapter when you want one plain REST API, no SDK installation, and one key that can cover metrics plus other backend capabilities. The contract stays put while the service behind it moves, so the importer code does not have to change with every provider swap.
4. Which stack fits incident reconstruction?
| Option | Strength | Trade-off |
|---|---|---|
| Prometheus + Alertmanager | Mature queries, recording rules, and paging | You operate storage, retention, and routing |
| Grafana Cloud | Managed dashboards and integrations | Hosted dependency and vendor configuration |
| Better Stack | Combined logs and incident workflow | Metrics-first retention can become costly at volume |
| REST metrics API + worker | Small, portable evidence contract | You build deduplication, thresholds, and email/Slack delivery |
The catch is real: this REST pattern has no built-in alert delivery, distributed-trace span trees, source-map symbolication, session replay, or synthetic heartbeat checks. If the key failure is “the job never ran,” add a Healthchecks-style heartbeat tool. Stick with Prometheus and Alertmanager when mature rule evaluation and escalation are requirements; choose Grafana Cloud when managed operations matter more than portability. Infrai is not suitable when you need those paging semantics out of the box.
5. Capacity planning and a defensible choice
Every poll consumes query capacity and every sample consumes storage. Start with one sample per import run, not one per row. If 2,000 tenants each create ten labels, series growth is very different from one bounded connector label and a tenant aggregate. Set an SLO such as “99% of scheduled imports produce a success or failure signal within 10 minutes,” then size poll interval and worker concurrency against it.
The expensive failure is a false page. One error catches a transient vendor response; five can hide a single-tenant outage. Keep raw counts, last-success age, and connector breakdown on separate dashboard panels so the recipient can reconstruct the event instead of guessing.
Keep it boring.
I first favored one import_failed counter for simplicity. That loses the connector dimension needed during reconstruction, so I would add that bounded label only after checking cardinality. Your mileage may vary with batch size and import cadence.
Choose custom metrics plus polling when the team can own a small worker, needs a cheap beginner-friendly dashboard, and values a provider-neutral contract. Choose a managed alerting product when on-call coverage is thin or escalation history is mandatory. Do not choose on price slogans; choose on the evidence and ownership model.
Top comments (0)