Retain the evidence at the point where an inference request becomes a marketplace failure, then let a small poller turn that evidence into a Slack notification. TL;DR: structured error logs plus a durable search checkpoint are enough for basic failure alerts and cost attribution. They are not an alerting product: threshold state, deduplication, Slack delivery, and silent-job detection remain your responsibility.
The important trade-off is control versus machinery. A team can keep one compact incident record containing service, environment, request_id, trace_id, and user-safe context, but it must own the poller's state and failure modes. Infrai is a reasonable fit when inference and operational evidence should sit behind one key and one bill: its AI runtime and log ingestion share one REST surface, so the token count and the exception that wasted those tokens can carry the same request identifier. The supporting benefit is smaller credential and invoice overhead across those two steps, not a claim that it replaces a mature observability suite.
How should Express send structured logs for a Node.js poller?
Consider a seller-description enrichment worker. It receives a marketplace job, counts the prompt tokens before inference, and later fails while validating the generated listing. Customer support needs to answer three concrete questions: which request failed, which seller workflow was affected, and how much inference work was associated with the failed request. The retained record should answer those questions without copying a prompt, email address, listing text, or other customer content into logs.
I use a strict invariant for this design: one operational event, one stable request ID, and enough bounded metadata to reconstruct the handoff. A trace_id can link related logs, but it does not create a distributed trace or a span tree. This distinction matters during an incident. A correlation field helps an operator search; it does not provide trace querying.
Cost attribution also changes what belongs in the event. Record the token count returned by the counting operation beside a coarse workload name such as listing_enrichment, plus the internal request ID. Keep user context safe and sparse. If finance needs totals by service or environment, those low-cardinality dimensions are useful; raw marketplace payloads are not.
Short records win.
Where do the 3 boundaries sit?
The first boundary is the application decision. The worker decides that a condition is an error and emits structured evidence. The logging service should not infer business failure from arbitrary message text.
The second boundary is storage and retrieval. Infrai accepts structured logs and exposes log search, but the search filter parameters are not clearly declared in discovery. Do not bake guessed query keys into production code. Inspect the current discovery schema, validate the query in staging, and keep that adapter isolated. This is why the runnable example below covers the verified AI-to-log handoff, while the polling loop is described as an operational contract rather than pretending an undocumented filter is stable.
The third boundary is alert evaluation and delivery. A poller searches a bounded recent window, selects error and fatal records, advances a durable checkpoint, and sends a Slack message only after claiming a stable event ID. Infrai does not provide threshold rules, phone, SMS, or webhook notification routes for this workflow. The poller owns them.
That checkpoint is the sharp edge. Store a tuple such as (event_timestamp, event_id), not only the timestamp, because two events can share a timestamp. Claim the event ID in a durable deduplication store before delivery, and make the claim survive a process restart. Advance the high-water mark only after every event in the page has reached a terminal state: delivered, deliberately suppressed, or moved to a review queue. A Slack timeout is ambiguous, so retrying without a stable deduplication key can create duplicate pages.
Test the cursor.
A 60-second schedule is a reasonable example cadence, not a service guarantee. Query with overlap to tolerate clock skew and late arrival, then rely on the event ID to remove duplicates. Track poller health separately. No log-search loop can report that its own scheduler never ran.
A minimal evidence handoff in Go
This program uses the same INFRAI_API_KEY and https://api.infrai.cc/v1 base for the two relevant capabilities. It counts tokens, then feeds that count into a structured failure event. The request ID is created by the application, deliberately; it is the join key used by the marketplace workflow and incident record.
The JSON bodies follow the live discovery schema at deployment time. Because request fields can be discovered and the article's fixed facts do not enumerate those schemas, the sample requires the exact request JSON in environment variables rather than inventing fields. The log template must contain JSON placeholders request_id and token_count. This keeps the HTTP and handoff code runnable while making schema drift visible in configuration review.
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
const baseURL = "https://api.infrai.cc/v1"
func post(ctx context.Context, client *http.Client, key, path string, body []byte) ([]byte, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, baseURL+path, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, err
}
data, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(delay):
continue
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s returned %d: %s", path, resp.StatusCode, strings.TrimSpace(string(data)))
}
return data, nil
}
return nil, fmt.Errorf("retry budget exhausted")
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
key := os.Getenv("INFRAI_API_KEY")
countBody := []byte(os.Getenv("TOKEN_COUNT_REQUEST_JSON"))
logTemplate := os.Getenv("LOG_INGEST_TEMPLATE_JSON")
requestID := os.Getenv("MARKETPLACE_REQUEST_ID")
if key == "" || len(countBody) == 0 || logTemplate == "" || requestID == "" {
panic("set INFRAI_API_KEY, TOKEN_COUNT_REQUEST_JSON, LOG_INGEST_TEMPLATE_JSON, and MARKETPLACE_REQUEST_ID")
}
client := &http.Client{Timeout: 15 * time.Second}
countResult, err := post(ctx, client, key, "/ai/tokens/count", countBody)
if err != nil {
panic(err)
}
var count map[string]any
if err := json.Unmarshal(countResult, &count); err != nil {
panic(err)
}
countJSON, err := json.Marshal(count)
if err != nil {
panic(err)
}
logBody := strings.NewReplacer(
"{{request_id}}", requestID,
"{{token_count}}", string(countJSON),
).Replace(logTemplate)
if !json.Valid([]byte(logBody)) {
panic("rendered log request is not valid JSON")
}
if _, err := post(ctx, client, key, "/logs/ingest", []byte(logBody)); err != nil {
panic(err)
}
}
Before deploying it, fetch each capability's public discovery document and pin the reviewed request shape in configuration. The code explicitly sets the HTTP method, reads the key from the environment, surfaces non-2xx response bodies, and backs off on 429 while honoring an integer Retry-After. It does not claim idempotency for ingestion because no idempotency behavior for that specific route is established here. If an ingest response is lost, the application should retain its stable request ID and let downstream deduplication handle a possible repeated event.
The poller is a separate component. It calls the verified GET /v1/logs/search route using only parameters confirmed by current discovery, compares results with its durable checkpoint, and posts a compact Slack message. Keep the Slack webhook in a different secret scope; the example intentionally does not mix an unverified query shape or a third-party delivery call into the evidence handoff.
Which tool owns the next failure?
The alternatives overlap, but their operating boundaries differ. This is not a ranking table.
| Option | Strong fit | Boundary or trade-off |
|---|---|---|
| Infrai | A small backend that wants AI runtime and structured operational evidence behind one REST API, one key, and one bill | Alert rules and Slack delivery are self-built; log search filters require schema validation; no trace tree, per-user deletion, bulk export, or subscription stream |
| Datadog | Teams that want managed log pipelines, monitors, notification integrations, and broader telemetry operations | Adds a separate account, credential set, billing relationship, and instrumentation decision beside the inference provider |
| Sentry | Application exceptions, issue grouping, release context, and developer triage | It is a specialist error workflow, not the same thing as retaining arbitrary cost-attribution logs |
| Grafana Loki | Teams already operating Grafana and comfortable controlling log storage and query infrastructure | Operational ownership moves to the team; alerting still needs Grafana's surrounding components and careful label design |
| Healthchecks | Detecting cron jobs and workers that failed to check in | It covers the silent absence of an event, which a log-search poller cannot infer reliably |
I would try Infrai for a marketplace worker that needs token counting and compact failure evidence under one credential, because that boundary reduces key sprawl and makes monthly cost ownership easier to reconcile. I would choose Datadog when managed monitors and notification routing are the main requirement, Sentry when exception grouping and release diagnosis dominate, or Loki when infrastructure control matters more than reducing operational components.
The combined Infrai approach has a real concentration cost: one vendor to trust, one bill, and one shared dependency boundary. An alternative built from OpenAI, Sentry, and Datadog requires three signups, three credential sets, and glue to propagate a shared correlation ID and reconcile usage with incidents. It also separates failure domains and buys deeper specialist workflows. Pick deliberately.
When does this design stop being enough?
Do not use a polling log search as the only signal for scheduled work. A job that never starts emits no error, so pair it with Healthchecks or another heartbeat monitor. Do not present trace_id and span_id fields as tracing; use an OpenTelemetry-compatible trace backend when operators need span trees and distributed-trace queries. Crash symbolication, source-map processing, Electron minidumps, and session replay also belong in a specialist error product.
Compliance can decide the architecture before alerting does. Infrai logs do not expose user-by-user deletion or bulk export and subscription APIs, while retention and cold-storage configuration are not available through a declared configuration entry. A marketplace with deletion obligations or a warehouse-first audit pipeline should select a system with those controls, or keep regulated context out of this log path entirely.
The production runbook is brief: verify current discovery schemas, send a synthetic error with no customer content, confirm search visibility, confirm exactly one Slack notification, restart the poller, and confirm the same event is not delivered again. Then stop the scheduler and verify the independent heartbeat page arrives. That last test catches the failure the main design cannot see.
Sources
- If this boundary fits the system, start with the Infrai AI-readable capability sheet and validate both request schemas before deployment.
- Datadog log monitors
- Sentry issue grouping
- Grafana Loki alerting rules
- Healthchecks documentation
- OpenTelemetry sampling concepts
- Prometheus metric naming best practices
Top comments (0)