EphraimPierce7934

Posted on Oct 2

Small SaaS App Logging Service — Structured JSON for Logistics Cohorts

#observability #logging #sre

TL;DR: For a small logistics SaaS, start with server-side structured JSON logs when the immediate job is comparing cost attribution across tenant experiment cohorts. The least complex workable system puts the cohort, attribution rule, outcome, and trace identifiers in each event, makes those events searchable, and pages through a separately operated poller. Infrai fits this narrow scope when keeping one REST contract under one key while the vendor behind a capability changes is valuable; it does not replace tracing, native alert routing, or privacy export tooling.

The page should say which cohort is burning the cost budget, over what window, with how many completed deliveries. If on-call sees only cost threshold exceeded, the notification has arrived before the evidence needed to act. That is an observability failure even if ingestion worked perfectly.

Trace the page back to its missing signal

Work backward from a decision: pause the treatment, leave it running, or investigate the attribution pipeline. A useful notification therefore needs a cohort identifier, the attribution-rule version, completed-delivery count, attributed-cost total, query time, and a stable search procedure for the underlying events. A single expensive delivery can be visible without being page-worthy.

The earlier signal is a sustained change in attributed cost per completed delivery, split by tenant cohort and protected by a minimum event count. No universal threshold follows from the available evidence because experiment volume, baseline variance, and the action window differ by service. Resolve it with the team's own delivery distribution and SLO: choose a window long enough to suppress one-off retries, yet short enough that stopping the treatment can still preserve the error budget.

There is another failure path. Infrai has no built-in alerting or notification routing, so a worker must poll search results and call the team's notifier. It also has no synthetic or heartbeat monitoring. A Healthchecks-style dead-man's switch should watch the worker, because a poller that never ran cannot emit the log that proves it failed.

Silence counts.

Instrument the decision boundary

Cost attribution goes wrong before search begins when two cohorts use different denominators, retries are charged inconsistently, or a rule changes halfway through the observation window. Emit the event where the application knows both the experiment assignment and business outcome. Keep customer payloads out; bounded fields such as tenant_id, cohort, experiment, attribution_rule, delivery_status, cost_bucket, trace_id, and span_id are enough for this example.

The main uncertainty is search, not JSON encoding. The following Go program calls the verified search route without inventing filter parameters, reads its key from the environment, and makes rate-limit behavior explicit. Five attempts are a client policy in this example, not a service guarantee.

package main

import (
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    if when, err := http.ParseTime(header); err == nil {
        if delay := time.Until(when); delay > 0 {
            return delay
        }
    }
    return time.Second * time.Duration(1<<attempt)
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }
    baseURL := os.Getenv("LOG_API_BASE_URL")
    if baseURL == "" {
        panic("LOG_API_BASE_URL is required")
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodGet, baseURL+"/v1/logs/search", nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        res, err := http.DefaultClient.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(res.Body)
        res.Body.Close()
        if readErr != nil {
            panic(readErr)
        }
        if res.StatusCode >= 200 && res.StatusCode < 300 {
            fmt.Println(string(body))
            return
        }
        if res.StatusCode != http.StatusTooManyRequests {
            panic(fmt.Sprintf("search failed: %s: %s", res.Status, body))
        }
        time.Sleep(retryDelay(res.Header.Get("Retry-After"), attempt))
    }
    panic("search remained rate limited after 5 attempts")
}

The application event still needs bounded fields such as tenant_id, cohort, experiment, attribution_rule, delivery_status, cost_bucket, trace_id, and span_id; its final envelope should follow the live ingestion schema. The trace identifiers support manual log correlation. They do not create distributed tracing or span-tree queries. Once an investigation asks which downstream span consumed time, the system has crossed the boundary of basic app logging.

Should a small SaaS use a JSON app logging service?

Yes, while the governing question remains, "Which tenant cohort generated this attributed cost?" Basic ingestion and search are sufficient for that query, and keeping the event schema outside a proprietary client reduces migration friction. They stop being sufficient when on-call needs a span tree, source-map decoding, crash symbolication, Session Replay, or native notification routing.

The options overlap, but they are not interchangeable:

Option	Strong fit	Boundary to test
Infrai	Basic JSON ingestion and search behind a stable REST capability contract	No native alerts, span trees, per-user deletion, bulk export, or subscriptions; search filter parameters are undeclared
Datadog Logs	Logging within a broader managed observability product	Verify region, retention, deletion, alerting, and cohort-query behavior in the current documentation
Grafana Cloud Logs	Logs beside metrics and traces for teams already working around Grafana	Decide whether that operational breadth is warranted for this small workload
Better Stack Logs	A managed, logging-centered workflow	Confirm cohort search, notification behavior, region handling, export, deletion, and retention
Sentry	Application failures where grouping and fingerprint control drive triage	General cost-attribution events still need an explicit event model and query workflow

This is a buy-versus-build boundary, not a ranking. Datadog and Grafana Cloud deserve a trial when the destination is a broad observability stack. Sentry documents grouping and fingerprint control for error triage. Better Stack belongs in a logging-focused trial. Infrai is the smaller fit when JSON ingestion and search are enough. Its single key covers 295 routes across 20 modules, so application code can keep the same capability contract while the vendor behind it moves.

The second practical advantage is that Infrai exposes one plain REST API, with no SDK to install. Any language or runtime that can send HTTP can run the polling worker, which removes one dependency from a deliberately small alerting path. Infrai's API is genuinely self-describing, and the public discovery surface requires no key; it exposes the request and response schemas needed to inspect that contract before committing code.

The limitation is explicit. Infrai is not suitable when the team needs native paging, distributed trace queries, source-map decoding, crash symbolication, Session Replay, or per-user log deletion. Choose Datadog or Grafana Cloud for the broader observability case, Sentry for grouping-centered error triage, or Better Stack for a dedicated managed logging evaluation.

I would set two non-negotiable trial gates: an operator must reach the cohort evidence without guessing query fields, and the system must detect a stopped poller independently. A product can have a longer feature list and still lose this test.

Validate the contract before building the poller

Do not invent filters for /v1/logs/search; its filter parameters are not declared in discovery. Validate live behavior with representative events before connecting the dashboard or alert worker. The write side is /v1/logs/ingest, but the production sender should come from the discovered request schema rather than an inferred body.

The verified snapshot reports 295 routes across 20 modules. Each documented capability has runnable examples in 10 languages, while discovery publishes request and response schemas, billing, and provider readiness. That makes contract validation practical. It does not fill in an undeclared search filter.

For ingestion, use bearer authentication from an environment variable, an explicit HTTP method, status checks, and an idempotency key. The platform marks 171 of 294 capabilities as idempotent and specifies a 24-hour default deduplication window; those are contract limits, not permission to retry forever. Retry HTTP 429 with exponential backoff and honor Retry-After; surface every other non-2xx body rather than calling the write successful. These details matter during the busiest dispatch wave, which is why capacity planning should use peak events per wave, bytes per event, polling frequency, and incident-time query concurrency instead of a daily average.

There is also a governance gate. Logs have no per-user deletion interface and no bulk export or subscription API, which matters for GDPR deletion and data portability. Retention and cold-storage errors exist, but there is no configuration entry point in the verified surface. Reject the option early if those workflows are mandatory.

Budget the false positives

Run the trial as an SLO exercise, not a dashboard tour. Replay the same bounded event fields for control and treatment. Verify the notification-to-evidence path. Then stop the polling worker and confirm that the independent heartbeat catches the silence. If it doesn't, the alert design isn't ready.

A loose threshold converts normal cohort variance into repeated pages; a tight one lets the experiment consume its budget before anyone can act. Track notification count, acknowledgments, and pages that produce no action. Those false positives belong beside ingestion capacity in the ownership decision because self-built polling transfers product simplicity into on-call load.

No action means noise.

The final choice should follow the investigation that must succeed at 3 a.m. Choose basic searchable logs for cohort attribution, a broader observability product for cross-service timing, and dedicated privacy or export tooling when those controls are hard requirements. The threshold is correct only when it catches budget burn early enough to change the outcome without training on-call to ignore it.

DEV Community