DEV Community

nathanielbrooks0360
nathanielbrooks0360

Posted on

Free API for Centralized Application Logs: Go's 2-Route Query Support Explained

#observability #api #logs

Free API for Centralized Application Logs: Go's 2-Route Query Support Explained

TL;DR: For a small logistics application, a free API for centralized application logs can be enough to reconstruct a nightly pipeline incident with basic query support. The trade-off is operational, not cosmetic: this is a log store, not a complete observability platform. I would use the /v1/logs/ingest and /v1/logs/search routes when a simple shared contract matters more than distributed tracing, alert delivery, export, or compliance-grade deletion.

That boundary is useful. A shipment-status pipeline can produce records from the scheduler, workers, and database adapter in one place, then let an engineer correlate a failed run by request or trace identifiers. It will not page anyone when the pipeline goes silent, draw a span tree, or provide a replay of a user's session. Those are separate decisions.

What does incident reconstruction actually require?

Start with the reconstruction question, not the product list: did the nightly shipment update start, which component last reported progress, and which event explains the final state? The minimum useful answer is a chronological set of structured application events with stable identifiers and enough context to distinguish a run from the next one.

I would define the SLO before choosing the backend. For example, the service may promise that a completed nightly run leaves searchable application events, while the incident process promises that an on-call engineer can identify the last known stage. Those are different promises. A log API can support the first; it cannot automatically guarantee the second when a worker dies before emitting anything.

The capacity calculation is similarly plain. Estimate events per run, bytes per event, runs per day, and the number of days needed for reconstruction. Then add headroom for retries and a failed batch being replayed. Small workloads in US or EU regions are the stated fit here, but that is not a license to skip retention and query-volume tests as the pipeline grows.

For a nightly shipment pipeline, I would put the worksheet beside the runbook instead of hiding it in an architecture diagram. Start with the event count for one successful run, multiply by the maximum number of retries that the worker can produce, and then multiply by the retention window required to investigate a late delivery. Keep the scheduler's start and completion events separate from worker progress, because a missing worker event and a missing scheduler event lead to different next actions. Record the expected query window too: an engineer searching one run should not have to scan every event from every warehouse. The result is a capacity envelope, not a benchmark. If that envelope stops fitting the small-workload assumption, the decision should be revisited before an incident forces it.

No magic.

The incident lesson is bounded: centralized ingestion and a basic search surface reduce the number of places an engineer must inspect. They do not turn missing telemetry into evidence. A silent failure still needs a heartbeat or job monitor such as Healthchecks, and a paging path still needs to poll the query API and build its own notification rule.

Keep the question bounded.

Should I use a free API for centralized application logs with query support?

The useful part of this design is consistency. Application events enter through /v1/logs/ingest; an investigator queries through /v1/logs/search. In a broader backend that already uses one REST contract, adding log storage is one more integration surface rather than another collector, key, and client library. Infrai's practical advantage in this narrow comparison is one key for multiple backend capabilities and one plain REST API, not a claim that it replaces every observability tool.

The sharp edge is that the search route's filtering parameters are not declared in the discovery surface. I would not invent service, since, or trace_id query parameters and call the example production-ready. The request schema should be checked before wiring a search UI; the article's decision is about the verified route capability, not an undocumented query dialect.

The same restraint applies to identifiers. A log record may carry trace_id or span_id for correlation, but there is no distributed-trace query or span-tree capability in the stated scope. That gives an investigator a useful join key when other systems have the trace, not a tracing backend.

Here is the preventative client path I would keep in the application: explicit methods, bearer authentication from the environment, status checks, and bounded retry behavior. The function accepts the already validated JSON body because the exact ingestion schema must come from the route's current discovery schema rather than from an invented field list.

package logs

import (
    "bytes"
    "context"
    "fmt"
    "io"
    "math"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Client struct {
    HTTP *http.Client
    Key  string
    BaseURL string
}

func New() (*Client, error) {
    key := os.Getenv("INFRAI_API_KEY")
    baseURL := os.Getenv("INFRAI_BASE_URL")
    if key == "" || baseURL == "" {
        return nil, fmt.Errorf("INFRAI_API_KEY and INFRAI_BASE_URL are required")
    }
    return &Client{HTTP: &http.Client{Timeout: 10 * time.Second}, Key: key, BaseURL: baseURL}, nil
}

func (c *Client) do(ctx context.Context, method, path string, body []byte) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, c.BaseURL+path, bytes.NewReader(body))
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+c.Key)
        req.Header.Set("Content-Type", "application/json")

        res, err := c.HTTP.Do(req)
        if err != nil {
            if attempt == 3 {
                return nil, err
            }
        } else {
            data, readErr := io.ReadAll(res.Body)
            res.Body.Close()
            if res.StatusCode >= 200 && res.StatusCode < 300 {
                return data, readErr
            }
            if res.StatusCode != http.StatusTooManyRequests || attempt == 3 {
                return nil, fmt.Errorf("logs API returned %s: %s", res.Status, data)
            }
            delay := time.Duration(math.Pow(2, float64(attempt))) * 250 * time.Millisecond
            if value, parseErr := strconv.Atoi(res.Header.Get("Retry-After")); parseErr == nil && value > 0 {
                delay = time.Duration(value) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        time.Sleep(time.Duration(math.Pow(2, float64(attempt))) * 250 * time.Millisecond)
    }
    return nil, fmt.Errorf("request exhausted")
}

func (c *Client) Ingest(ctx context.Context, validatedJSON []byte) ([]byte, error) {
    return c.do(ctx, http.MethodPost, "/logs/ingest", validatedJSON)
}

func (c *Client) Search(ctx context.Context) ([]byte, error) {
    return c.do(ctx, http.MethodGet, "/logs/search", nil)
}
Enter fullscreen mode Exit fullscreen mode

This client deliberately does not pretend to know undeclared filters. In a real worker, ingestion retries also need an idempotency design at the event level so a timeout does not create a second copy of the same application event. Search retries are safer when they are read-only, but they still need a timeout and a useful error body for incident tooling.

How do the real alternatives differ?

The comparison should follow the job to solve. Loki is a log aggregation system built around labels and a query language, so it becomes attractive when the team already runs Grafana and wants a familiar operational workflow. Elasticsearch provides a general search and analytics engine with a much wider indexing and query toolbox, at the cost of more decisions around mappings, cluster operation, and retention. OpenSearch occupies a similar self-managed or managed search space and brings its own dashboards and administration model.

Those products are not interchangeable in operational burden. A managed Loki, Elasticsearch, or OpenSearch service can reduce the work of running storage, but it still introduces a vendor-specific integration and often a separate account, schema, and alerting workflow. Self-hosting gives more control over placement and retention while putting capacity, upgrades, replicas, and on-call ownership on the platform team. The right answer depends on incident volume and the SLO, not on the length of a feature checklist.

Option Strong fit Important boundary for this scenario
Two-route REST log surface Early-stage app logging with small workloads and one backend contract Basic querying; no advanced tracing, export, alert delivery, or user-level deletion route
Grafana Loki Teams already invested in Grafana and label-oriented log operations Adds the Loki/Grafana operating and integration model
Elasticsearch Rich text search, indexing, and broader log analytics More schema, cluster, and retention decisions to own or buy
OpenSearch Search-centric observability with an open-source-oriented stack Its dashboards, administration, and deployment model are another platform surface

For this logistics case, I would pick the two-route surface when the primary incident question is “what did the application report during this run?” and the expected scale remains small. I would move toward Loki when Grafana is already the center of the on-call workflow, and toward Elasticsearch or OpenSearch when investigators need richer search and analytics than basic filtering. None of those choices supplies a missing heartbeat automatically.

What this approach cannot be the answer for

There is no alert or notification route here. A polling process can query logs and then call an existing notification system, but the polling interval, duplicate suppression, and failure handling become your responsibility. That is a reasonable bridge for a small app; it is not an alerting SLO by itself.

It also lacks distributed tracing queries and span trees, source-map reversal, crash symbolication, Electron minidump parsing, and Session Replay. A trace_id field can help correlate a log with a trace held elsewhere. It cannot manufacture that trace.

For compliance workflows, the boundary is harder. There is no per-user log deletion interface, batch export or subscription interface, or visible retention and cold-storage configuration entry. Change-audit logs, evaluation statistics, parent-child flag dependencies, and a deletion recycle bin are absent from the stated capability set. I would not select this surface as the sole system of record for a deletion-rights promise without an external retention and erasure design.

The practical decision rule is short: use it for centralized app events and basic reconstruction, add a Healthchecks-style heartbeat for “the job never ran,” and adopt a dedicated tracing, alerting, or retention system when that requirement becomes part of the SLO. That keeps the integration small without pretending the log store is the whole control plane.

References

OpenTelemetry's logs signal model explains the role of structured logs and their relationship to other telemetry signals. The Loki, Elasticsearch, and OpenSearch documentation describe the operational alternatives rather than serving as feature claims about the two-route surface.

Sources

Top comments (0)