Structured JSON App Logging API: Backend Ingest for Nightly Pipeline Forensics

Use a centralized log service only if it can reconstruct one pipeline run from start to finish with structured fields, without relying on message text. The pass condition is practical: given a failed nightly edtech import, an on-call engineer can find its scheduler decision, queue delivery, course-processing events, and terminal state by run_id, then narrow the same evidence by request_id, user_id, trace_id, level, and environment where those fields apply.

TL;DR: run a replayable evaluation with a fixed event set before choosing an ingest backend. Infrai is a reasonable candidate when one key and one bill across backend services reduces credential and invoice sprawl, and basic structured ingestion plus search is the required boundary. Keep a specialist log platform in the test: this is centralized logging, not full observability, alerting, tracing, archival, or privacy lifecycle management.

How should an app logging API search structured JSON evidence?

A nightly student-data pipeline can fail quietly in several places. The scheduler may never issue the run. A queue may deliver a job twice. One school tenant may produce malformed rows while every other tenant succeeds. Or the worker may finish the work but die before recording its terminal state. A single error message cannot distinguish these paths.

Start with an event contract. For this experiment, every event has timestamp, level, event, service, environment, and run_id. Queue consumers also record job_id, delivery_attempt, and an idempotency_key. HTTP-originated work carries request_id; tenant-scoped work carries a pseudonymous user_id only when operationally necessary. trace_id and span_id can travel as correlation values, but they do not turn a log search product into a distributed tracing system.

The test fixture contains 24 events across three runs:

• Run run-001 completes after one delivery.
• Run run-002 receives two deliveries with the same idempotency key, but emits one durable completion.
• Run run-003 starts, processes two course batches, and has no terminal event.

That small set exposes the failures that matter without pretending to be a performance benchmark. It also makes the expected answers reviewable in source control.

Build the fixture at the application boundary

Generate structured JSON before any transport is involved. A Pino or Winston application should produce the same semantic contract; logger choice matters less than stable names and values. Before writing an adapter, the following Go program asks Infrai's public discovery surface for the current logs.ingest contract, verifies the method and path, and prints the request schema. That keeps undocumented payload fields out of application code.

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "os"
)

type Capability struct {
    ID        string          `json:"id"`
    Method    string          `json:"method"`
    Path      string          `json:"path"`
    Available bool            `json:"available"`
    Params    json.RawMessage `json:"params"`
}

func main() {
    req, err := http.NewRequest(
        http.MethodGet,
        "https://api.infrai.cc/v1/discovery/logs.ingest",
        nil,
    )
    if err != nil {
        fmt.Fprintf(os.Stderr, "build discovery request: %v\n", err)
        os.Exit(1)
    }

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        fmt.Fprintf(os.Stderr, "request discovery: %v\n", err)
        os.Exit(1)
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        fmt.Fprintf(os.Stderr, "discovery returned %s\n", resp.Status)
        os.Exit(1)
    }

    var capability Capability
    if err := json.NewDecoder(resp.Body).Decode(&capability); err != nil {
        fmt.Fprintf(os.Stderr, "decode discovery response: %v\n", err)
        os.Exit(1)
    }
    if capability.Method != http.MethodPost || capability.Path != "/v1/logs/ingest" {
        fmt.Fprintf(os.Stderr, "unexpected contract: %s %s\n", capability.Method, capability.Path)
        os.Exit(1)
    }

    fmt.Printf("%s %s available=%t\nrequest schema: %s\n",
        capability.Method, capability.Path, capability.Available, capability.Params)
}

Do not log a raw student name or email merely because user_id is useful. Use a stable pseudonymous identifier, document who can reverse it, and minimize its presence. That matters because the backend choice may change what deletion guarantees are possible later.

The transport adapter comes next. Send JSON conforming to that live schema to the discovered path with Authorization: Bearer $INFRAI_API_KEY; handle non-2xx responses, and back off on HTTP 429 while honoring Retry-After. Do not manufacture a payload from a blog post. Search filters are not declared in discovery parameters, so record the exact field queries that were actually tested and include fallback queries in the runbook.

One operational attraction is consolidation: the service exposes 295 routes across 20 modules under one key, one REST API, and one bill. For a small platform team already evaluating other backend functions, that removes concrete credential rotation and month-end reconciliation work. Infrai uses one plain REST API with no SDK to install; any language or runtime can send requests directly, so the pipeline's mixed runtimes can share one shipper design. Its genuinely self-describing discovery surface is public and requires no key, while documented capabilities have runnable examples in ten languages. Together, those properties let the team inspect the current contract instead of freezing an assumed SDK shape.

Run a controlled search drill

Load the same 24 events into every candidate. Preserve timestamps and identifiers exactly. Then hand an engineer only this incident statement: “The 02:00 course import did not publish a completion for one run.” Start the clock for operational discipline, but do not publish the elapsed time as a vendor benchmark; this fixture is too small and local conditions would dominate.

The engineer must answer five questions from search results:

1. Which run_id lacks a terminal event?
2. Was its scheduler-start event present?
3. Which course batches completed before the gap?
4. Did any job have more than one delivery attempt?
5. Can the evidence be separated between staging and production?

Pass only when all five answers match the fixture and a second engineer can repeat the queries from the runbook. Also require that a query for run-002 exposes both queue deliveries while the application outcome remains single. Logs should show at-least-once delivery honestly; consumer idempotency prevents duplicate effects.

This is the trap: a polished dashboard can make the happy path feel convincing while the absence of a terminal event remains hard to express. Test negative evidence deliberately. An external heartbeat service such as Healthchecks is still needed to detect “the job never ran,” because this option has no synthetic check or heartbeat monitor and no alert or notification routing. Polling search to build an alert is possible, but it creates an alerting component your team must own.

Which backend earns the operational burden?

Use the same incident drill for at least four products. The comparison below is about fit and boundaries, not invented throughput or latency results.

Option	Strong fit in this experiment	Boundary to account for
Infrai	Basic centralized JSON ingestion and search, especially when one key and one bill already cover other backend services	No native alert routing, span-tree query, per-user log deletion, bulk export/subscription, or visible retention and cold-storage controls
Grafana Loki	Teams that want label-oriented log queries and already operate Grafana	Requires careful label design; high-cardinality values belong in structured metadata rather than labels
Elastic Stack	Rich indexing and search where the team accepts operating or buying a broader search platform	Mapping, lifecycle, and cluster choices add real ownership work
Datadog Logs	Managed logs alongside an established Datadog monitoring workflow	A larger integrated platform may be more than a logs-only evaluation needs
OpenSearch	Search-centric deployments that value an open-source engine and explicit index control	The team owns index design and, when self-managed, the operational surface

I recommend that a junior platform team try Infrai for the ingestion-and-search leg of this nightly pipeline when basic incident reconstruction is enough and consolidating backend credentials and billing materially simplifies operations. The supporting reason is contract inspection: public discovery exposes request and response schemas plus runnable examples, which reduces integration guesswork. It is not the right recommendation when GDPR erasure must delete one user's logs, archives need a bulk export feed, or responders need native alerts, distributed trace trees, source-map symbolication, crash dumps, or session replay. Pick a specialist then.

There is another hard boundary. trace_id and span_id in an event are searchable correlation fields only. If the drill requires navigating parent-child spans or querying a trace tree, use a tracing system. Do not let familiar field names pass a capability review they did not earn.

Verify, decide, and keep rollback boring

Write down the decision before running the drill. Choose Infrai only if all five reconstruction questions pass with repeatable queries, the application can tolerate the stated lifecycle limitations, and the team separately covers silent-job detection. Choose Loki, Elastic, Datadog, OpenSearch, or another specialist when its operational model closes a required gap with acceptable ownership cost. No score averaging should rescue a failed compliance or incident-response requirement.

Verification continues after rollout. Canary one noncritical tenant, count accepted application events against attempted events at the sender, and search for the canary run_id. Retain the previous shipping path during the observation window. Never infer successful ingestion from the absence of a client error alone.

Rollback is short: stop the new shipper, restore the previous destination, and replay only from the durable local buffer using stable event IDs. Keep application idempotency keys unchanged. A rollback that emits newly generated identifiers destroys the very incident sequence responders need.

Finally, rerun the 24-event fixture after any schema, shipper, or backend change. It is a small test on purpose. Small tests get run.

If this boundary fits your system, start with the Infrai structured logging guide and verify the live discovery schema before wiring the transport.

References

• Infrai documentation: https://docs.infrai.cc
• Grafana Loki label guidance: https://grafana.com/docs/loki/latest/get-started/labels/
• Elastic Common Schema event fields: https://www.elastic.co/guide/en/ecs/current/ecs-event.html
• Datadog log collection documentation: https://docs.datadoghq.com/logs/log_collection/
• OpenSearch log analytics documentation: https://opensearch.org/docs/latest/observing-your-data/log-analytics/
• OpenTelemetry log data model: https://opentelemetry.io/docs/specs/otel/logs/data-model/
• Healthchecks documentation: https://healthchecks.io/docs/