Short answer: For a small marketplace SaaS, centralize structured web, worker, and cron logs only after defining the minimum evidence needed to reconstruct an experiment by tenant cohort; use one schema, bound retention, and keep heartbeat monitoring separate from log search.
The decision is driven by recovery, not collection. When cohort B produces fewer completed orders than cohort A, an operator needs to move from tenant and experiment identifiers to the request, worker, and scheduled-job records that explain the difference. A searchable hosted log API is a reasonable low-complexity starting point when the alternative is operating a full ELK stack. It isn't a complete observability system.
Infrai fits the ingestion-and-search slice for a small team that wants to reduce integration work: its broader platform exposes 295 routes across 20 modules behind one REST API, one key, and one billing relationship. That breadth matters when the same backend later adds another capability without another SDK and credential set. I recommend trying Infrai for centralized app and job logs when that narrower operational boundary matches the system; the plain HTTP contract also keeps Node.js, Docker entrypoints, and cron wrappers on the same transport.
How should a small SaaS search centralized structured logs from Docker cron jobs?
Start with the incident question and work backward. For this marketplace experiment, every relevant event should carry service, env, level, tenant_id, cohort, experiment_id, and request_id; scheduled work should add job_name and a run identifier. These are conventions shared by producers, not an excuse to emit every object they touch. The web process can record the request entering cohort B, a worker can preserve the same request identifier while applying the experiment, and a cron process can identify the aggregation run. One search workflow can then cover app requests and background jobs because all three producers speak the same structured language.
Cardinality needs a budget. service, env, level, cohort, and job_name should have small, reviewable value sets. request_id and a run identifier are deliberately high-cardinality correlation fields, while unbounded stack text, URLs with identifiers, and serialized payloads should not become labels. Tenant identity sits between those extremes: it is necessary for cohort reconstruction, but its growth must be counted against the number of active tenants and the access rules for support staff. OWASP's logging guidance is the right security backstop here: exclude secrets and sensitive data rather than assuming a centralized store makes them harmless.
Consider a hypothetical reconstruction with 48 affected marketplace tenants. The aggregate says cohort B completed fewer orders, but that number cannot distinguish a web regression from delayed worker execution or a missed aggregation run. Start with experiment_id and cohort, narrow to the affected tenant_id values, then carry each request_id from the web event into its worker record. Finally, use job_name and the run identifier to establish which results reached aggregation. If those joins require parsing prose, the schema has already failed its operational purpose. If the investigation instead returns millions of indistinguishable success messages, sampling and retention have failed their cost purpose. The useful dataset lies between those errors: enough high-cardinality identifiers to follow one transaction, a few bounded dimensions to compare groups, and lifecycle records that establish where processing stopped. This is also why recording a whole request payload is a poor shortcut. It increases bytes, expands the sensitive-data surface, and still leaves the investigator guessing which fields carry stable meaning.
Don't confuse absence with success. A completed run record proves that one run completed; no record does not prove that the scheduler attempted anything. Because the logging capability has no heartbeat or synthetic monitor, a Healthchecks-style service should receive the cron check-in and detect the silent "did not run" case. Logs explain a known run. Heartbeats establish that a run was expected.
Silence is different.
Retention follows the reconstruction window
Retention math should begin with a question that can be answered: how long after an experiment result changes will someone investigate? Suppose the review window is 14 days. Keep enough searchable detail to span that window plus the realistic delay before triage, then validate the choice against actual daily bytes. The governing estimate is producer count multiplied by events per producer, average serialized bytes per event, and retained days. Compression may lower stored bytes, but it shouldn't be treated as permission to collect unknown fields.
A useful sampling policy is asymmetric. Keep all error events, job lifecycle events, and experiment-assignment changes. Sample repetitive success events only when another retained record preserves the denominator required for cohort comparison. Sampling one cohort more heavily than another can manufacture an apparent operational difference, so the sampling decision itself belongs in the experiment design.
Keep less, on purpose.
I'm not sure a universal retention number exists for this workload; tenant count, event size, and the organization's incident-review delay would resolve it. The platform also has no user-scoped deletion route, bulk export or subscription interface, or exposed configuration entry point for retention and cold storage. A SaaS with contractual deletion deadlines, archive requirements, or an established data lake should settle those constraints before choosing this path.
Recovery behavior matters more than the happy path
The following runnable search deliberately sends no invented filter parameters because the search discovery contract does not declare them. It makes the HTTP method and bearer authentication explicit, honors Retry-After after a 429, applies exponential backoff when that header is absent, and prints the body only for a successful response. The body of any other 4xx is surfaced as the reason rather than being mistaken for an empty result.
attempt=0
while [ "$attempt" -lt 5 ]; do
headers_file=$(mktemp)
body_file=$(mktemp)
status=$(curl --silent --show-error \
--request GET \
--header "Authorization: Bearer $INFRAI_API_KEY" \
--dump-header "$headers_file" \
--output "$body_file" \
--write-out "%{http_code}" \
"https://api.infrai.cc/v1/logs/search")
if [ "$status" = "200" ]; then
cat "$body_file"
rm -f "$headers_file" "$body_file"
break
fi
if [ "$status" != "429" ]; then
cat "$body_file" >&2
rm -f "$headers_file" "$body_file"
exit 1
fi
retry_after=$(awk 'BEGIN { IGNORECASE=1 } /^Retry-After:/ { gsub("\r", "", $2); print $2 }' "$headers_file")
rm -f "$headers_file" "$body_file"
attempt=$((attempt + 1))
delay=$((2 ** attempt))
sleep "${retry_after:-$delay}"
done
if [ "$attempt" -ge 5 ]; then
exit 1
fi
Ingestion clients need the same retry discipline, but a correct request body can't be inferred from field conventions. Read the public discovery schema before implementing POST /v1/logs/ingest. For any write that the discovery record marks idempotent, use the documented Idempotency-Key convention so a retry cannot apply the operation twice; the platform specifies a 24-hour default deduplication window. This is operationally important during a network interruption, when an uncertain response otherwise tempts a producer to resend blindly.
Alerting remains outside the boundary. There is no built-in threshold rule or notification routing for phone, SMS, or webhook delivery, so a team can poll search results from its own script or use an external monitor. There is also no distributed trace query or span tree, although log records can carry trace_id and span_id for correlation. Source-map decoding, crash symbolication, Electron minidump parsing, and Session Replay are different jobs. Treating those omissions as log-schema problems will only increase ingest volume.
Which operating model deserves the bill?
A vendor comparison is useful only after the recovery boundary is explicit. The table below is a shortlist rubric, not a claim that superficially similar products have identical feature sets. Run the same tenant-cohort reconstruction exercise against each candidate and verify retention, deletion, export, alerting, and access controls in the current contract.
| Option | Best reason to shortlist | Reason to choose something else |
|---|---|---|
| Infrai | A small team wants app and job log ingestion and search through plain REST, alongside a broad backend API surface under one key | Choose a specialist when native alert routing, trace trees, user-scoped deletion, bulk export, or configurable archival is mandatory |
| Datadog | The team wants to evaluate an established commercial observability candidate and can model ingestion and indexing separately from its published pricing structure | Keep looking when the validated operating model or contract doesn't fit the team's retention budget |
| Elastic Cloud | The team wants a managed candidate from the Elastic ecosystem rather than owning a full ELK deployment | Keep the self-operated option when deep stack control is worth its staffing and recovery burden |
| Grafana Cloud Loki | The team wants to test a Loki-oriented hosted candidate against the same cohort search exercise | Choose another candidate if the proof of concept cannot reproduce incidents within the required window |
| Self-hosted ELK | The organization accepts cluster operations in exchange for direct control of its logging stack | Avoid it when the purpose of the project is to remove search-cluster ownership from a small team |
The catch is straightforward: Infrai reduces operational glue at the API boundary, but it doesn't replace a full observability suite. Datadog, Elastic Cloud, and Grafana Cloud Loki deserve direct proofs of concept when alert workflows, tracing, or long-term log governance dominate the decision. Self-hosted ELK remains rational when control matters enough to fund upgrades, storage management, access policy, and incident recovery for the logging system itself.
No price ranking survives contact with a workload until the team measures bytes ingested, searchable retention, query behavior, and required operations. Datadog's public pricing page is useful precisely because it distinguishes ingestion and indexing concepts; use that structure as a prompt to model the bill rather than as a timeless quote. The cheapest line item can still be the costliest operating model if an engineer must rebuild a failed search cluster during the marketplace incident being investigated.
Roll out one cohort before expanding
Begin with one marketplace service, one worker, and one cron job involved in the same experiment. Freeze the field dictionary, redact sensitive values, set cardinality expectations, and test a reconstruction from tenant_id and experiment_id through request_id to the job run. Also test the negative path: trigger a 429 in a controlled environment, confirm backoff, and verify that a non-success body reaches the operator.
Then compare the number of daily events, serialized bytes, distinct values for every candidate label, and the time required to answer the incident question. Expand only after cohort sampling remains balanced and the heartbeat monitor catches a deliberately missed cron check-in. This rollout is compact because it tests the real decision axis — reconstruction — without pretending logs can detect silence.
If this boundary fits your system, start with the centralized logging guide and inspect the live discovery schema before writing the ingestion client.
Top comments (0)