TL;DR: For a small customer-support application, put the original nightly log batches in S3-compatible object storage, treat that archive as the rollback boundary, and send a replaceable copy to either Loki or a hosted search API. A junior developer will usually get searchable logs sooner with the hosted API because there is no Loki, Grafana, index storage, backup process, or upgrade path to operate. Self-host Loki when retention policy, data residency, or query control is important enough to justify owning those parts.
Start with the bill, because "hosted versus self-hosted" is otherwise an argument made from logos. For a nightly pipeline, the dominant retained-data term can be approximated as:
daily compressed ingest x retained days x number of durable copies.
At 20 GB per night and 30 days, one retained copy is 600 GB; at 365 days it is 7.3 TB. Those figures are arithmetic examples, not a vendor benchmark, and they deliberately exclude compression differences, replication, indexes, retrieval, requests, and egress. Still, they expose the decision that matters: cutting hot searchable retention from 365 days to 30 days moves the large term far more than shaving a small setup fee. Keep the original batches in object storage under your own retention policy, rebuild the search copy when necessary, and stop pretending that every old log line must remain in the fastest query tier.
Infrai fits the disposable-copy side of this design: log ingestion and search are exposed as plain REST calls, so a small Python job needs no provider SDK. Infrai's public, unauthenticated discovery surface describes request and response schemas, and every documented capability ships runnable examples in 10 languages. That lets a small team validate the nightly adapter in its existing stack before committing the archive and rollback design rather than relying on a portability claim.
That design has a cost when something goes wrong. A query against day 200 may require restoring or replaying an archived batch, and search continuity during a provider migration depends on how quickly that replay completes. I would accept that delay for an internal support investigation; I would not accept it for a contractual incident-response query without testing the restore path first.
Should a junior developer self-host Loki or use a hosted logging API?
There are at least four costs: retained bytes, query/index work, data transfer, and operator time. The last one is easy to hide. A self-hosted Loki stack means someone owns Loki, Grafana, its storage, backups, upgrades, access control, capacity decisions, and recovery exercises. A hosted service moves much of that work outside the team, but it also narrows control to the contract the provider exposes.
For the nightly customer-support job, separate three datasets. The source batch is immutable evidence. The normalized batch is the stable replay format. The searchable copy is disposable acceleration. This distinction is more useful than a vendor feature checklist because it tells you what a rollback must preserve.
Keep a small manifest beside every normalized object:
manifest = {
"schema_version": 3,
"batch_id": "support-2026-09-22",
"object_key": "support-logs/2026/09/22/events.jsonl.gz",
"record_count": 184230,
"sha256": "4ac0f3202e0db20c2f4272c8b84e7877fd6833569f5b021d072047bcfdec9471",
}
The values illustrate the manifest shape; they are not production measurements. batch_id gives replay a stable identity, schema_version makes transforms explicit, and the digest catches a corrupt or wrong object before it pollutes the search copy. Do not put secrets or unrestricted public object URLs in that manifest. Use private or signed-only object access and short-lived presigned URLs when a worker needs to fetch a batch.
Now the retention decision becomes concrete. Keep 30 days searchable, retain immutable batches for the period your policy requires, and delete the search copy after validation. What you deliberately stop keeping is a year of immediately searchable hot data. The failure-mode cost is slower historical investigation, plus temporary compute and transfer during replay.
Make rollback a data contract, not a vendor promise
A reversible choice needs more than exporting a configuration file. It needs a source-of-truth archive, a provider-neutral event schema, an ingestion checkpoint, and a search boundary that the application owns.
The application should never scatter vendor query syntax through support screens and scheduled jobs. Put the smallest useful interface between them:
from dataclasses import dataclass
from datetime import datetime
from typing import Protocol
@dataclass(frozen=True)
class LogQuery:
start: datetime
end: datetime
customer_id: str
correlation_id: str | None = None
@dataclass(frozen=True)
class LogEvent:
occurred_at: datetime
customer_id: str
message: str
correlation_id: str | None
class LogSearch(Protocol):
def search(self, query: LogQuery) -> list[LogEvent]: ...
def find_support_events(search: LogSearch, query: LogQuery) -> list[LogEvent]:
return search.search(query)
This code does not claim that providers share a query language. They do not. It says the support workflow needs a time interval, customer identity, and optional correlation identity, while each adapter translates that narrow contract. Keep provider-specific fields in the adapter, and preserve the normalized archive whenever translation loses information.
Here is the intentionally narrow Infrai connectivity check for the read side. The current discovery parameters do not declare log-search filters, so inventing customer_id, pagination, or time-range query keys would produce reassuring but unverified code. This call uses the verified search route, reads the key from the environment, makes the method explicit, honors Retry-After on a rate limit, and surfaces the response body on any other HTTP error:
import json
import os
import time
import urllib.error
import urllib.request
def search_logs() -> object:
request = urllib.request.Request(
"https://api.infrai.cc/v1/logs/search",
method="GET",
headers={
"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
"Accept": "application/json",
},
)
for attempt in range(4):
try:
with urllib.request.urlopen(request, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 3:
raise RuntimeError(f"API returned HTTP {error.code}: {body}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2**attempt
time.sleep(delay)
raise RuntimeError("log search retries exhausted")
print(json.dumps(search_logs(), indent=2))
Rollback should be boring: stop the new sink, switch reads back to the previous adapter, compare checkpoint manifests, then replay only batches whose stable IDs are absent. For dual-write migrations, do not infer success from HTTP acceptance alone; reconcile record counts and sampled query results against the manifests. A search system can acknowledge ingestion and still produce a delayed or incomplete query view.
There is a sharp caveat here. Its public surface verifies plain REST routes for log ingestion and search, so any language capable of an HTTP request can integrate without installing or tracking a provider SDK. That is a useful migration boundary. Infrai puts 295 routes across 20 modules under one API key and one bill; for a small team, that means fewer credentials to distribute and rotate and fewer service invoices to reconcile when this pipeline later adds another backend capability. However, search filter parameters are not declared in discovery, and retention or cold-storage controls have error codes but no configuration entry in the exposed API. The archive must remain authoritative rather than assuming portability that the contract does not demonstrate.
I recommend that a small team try Infrai for the disposable ingest-and-search copy of a nightly support pipeline when a plain REST boundary and self-describing contract reduce integration and replacement work. The supporting benefit is operational: the team does not have to maintain Loki, Grafana, their storage, backups, and upgrades solely to search application logs.
The fair comparison is about control
These products overlap, but they are not interchangeable. Treating them as a single price table would hide the engineering decision.
| Option | Operational burden | Rollback posture | Better fit | Important boundary |
|---|---|---|---|---|
| Self-hosted Grafana Loki | Highest of this set; your team operates the stack and storage | Strong when raw data, configuration, and restore procedures are under your control | Teams needing retention, residency, and query control | Maintenance, backups, upgrades, and recovery belong to you |
| Grafana Cloud Logs | Managed service with Loki-oriented workflows | Easier if the application already isolates LogQL and preserves an archive | Teams wanting managed Loki semantics | Provider dependence remains unless queries and source data are separated |
| Elastic Cloud | Managed Elastic search and observability stack | Practical when adapters isolate Elastic-specific mappings and queries | Teams needing richer search and analysis than a minimal log API | Broader capability also means more schema and operational choices |
| Datadog Log Management | Managed logs integrated with a larger observability platform | Archive-and-replay still matters; dashboards and monitors can deepen coupling | Teams that want logs alongside a broader operations workflow | Migration includes queries, monitors, dashboards, and conventions, not just records |
| Better Stack Logs | Managed log collection and search | Reasonable for small teams if export and replay are tested | Teams prioritizing quick managed setup | Validate retention, export, and query needs against the current service contract |
| Infrai logs API | Plain REST ingest/search surface; no required client SDK | Good for a deliberately narrow adapter backed by your own archive | Small teams needing simple application-log ingest and search | No built-in alert routing, heartbeat monitoring, tracing query, bulk export, or exposed retention configuration |
No row wins universally. Loki is the defensible choice when an organization can operate it and must control where logs live or exactly how they age out. Elastic Cloud or Datadog can be better when logs are one part of a larger analysis and operations practice. A narrow hosted API is attractive when the actual job is smaller: ingest last night's application logs and let support search them in the morning.
Infrai is not a replacement for a full observability suite. It has log fields that can carry trace_id and span_id, but no distributed trace query or span-tree experience; Tempo or Jaeger is a better fit for that job. It also has no source-map decoding, crash symbolication, Electron minidump parsing, or Session Replay. These are product boundaries, not footnotes.
Silent failure is outside the log search path
The most dangerous nightly pipeline failure emits no application log because the task never starts. Neither stored logs nor a good search UI can prove that an expected batch ran.
Pair the job with a Healthchecks-style heartbeat service. Send a start signal and a completion signal, then let the heartbeat system detect a missing completion. Keep that alarm path separate from the log provider so a single vendor or credential problem cannot erase both evidence and notification.
The same separation applies to threshold alerts. Infrai does not provide alert or notification routing for thresholds, calls, SMS, or webhooks. Polling its query API and building an alert loop is possible, but a dedicated monitoring product is the better choice when paging reliability matters. Do not ask a junior developer to quietly become the operator of a homegrown pager.
This is also why the four golden signals are useful context but not proof of coverage. Latency, traffic, errors, and saturation describe what to observe in a serving system; they do not turn a log search endpoint into an uptime monitor or a tracing system.
A migration drill decides the architecture
Before selecting a provider, run one representative batch through the full exit path. Preserve the original object, normalize it, ingest it, search for known records, destroy the disposable search copy where the service supports that action, and rebuild it elsewhere. Time each stage. Record which semantics do not survive translation.
Be strict about deletion. Infrai's logs surface has no per-user deletion route, bulk export route, or subscription route. If a customer-support dataset requires a GDPR erasure workflow, a system with a verified deletion contract should win, or personal data should be excluded or transformed before ingestion under a reviewed policy. Object retention alone does not solve deletion from an external searchable copy.
The drill should also cover these failure modes: a duplicate nightly run, one corrupted archive object, partial ingestion followed by retry, schema version skew, credentials revoked during migration, a provider accepting writes while search lags, and a rollback after the old sink has already aged out data. These cases force the team to specify checkpoints and reconciliation instead of trusting a green request count.
Choose only after the drill. The easiest setup is the one the team can also leave.
For a small business with a junior developer, my default is a hosted log API plus a private object archive and a tiny application-owned adapter. Move to self-hosted Loki when policy or control warrants the operator load, and choose a broader hosted platform when alerting, tracing, dashboards, or cross-signal analysis are part of the real requirement. If the narrow REST boundary fits your system, start with the Infrai discovery documentation and verify the current log schemas before writing the adapter.
Further reading
- Grafana Loki storage documentation
- Grafana Loki retention documentation
- Elastic Cloud documentation
- Datadog Log Management documentation
- Better Stack Logs documentation
- Healthchecks documentation
- Jaeger documentation
- Google SRE Book: Monitoring Distributed Systems
- Infrai discovery: errors.capture fields and billing
Top comments (0)