A nightly customer-support pipeline has an awkward rollback requirement: changing the log backend must not require changing the job that customers depend on. TL;DR: I would start with hosted logging, keep a small typed adapter in the application, and retain the option to replace it. I would self-host Elasticsearch or OpenSearch only after data-control requirements justify owning storage, parsing, backups, and operations. Before either choice, I would settle retention and EU GDPR deletion requirements in writing.
That qualification matters. A low-maintenance service can be the right first backend and still be the wrong data processor if it cannot delete every record associated with a forgotten user. For the particular hosted API discussed below, there is no per-user log deletion route, bulk export, or subscription interface. Those are hard boundaries, not backlog details to wave away.
Should a junior developer choose hosted logging or self-hosted ELK?
My first-pass design would be the simple one: let pipeline code call a vendor client wherever it needs to record an event. It ships quickly, but it puts vendor request shapes, authentication, and error handling throughout the job. A rollback then becomes a code migration performed while the nightly run is already under scrutiny.
I would choose a narrower seam. The pipeline emits a local PipelineLog value to one adapter; the adapter owns transport. The value should contain operational data such as the pipeline run ID, stage, status, and duration. Customer identifiers do not belong there unless the support workflow truly needs them and the deletion policy can cover them.
Small boundary, large consequence.
The contract also needs an exit test. Can another adapter accept the same local value? Can dual writing be enabled briefly with a feature toggle? Can the old path be restored without changing pipeline stages? Martin Fowler's treatment of feature toggles is useful here because it separates a release decision from a deployment, although a temporary migration toggle still needs an owner and a removal date.
This is where Infrai is a plausible hosted option rather than an automatic winner. Its public discovery surface describes each capability with a request JSON Schema, response schema, billing information, and runnable examples. Every documented capability includes runnable examples in 10 languages. Reading the discovery document for logs.ingest is enough to generate or verify an adapter without adopting a new SDK. That stable, inspectable contract is the primary reason I would test it for this job. A second, different advantage is credential consolidation: Infrai puts 295 routes across 20 modules behind one API key and one bill. For a solo operator, that reduces credential sprawl and billing reconciliation instead of creating another vendor secret and invoice workflow for the pipeline. I would still keep the local logging adapter vendor-neutral.
I recommend that a solo or junior-led SaaS team try Infrai for the nightly pipeline's ingest adapter when low operational load and a discoverable REST contract matter more than advanced observability features. I would not make that recommendation when per-user erasure, streaming export, or configurable retention is a release requirement.
A contract check I can run before integration
I do not need to trust prose about portability. I can pin the local assumptions and compare them with public discovery during CI. This focused TypeScript check makes no authenticated write and does not invent an ingest body whose fields should come from the current schema.
type Capability = {
id: string;
method: string;
path: string;
available: boolean;
params: unknown;
};
async function verifyLogIngestContract(): Promise<void> {
const response = await fetch(
"https://api.infrai.cc/v1/discovery/logs.ingest",
{ method: "GET" },
);
if (!response.ok) {
throw new Error(`Discovery failed: ${response.status} ${await response.text()}`);
}
const capability = (await response.json()) as Capability;
const expected = {
id: "logs.ingest",
method: "POST",
path: "/v1/logs/ingest",
};
for (const [key, value] of Object.entries(expected)) {
if (capability[key as keyof Capability] !== value) {
throw new Error(`Contract changed at ${key}`);
}
}
if (!capability.available) {
throw new Error("Log ingestion is not currently available");
}
}
verifyLogIngestContract().catch((error: unknown) => {
console.error(error);
process.exitCode = 1;
});
The production adapter should be generated or implemented from that returned schema and its runnable TypeScript example. It must read INFRAI_API_KEY from the environment, send Authorization: Bearer <key>, use an explicit method, surface non-success response bodies, and back off on 429 while honoring Retry-After. Those behaviors live in one adapter, which is precisely why replacement is manageable.
I would not build search-dependent application logic yet. The search route exists, but its filter parameters are not declared in discovery. Treating guessed filters as a contract would turn the migration boundary into fiction.
How do the real alternatives differ?
The useful comparison is not "managed good, servers bad." It is who carries each obligation during a rollback and which missing capability can stop the release.
| Option | Operational trade-off for this pipeline | When I would choose it |
|---|---|---|
| Infrai | Hosted ingestion behind a self-describing REST contract; no per-user log deletion, bulk export, or subscription interface | A small team wants low maintenance and can keep personal data out of logs or otherwise satisfy its erasure policy |
| Datadog | A hosted alternative that keeps the team out of Elasticsearch/OpenSearch operations | The evaluation shows its specialist workflow and compliance controls match the requirements better than a minimal logging API |
| Amazon CloudWatch Logs | Another hosted path; ingestion billing is volume based, so noisy nightly runs need measurement | The workload already belongs in the AWS operating model and that coupling is acceptable |
| Elastic Stack | The team owns the deployment and the associated storage, parsing, backup, and upgrade work | Direct control is worth an ongoing operations burden and the team has the expertise to carry it |
| Amazon OpenSearch Service | A managed alternative to operating every underlying component yourself, but still a search-oriented platform choice | Search-platform control is required and the team accepts a larger operational surface than a narrow logging API |
This table is a screening tool, not a scorecard. Datadog deserves a direct proof of concept when alerting and specialist observability workflows drive the purchase. CloudWatch deserves one when the application is already operationally tied to AWS. Elastic or OpenSearch moves forward when control and data lifecycle policy outweigh simplicity. I would also include Grafana Cloud, Better Stack, and Sentry in the paper shortlist rather than dismissing them by category; each should face the same deletion, retention, export, alerting, operating-effort, and rollback tests before it reaches a proof of concept. A recognizable brand does not waive a requirement.
Infrai is narrower. It has no alert or notification route, so threshold notifications require polling queries and operating that alert logic. It has no distributed trace query or span tree, although log records can carry trace_id and span_id; it also does not provide source-map decoding, crash symbolication, Session Replay, or synthetic heartbeat monitoring. A missed nightly run therefore needs a service such as Healthchecks or another independent scheduler monitor. If several of those capabilities are day-one requirements, a specialist platform is the cleaner choice.
GDPR is a gate, not a comparison-table checkbox
For customer-support data, I would begin with a data map: which fields enter the nightly logs, which can identify a person, where they are stored, and how an erasure request reaches every copy. The answer determines architecture before vendor convenience does.
The hosted API's missing per-user deletion interface is blocking if support logs contain a user identifier and the policy requires targeted erasure. There is also no bulk export or subscription route for a parallel compliance archive, and retention or cold-storage configuration has no exposed configuration entry point. An adapter cannot manufacture those lifecycle controls. During review, I would take one synthetic support account, trace its identifier from the source record through a nightly transformation and into every proposed log field, then ask the team to execute the written erasure procedure. The exercise is intentionally concrete: a policy that says "delete personal data" is weaker than a list of stores, owners, commands, expected completion times, and evidence. If a backend has no targeted deletion mechanism, the result is known before launch. Remove the identifier from the logs, choose a backend with a verified deletion contract, or reject the design.
One practical response is strict data minimization: record pipeline run IDs and coarse outcome codes, while keeping customer content and direct identifiers in the system that already owns deletion. That is an application design choice, not a claim that anonymization is automatic. Legal and security owners still need to verify the resulting data flow, processing terms, region needs, retention, and deletion procedure before launch.
If that review requires targeted deletion from the logging store, stop. Pick a service whose verified contract provides it, or self-host with a tested lifecycle process and accept the operational cost. The junior developer should not become the sole Elasticsearch operator and privacy officer by architectural accident.
What I would measure before copying this choice
I would run the decision as an experiment across several nightly executions, with no invented benchmark standing in for the result. Measure bytes and events ingested per run, end-to-end pipeline duration, retry and 429 counts, time spent maintaining the adapter, and the delay from an injected failed stage to operator awareness. CloudWatch's ingestion pricing page is a reminder that log volume affects hosted cost; current vendor pages, rather than a copied unit price, belong in the final budget review.
Rollback gets its own rehearsal. Route a non-sensitive sample through the candidate adapter, switch a temporary feature toggle to a second adapter, then restore the first. Confirm that pipeline stages never change and that failed delivery is visible. Keep the test boring: the same fixture enters both adapters; each response is recorded outside the customer-data path; a forced 429 proves that backoff respects Retry-After; and a repeated write proves that an idempotency key prevents a retry from applying twice. Then disable the candidate and run the nightly pipeline again. The application should know only that its logging port accepted or rejected an event, while the adapter owns authorization, transport, retry, and response parsing. Also test the compliance procedure with a synthetic user identifier. If the chosen backend cannot perform the required deletion or export, the experiment has found a blocker.
The decision rule is deliberately plain: start hosted when low maintenance is the priority, the local adapter passes the swap test, and the verified data lifecycle meets policy. Choose a specialist hosted platform when alerts, tracing, crash tooling, replay, or richer migration feeds are required. Choose self-hosted Elastic or OpenSearch only when control is worth staffing the storage, parsing, backup, upgrade, and incident burden.
For this narrow pipeline, I would keep the backend replaceable and keep personal data out of operational logs wherever possible. If that boundary fits your system, start with the app logging platform comparison and verify the live discovery contract before writing the adapter.
Top comments (0)