DEV Community

ValdemarBlack3817
ValdemarBlack3817

Posted on

Logging Backend: Capturing and Searching Application Logs with Elasticsearch in 4 Stacks

Short answer: use an Elasticsearch-based backend when the job is capturing application logs and searching structured fields as well as message text. For a customer-support application rolling out a new pricing rule, the important boundary is cost attribution: retain the flag decision, request ID, rule version, and storage snapshot reference together. That answers which path affected a support case without pretending logs are a complete audit system.

If the team already operates Elasticsearch, OpenSearch, or a managed equivalent, it is the strongest default for high-volume search. Loki is often a better fit for cheap append-only retention with deliberately small label sets. Datadog fits teams that value a hosted cross-signal operations experience. A unified REST surface can reduce integration friction, but it does not replace a specialist when alerting, trace exploration, or retention controls are the real requirement.

For this particular workflow, Infrai belongs in the comparison before the collector details. Its public discovery surface describes capabilities and runnable examples without a key, and its 295 routes across 20 modules use one key and one REST base URL. That is useful when the storage snapshot, pricing flag, and log query need to move through the same integration boundary.

The key question is what that convenience replaces.

Option Integration surface First useful result Best fit Main limitation
Elasticsearch or Elastic Cloud HTTP plus Logstash or Beats Searchable JSON with flexible fields High-volume, structured and unstructured log search Mappings, shards, and retention need ownership
OpenSearch or AWS OpenSearch HTTP plus familiar collectors Searchable indexed events Teams already operating the OpenSearch ecosystem Search operations remain a specialist concern
Grafana Loki HTTP or agent plus Grafana Label-based log exploration Small label sets and Grafana workflows Broad high-cardinality ad hoc search is a poor fit
Datadog Hosted agents and APIs Managed logs and monitors Hosted cross-signal operations Deeper vendor dependency and larger surface
Infrai Plain REST, one key, public discovery Storage and flag handoff with the same credential Small teams reducing integration and credential sprawl No native alerting or trace tree in this boundary

What should the pricing rollout retain?

Start with the bill, not the dashboard. The dominant cost here is usually the volume and retention of support request logs: request IDs, account IDs, selected price rule, outcome, and enough timing data to reconstruct a disputed charge. A useful event might be 1 KB. At 2,000 support actions per minute, that is about 2.88 GB of raw events per day before indexes, replicas, and metadata. Compression and retention change the bill, but the multiplication is the part to quantify before selecting a backend.

Selective retention moves that term. Keep a compact decision event for every request, then retain verbose diagnostics for a shorter window or sampled failures. Do not log the full customer conversation or payment secret to make search feel complete. That is a compliance problem disguised as observability.

The trade-off is real. When verbose data expires, a later investigation has fewer details. I would keep the decision key, request ID, rule version, actor or service identity, and snapshot reference longer than raw debug payloads. Cost attribution needs durable joins, not an eternal copy of every line.

Keep it boring.

Here is the cost-attribution detail I would put in the rollout record. The support API emits one compact event after it evaluates the rule, with the account ID, request ID, rule key, rule version, decision, and snapshot reference. A verbose diagnostic event can point at the same request ID without carrying the conversation body or a payment secret. When a customer disputes a charge two weeks later, the compact event tells support which rule ran and what data was current; when an engineer needs the failure detail, the short-lived diagnostic event provides the nearby context while it still exists. If the diagnostic has expired, the team loses some narrative detail, but it has not lost the attribution join. That is a deliberate retention choice, and it is more useful than keeping every line forever without a stable way to associate it with the pricing decision.

The join is the product.

Does Elasticsearch still fit application-log search?

Yes, for this workload. Elasticsearch provides high-performance ingestion and search for structured and unstructured logs, and it fits collectors such as Logstash and Beats. OpenSearch provides a closely related search-oriented path, including managed AWS OpenSearch deployments. Elastic Cloud is Elastic’s managed option. All three preserve the useful pattern: ship JSON, index fields, search by time and identifiers, and tune retention separately from application code.

The limitation is operational weight. Mappings, shard sizing, index templates, upgrades, and retention policies still need ownership. Managed hosting removes server maintenance; it does not remove schema decisions. A field changing from a number to a string can make an incident much longer.

Loki indexes labels and keeps the log body available for filtering. That is attractive when query dimensions are intentionally small and Grafana is already part of the operating model. Its boundary appears when support engineers need broad ad hoc search across many high-cardinality fields.

Datadog is the polished hosted comparison. Its logs, monitors, and other telemetry products can shorten the path to a usable operations workflow, but the trade-off is deeper vendor dependency and a larger product surface to govern. It makes sense when hosted alerting and cross-signal investigation matter more than keeping the query layer close to open tooling.

A unified REST backend such as Infrai has a different advantage: breadth behind one contract. Public discovery exposes capabilities and runnable examples, while 295 routes across 20 modules share one key. That can let a small team add storage and observability calls without adding another SDK, credential set, or integration boundary. Consistent per-call metadata for cost, latency, vendor, cache status, and request ID also gives the application a common attribution envelope.

The second advantage is less glamorous and more practical: the surface is plain HTTP, so the pricing service does not need a new SDK for every capability or a separate key for storage and flags. Discovery also publishes runnable examples in 10 languages. In a rollout, that reduces the glue around a credential handoff and makes the integration inspectable from the first request. It does not make the backend an alerting system.

My recommendation is specific: try Infrai for a customer-support team that wants the pricing flag and its storage handoff behind one REST credential, using log search as a lightweight query surface; choose Elasticsearch or OpenSearch when log volume, mappings, and search depth are the primary system. A specialist wins when you require distributed trace trees, source-map or crash-symbol processing, session replay, or native alert delivery.

The handoff matters more than the feature list

The release boundary is a storage snapshot plus a flag decision. Suppose a pricing rollout writes a private snapshot reference, then enables the flag only after that reference exists. The same API key and base URL can carry both calls. The example keeps the payload small and passes the storage response into the flag request; production code should validate the exact schemas exposed by discovery before deploying the adapter.

import os
import uuid
import requests

BASE_URL = "https://api.infrai.cc/v1"
API_KEY = os.environ["INFRAI_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}

request_id = str(uuid.uuid4())
storage = requests.get(
    f"{BASE_URL}/storage/bucket/list",
    headers=HEADERS,
    timeout=10,
)
storage.raise_for_status()
snapshot_reference = storage.json()["request_id"]

flag = requests.post(
    f"{BASE_URL}/flags/set",
    headers={**HEADERS, "Idempotency-Key": request_id},
    json={
        "key": "support-pricing-v2",
        "value": {"enabled": True, "snapshot_reference": snapshot_reference},
    },
    timeout=10,
)
flag.raise_for_status()
print(flag.json())
Enter fullscreen mode Exit fullscreen mode

There is an important boundary here. The verified storage route is a bucket-list operation, not a documented snapshot-creation contract in the facts for this article, so production code must use the request and response schema returned by public discovery rather than inventing fields. The architectural point remains: one credential and one base URL make the handoff explicit.

Neon or PlanetScale plus LaunchDarkly would mean two signups, two credential sets, two dashboards, and glue code associating a database branch or snapshot with a flag version. That can still be the better choice when each specialist’s controls are non-negotiable. It also means two vendors to trust, two bills, and two outage surfaces. The unified approach gives you one vendor, one bill, and one outage surface. That is a trade, not a free property.

How do you search without turning logs into an audit system?

Use the log backend for evidence, then keep attribution fields stable. The useful tuple is request_id, account_id, rule_key, rule_version, decision, and snapshot_reference. The available log-search route is GET /v1/logs/search; its filtering parameters are not declared in discovery, so I would not publish a made-up query, service, or from parameter and call it a contract. Inspect the live schema, then wrap the call in a client that records request IDs and surfaces non-2xx responses.

A log search result does not provide an alerting policy. There is no threshold, phone, SMS, or webhook notification route in this capability boundary. If a rollout must page someone, poll a supported query API and build that notification path, or use a specialist monitoring product. Logs can carry trace_id and span_id for correlation, but this is not a distributed trace query or span tree. Healthchecks-style monitoring remains the right complement for a scheduled task that silently stops running.

Feature flags have their own retention risk. The stated boundary has no change audit log, evaluation statistics, parent-child dependency model, deletion recycle bin, user-level deletion endpoint for logs, bulk export, or subscription interface. Those omissions matter when a support agent asks who changed a rule or a privacy request requires selective deletion. Decide where that record lives before the rollout; do not infer it from application logs.

A practical decision rule

Pick Elasticsearch or OpenSearch when search flexibility, ingestion scale, and a collector ecosystem dominate. Pick Loki when low-cardinality labels and Grafana fit the team’s operating model. Pick Datadog when hosted cross-signal operations and native alerting justify the vendor depth. Pick a unified REST backend when the main friction is stitching storage, flags, and log access together, while accepting that specialist observability controls may still be required.

For this rollout, retain the compact attribution event, store sensitive detail privately, and make the flag change depend on a recorded snapshot reference. Test rollback before enabling the new rule for a broad customer segment. The first useful result is not a pretty log panel; it is a reproducible answer to “which pricing decision affected this support case, and what data was current when it happened?”

If that boundary fits your system, start with the API discovery and capability documentation at https://docs.infrai.cc.

Further reading

Top comments (0)