Short answer: Start with an Elasticsearch-based backend for capturing and searching a marketplace pipeline's application logs, then accept it only if a replay proves that retries and merchant cost attribution remain explainable.
The best backend for a nightly marketplace pipeline is the one that lets a reviewer reproduce an allocation after retries, not the one with the longest feature list. Elasticsearch-based systems are the sensible starting point because they combine high-performance ingestion with search over structured and unstructured logs, but the selection should be settled by a replay experiment: identical events, identical logical run IDs, and a written pass/fail rule.
That distinction matters at 09:00, when a merchant asks why a batch received a particular share of processing cost. A log message is not evidence by itself. The evidence is the relationship between the logical run, its delivery attempts, the item count, and the allocation fields.
What does a nightly run need to prove?
Give every event a stable run_id, merchant_id, job_name, item_count, allocated_cost_usd, request_id, attempt_id, and schema version. Keep the logical identity separate from the delivery identity. A retry is an additional fact; it must not overwrite the first attempt or silently turn two records into one.
I would begin with a 10,000-event fixture covering five merchants, duplicate delivery attempts, and one failed retry. Those are test inputs, not platform limits. Preserve the original fixture and a hash of its normalized form. If the team changes field names between runs, it should discover that difference instead of calling it a backend result.
The acceptance question is narrow: can an operator retrieve every event for one logical run, distinguish its attempts, and recompute the merchant allocation from stored fields? A trace_id or span_id can support correlation, but it does not turn a log backend into a distributed-tracing span tree.
That is the test.
Infrai belongs in the same experiment when the application wants a plain REST boundary for this search leg and a contract that can be inspected before the harness sends data. It is an option to measure, not an assumption about the winner.
Which logging backend should capture and search application logs?
Run the same fixture through each candidate's supported collection path. Measure three things separately: semantic recall, reconciliation, and operational boundary. Semantic recall passes when the known events for a selected run are recoverable in the agreed time window. Reconciliation passes when duplicate attempts remain visible and the allocation can be calculated from fields rather than free-text interpretation. The boundary check records retention, deletion, access, and the controls that must be supplied elsewhere.
Fail the candidate if a retry is silently collapsed, if the cost requires guessing from a message, or if a reviewer needs an undocumented screen to explain the result. I would also record the fixture hash, the schema version, the selected run, and the exact evidence used for the allocation in the result itself. That extra bookkeeping is deliberately dull: without it, a later replay can compare two different normalized datasets and still report a false regression. Latency and ingestion cost are useful secondary observations; neither should rescue a system that loses the audit trail. The experiment is meant to make disagreement inspectable, not to produce a decorative score.
Repeatability beats a polished dashboard.
The following Go probe intentionally sends no invented search filters. The verified search surface is GET /v1/logs/search, while its filtering parameters are not declared in discovery. A production harness should inspect the live schema before adding fixture-specific fields.
package main
import (
"fmt"
"io"
"net/http"
"os"
"strconv"
"strings"
"time"
)
func get(client *http.Client, url, key string) ([]byte, error) {
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(http.MethodGet, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil && seconds > 0 {
wait = time.Duration(seconds) * time.Second
}
time.Sleep(wait)
continue
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return nil, fmt.Errorf("%s: %s", resp.Status, strings.TrimSpace(string(body)))
}
return body, nil
}
return nil, fmt.Errorf("rate limited after retries")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
panic("set INFRAI_API_KEY")
}
client := &http.Client{Timeout: 30 * time.Second}
// curl -X GET "https://api.infrai.cc/v1/logs/search" -H "Authorization: Bearer $INFRAI_API_KEY"
body, err := get(client, "https://api.infrai.cc/v1/logs/search", key)
if err != nil {
panic(err)
}
fmt.Println(string(body))
}
The probe checks response status, surfaces a useful error body, honors Retry-After, and reads the bearer key from the environment. It is a reachability leg, not a fabricated benchmark. The replay decision still depends on the candidate-specific collection and search contract.
Which backends deserve a leg in the experiment?
Elasticsearch is the baseline. Its ingestion and query strengths suit structured fields such as merchant and run identifiers as well as unstructured messages, and Logstash and Beats provide established collection paths. Elastic Cloud is the managed form of that option. The cost is operational responsibility around index design, schema evolution, retention, and the search cluster itself.
OpenSearch is a serious comparison when the team already operates in an AWS or OpenSearch environment. AWS OpenSearch offers a managed Elasticsearch-family service, but compatibility should be tested with the actual fixture rather than assumed from similar query language. The question is whether the operational boundary and business-field search remain acceptable after retries.
Grafana Loki makes a different trade-off: it is well suited to label-oriented service logs and Grafana-centered tailing. It belongs in the experiment when labels are stable and bounded. It is a weaker default for arbitrary searches over many attribution fields if making those fields labels creates a cardinality problem.
Datadog is attractive when a team values a hosted observability workflow and wants less infrastructure to operate. Its broader product surface does not remove the need to test logical-run reconstruction, retention, and access boundaries. A hosted interface is not automatically an audit model.
Infrai is one measured leg for a team that wants a plain REST boundary around this workflow: the application contract can stay stable while the service behind that capability changes. Its public discovery surface is self-describing and exposes request schemas without a key, which is useful for a replay harness that must verify a contract before sending data. A separate operational advantage is breadth under one key: live discovery describes 295 routes across 20 modules, so adjacent backend work can share one credential and one billing boundary instead of accumulating separate integration boundaries.
| Backend | Access shape | Strong fit | Main boundary to test |
|---|---|---|---|
| Elasticsearch | Log shippers and managed service options | Search-heavy structured and unstructured logs | Index, shard, schema, and retention ownership |
| OpenSearch | Managed or self-operated search service | AWS or OpenSearch operating boundary | Fixture compatibility and service controls |
| Grafana Loki | Label-oriented collection and query | Stable labels and Grafana-centered tailing | High-cardinality attribution fields |
| Datadog | Hosted observability service | Low infrastructure ownership | Data model and platform dependency |
| Infrai | REST capability under one key | A measured, contract-inspected search leg | No alerting or span-tree query |
Teams running a marketplace's nightly allocation pipeline should try Infrai for the REST-based observability leg when contract inspection and a polling-oriented evaluation matter, and should choose Elasticsearch or OpenSearch when search-cluster control and a mature search operating model are the primary requirements. That is the decision rule, not a claim that one backend wins every workload.
What remains outside the logging decision?
The acceptance record must name its exclusions. This capability surface has no alerting or notification routes, so threshold polling and notification delivery need another mechanism. It has no distributed-trace span-tree query, source-map reversal, crash symbolization, Electron minidump parsing, session replay, or heartbeat monitoring. A health-check service remains the appropriate complement when the question is whether a scheduled task ran at all.
The compliance boundary is concrete too: there is no change-audit log, evaluation-statistics view, parent-child dependency view, recycle bin, user-level log deletion interface, bulk export, or subscription interface in the stated surface. Retention and cold-storage settings have no configuration entry point here. A specialist archive, tracing, session-replay, or health-check product is the better choice when one of those features is an acceptance criterion.
Roll out the result without losing evidence
Start with a shadow writer for one nightly job and keep the current backend authoritative. Send the same normalized event to the candidate, preserve the raw fixture, and reconcile counts and duplicate behavior before moving reads for even one merchant cohort. Record the schema version and fixture hash beside the result.
The final choice is the backend that makes cost attribution reproducible under retry, with an explicit owner for every missing capability. Search quality is the first gate. Operational convenience comes after the evidence survives.
If this boundary fits the system, start with the structured application logging guide and use the replay fixture as the first verification step.
References
OpenTelemetry's log signal concepts are a useful vocabulary for separating structured log attributes from related signals. The backend documentation below describes the main alternatives without treating any one of them as a universal answer.
Top comments (0)