Log Evidence for Payment Incidents: Self-Hosted Loki, Elastic Cloud, or a Logs API?

#logging #observability #fintech #devops

A payment dispute lands about six weeks after the charge, and someone has to say what the system actually did — which authorization went out, what the processor returned, whether the retry fired twice. If you're a small business running a fintech app, use a hosted log search API when nobody on the team wants to own an index, and self-host Loki when you already run Kubernetes and would rather the storage line stay in your own cloud bill. Elastic Cloud is the middle path, and you pay for it with a cluster you now maintain.

Retention is the axis. Not query syntax.

Every option here can find the string charge_failed. Only some of them still have it 90 days later at a price you were willing to pre-approve, and that number decides the architecture more than any feature list does. So the comparison that matters is what each choice does to your evidence window and to the invoice that pays for it.

What "enough evidence" means when a customer disputes a payment

Before: logs are a debugging convenience. You keep a week, you sample the noisy endpoints, and nobody minds, because the only consumer is an engineer with a fresh memory of the deploy.

After: logs are evidence. The consumer is a support agent, a chargeback form or an auditor, and they all arrive late. Two defaults flip at that moment — retention goes from days to months, and completeness starts to matter more than volume.

Picture the chain behind one failed card charge. Checkout writes an intent. The payment service calls the processor. The processor calls your webhook back, possibly twice. A worker retries the ledger write and gives up. Five hops, four services, one annoyed customer. To reconstruct that you need the five lines joined by a shared trace_id, each carrying which service emitted it and which environment it came from. Miss the join key and you don't have evidence, you have five unrelated strings that happen to mention the same amount.

The other half of the job is deciding what you must not keep. GDPR's data minimization principle is the reason a log line should carry a payment intent id instead of a card number, and a hashed email instead of the address. Keeping less is also the only retention strategy that scales, since the GB you never ingested is free forever.

Should a small business self-host Loki or buy a hosted log search API?

Loki's model is labels plus object storage — it indexes a small label set and brute-forces the rest at query time, which is why its storage line stays genuinely small. You're paying object-storage rates for compressed chunks. What you're really buying, though, is a job: somebody runs the ingester, the compactor and the query frontend, sizes the cache, and notices when a retention change stopped applying. On a team of three, that somebody is also the person writing the payment integration.

Elastic Cloud inverts the trade. You get a real query engine — aggregations, structured filters, the lot — and in exchange the index is expensive to keep hot, so you end up designing tiers and lifecycle policies around it. For a dispute workflow that touches maybe forty log lines a week, that's a lot of machinery guarding a rare read.

Hosted log APIs are the third shape. You POST JSON over HTTPS, the vendor owns the index, and you're billed roughly per GB ingested plus retention; Axiom and Better Stack live here, Datadog lives here with a much larger product wrapped around it, and CloudWatch is the reference for how the per-GB shape behaves as volume grows. Infrai belongs in that same group with a different bet — logs are one module on a single API that also covers queues, object storage, feature flags and outbound email, all under one key and one bill. For a small fintech team that already knows it needs scheduled jobs and transactional receipts, the interesting part isn't the log search itself, but that the evidence pipeline stops being a fourth vendor, a fourth key and a fourth invoice to reconcile at month end.

Tagging lines so the bill points at a service, not at "logs"

Cost attribution is the part small teams skip and then cannot retrofit. A log line is immutable once written; if it went in without a service tag, no query will tell you six months later which deploy tripled your ingest.

Write the attribution keys at ingest time. Two fields carry most of the weight — service and environment — and both come back in a hosted search result alongside message, level and timestamp. With those present you can answer the only cost question anyone asks at review time, which is whether the part of the system generating the volume is also the part generating revenue.

Then pull the boring lever. Debug lines from a healthy checkout path are not evidence and shouldn't leave production at all, while error and warning lines from the payment path are, and should outlive them by months. That's a filter in your logger rather than a feature you buy, and at a few GB a month it's the whole difference between an invoice you can explain and one you can only pay.

A minimal ingest call, and what to check on the way back

Here's the write side end to end — one HTTP call, an explicit method, an idempotency key so a retried batch doesn't become two records, and a 429 path that backs off instead of hammering.

// Write side: POST /v1/logs/ingest
const BASE = process.env.INFRAI_BASE_URL!;   // the provider's /v1 base URL
const KEY = process.env.INFRAI_API_KEY!;     // ifr_... , never a literal in source

type EvidenceLine = {
  level: "info" | "warning" | "error";
  message: string;
  service: string;        // attribution key: who spent the ingest budget
  environment: string;    // prod / staging, so one bill never hides the other
  timestamp: string;      // ISO8601
  trace_id: string;       // joins the five hops of one payment attempt
};

export async function shipEvidence(line: EvidenceLine, eventId: string): Promise<void> {
  for (let attempt = 0; attempt < 4; attempt++) {
    const res = await fetch(`${BASE}/logs/ingest`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${KEY}`,
        "Content-Type": "application/json",
        "Idempotency-Key": eventId,   // same event retried => still one stored line
      },
      body: JSON.stringify(line),
    });

    if (res.ok) return;

    if (res.status === 429) {
      const retryAfter = Number(res.headers.get("Retry-After"));
      const waitMs = Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : 400 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      continue;
    }

    throw new Error(`logs.ingest ${res.status}: ${await res.text()}`);
  }

  throw new Error("logs.ingest: rate limited after 4 attempts");
}

Two things in there are worth copying whichever vendor you land on. The idempotency key makes the retry safe, so the same event inside the dedup window collapses to one stored line — which matters a lot when the retry loop is itself the thing producing your evidence. And the status check is explicit: a 4xx body carries the reason, and swallowing it is how you find out three weeks later that half the evidence never landed.

Reading back is a GET on /v1/logs/search, and the result set arrives as items carrying message, level, timestamp, service and environment, plus a total. The filter parameters for that endpoint aren't declared in the public discovery schema, so if your investigation workflow leans on complex boolean queries, verify the live capability entry before building a UI on top of it. Pulling a time window and filtering client-side on trace_id — which is all the dispute workflow above needs — is fine. Log analytics as a customer-facing product feature is a different weight class, and Elastic's query DSL is honestly the better tool there.

Where each option runs out of road

Option	How logs get in	What you operate	Where it stops
Loki + Grafana, self-hosted	Agent push (Promtail/Alloy)	Ingester, compactor, object store, upgrades	Label-first search; full-text is a scan
Elastic Cloud	Bulk HTTP or Elastic Agent	Index lifecycle, tiers, shard sizing	Months of hot index is heavy for three engineers
Axiom / Better Stack	HTTP ingest or agent	Nothing	Ingest volume is the budget lever you actually pull
Datadog	Agent plus integrations	Nothing	Wide product surface, sized for bigger teams
Infrai logs API	Plain HTTP POST, one key	Nothing	Shallower query surface, no alerting pipeline

Two objections come up whenever this comparison gets made, and both are fair.

The first is alerting. A minimal hosted logs API doesn't support threshold rules or notification routing, so "page me when charge_failed spikes" becomes a small job of your own that polls the search endpoint, plus something like Healthchecks for the silent case where the job that should have run didn't run at all. If paging is the actual reason you're buying observability, Grafana's alerting or a product built around incident routing is the better buy, and I'd rather say that plainly than pretend a log store is an alerting stack.

The second is erasure and export. A hosted API that lacks a per-subject delete route and a bulk export feed pushes both problems to write time: pseudonymize before ingest, keep the mapping in your own database where deletion is a normal UPDATE, and accept that the log store holds tokens rather than people. When your compliance reviewer wants per-record deletion inside the log store itself, self-hosted Loki or Elastic — where you own the index — is the honest answer. Same story for tracing: a trace_id field lets you join lines, but it isn't a span tree, so latency archaeology across services still wants OpenTelemetry pointed at a tracing backend.

So the decision rule, compressed. Already running Kubernetes with someone who enjoys it? Loki. Log search is a product feature, not an internal one? Elastic Cloud. Want the evidence to exist without a fourth thing to operate, and would rather add the next backend capability as one more endpoint than one more contract? A hosted logs API earns the slot — and your mileage may vary on which one, since the differences at small volume are mostly about what else you're already buying from that vendor. Pick your retention window first. That's the thing you're actually paying for.

DEV Community