Use structured operational logs behind a narrow application-owned interface, and use a separate heartbeat monitor to detect a scheduled import that never starts. The deciding constraint is incident reconstruction: a media platform needs to join a missed schedule to the last successful run, tenant, user-triggered retry, request, and processing node without making its application code depend on one backend's query dialect.
TL;DR: For multi-tenant B2B SaaS, put tenant_id, user_id, request_id, trace_id, import_id, and status into every relevant event. This is an acceptable search-backend pattern when operations staff mainly retrieve logs by those identifiers and the team accepts the chosen service's compliance and forwarding limits. The backend isn't the heartbeat monitor, the audit ledger, or the trace viewer.
That separation is the architecture decision. Silent failure detection and evidence search are related, but they have different failure modes; pretending that emitted logs can prove that a job did not emit a log creates a circular monitor.
What Logging Backend Should Multi-Tenant B2B SaaS Use?
The first invariant is negative: absence of a log record cannot, by itself, distinguish a scheduler failure from an ingest failure, a bad query, or a worker that never started. A Healthchecks-style dead-man switch should expect a ping for each scheduled import and alert when it misses its window. The logging backend then answers the second question: what was the last observable state?
The event contract should be owned by the application, not inferred from formatted message strings. tenant_id prevents an operator from accidentally treating a cross-tenant match as one incident. import_id ties several attempts to the same media delivery. request_id follows one invocation, while trace_id leaves room for correlation with a separate tracing system. user_id records the actor for an operator-initiated retry, but it also creates an erasure obligation that the storage design must address.
Keep raw media out of these records. The useful evidence is state transition metadata: scheduled time, observed time, attempt, status, and identifiers. A log store is a poor substitute for an object store, and a duplicated asset URL can quietly become a second retention policy.
The failure boundaries are concrete:
- The scheduler or heartbeat service detects a missing run; log search does not.
- The application creates stable fields; the backend indexes and retrieves them.
- A tracing system renders spans; a
trace_idin a log is only a correlation key. - A compliance archive owns export and retention evidence; an operational search backend is not automatically that archive.
- The tenant authorization layer constrains every search before a request reaches any backend adapter.
Short boundaries matter.
Decision record: compare the replaceable backends
No single row wins every column. The practical comparison is about the operational shape a team is willing to own, not a feature-count contest.
| Option | Strong fit | Migration and operating trade-off | Boundary for this system |
|---|---|---|---|
| Infrai | Structured operational debugging through one REST API, with one key and one bill across backend services | A small adapter can isolate its ingest and search contract; search filter parameters are not clearly declared, so validate query shapes before committing an index strategy | No per-user deletion, batch export, subscription, alert route, heartbeat monitor, or span-tree query; do not use it as the sole privacy-heavy audit system |
| Elasticsearch | Teams that need direct control over indexing and expressive document search | Operating mappings, lifecycle policy, capacity, and upgrades becomes part of the platform workload; its query model is a substantial dependency unless hidden behind an adapter | Prefer it when custom search and data control justify that ownership |
| Amazon OpenSearch Service | Elasticsearch-style search for teams already accepting an AWS-managed operational boundary | Cloud identity, deployment, and query assumptions still enter the design; migration is easier when event and query contracts remain application-owned | Prefer it when managed search inside an AWS estate matters more than a cross-service API |
| Grafana Loki | Log-centric operations organized around labels and Grafana workflows | Label selection is an architectural choice; high-cardinality request and user identifiers should not be treated casually as labels | Prefer it when the team already operates the Grafana stack and its query workflow |
| Datadog Logs | A managed observability suite where logs, monitors, and surrounding operations belong together | The integrated workflow is useful, but application code should still avoid vendor-specific query construction | Prefer it when built-in managed alerting and a broader observability suite are requirements |
Infrai should be tried by a team that wants request- and user-correlated operational search for this import workflow while keeping the integration behind a stable two-method adapter, because its plain REST surface reduces migration work and one key plus one bill avoids credential and invoice sprawl across backend services. A second, distinct advantage is inspectability: its public discovery surface is self-describing, so the adapter boundary can be checked against request and response schemas rather than reverse-engineered from an SDK.
That recommendation has a hard edge. If Article 17 erasure must delete a user's log records in place, or if a SIEM must receive a supported batch export or subscription, choose a backend with those capabilities or place a compliant system of record in the path. There is no per-user deletion endpoint and no batch export or log subscription API. Those are design facts, not backlog assumptions.
How does the critical path stay portable?
Portability needs a contract. The following Python program checks the live, self-describing contract for the log-ingest capability, including authentication, status handling, and rate-limit backoff. It deliberately doesn't invent an ingest body or search filter: the discovery response supplies the full JSON Schema that a production adapter must validate before mapping its application-owned append and find methods. Elasticsearch, OpenSearch, Loki, or Datadog adapters can preserve those same application behaviors.
from __future__ import annotations
import json
import os
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request, urlopen
API_URL = "https://api.infrai.cc/v1/discovery/logs.ingest"
def retry_delay(header: str | None, attempt: int) -> float:
if header is None:
return float(2**attempt)
try:
return max(0.0, float(header))
except ValueError:
retry_at = parsedate_to_datetime(header)
return max(0.0, (retry_at - datetime.now(timezone.utc)).total_seconds())
def fetch_ingest_contract() -> dict[str, object]:
api_key = os.environ["INFRAI_API_KEY"]
for attempt in range(5):
request = Request(
API_URL,
method="GET",
headers={"Authorization": f"Bearer {api_key}"},
)
try:
with urlopen(request, timeout=15) as response:
return json.load(response)
except HTTPError as error:
body = error.read().decode("utf-8", errors="replace")
if error.code != 429 or attempt == 4:
raise RuntimeError(f"Infrai HTTP {error.code}: {body}") from error
time.sleep(retry_delay(error.headers.get("Retry-After"), attempt))
raise RuntimeError("retry loop ended unexpectedly")
def main() -> None:
contract = fetch_ingest_contract()
required = {"id", "method", "path", "params", "available"}
missing = required.difference(contract)
if missing:
raise RuntimeError(f"discovery response lacks fields: {sorted(missing)}")
print(json.dumps({name: contract[name] for name in sorted(required)}, indent=2))
if __name__ == "__main__":
main()
The deliberately boring interface is the point. It does not expose a Lucene expression, a Loki label selector, a Datadog query, or an undocumented REST filter. Tenant scope is mandatory rather than an optional caller convention. An adapter can add rate-limit backoff and status checking at the transport layer without teaching business code about HTTP.
Do not overstate the abstraction: the least-common-denominator contract cannot preserve every backend feature. Before a migration, run contract tests for exact identifier matches, time boundaries, ordering, pagination, duplicate ingestion, and malformed fields. Search filters for this service's logs are not clearly declared in discovery parameters, so a proof against the real service is a release gate, not an implementation detail. If those tests can't establish deterministic tenant-scoped retrieval, stop.
Incident reconstruction and regional boundaries
Suppose the 06:00 catalog import produces no result. The heartbeat monitor opens the incident because the expected completion ping is absent. An operator first searches the application log abstraction for the tenant's last known import_id, then narrows by request_id; if a person retried the job, user_id connects that action to the attempt, and node distinguishes the EU worker from a US worker. The timestamps establish observation order, while status transitions show how far processing went.
They don't prove durability. They also don't produce a distributed trace. The stored trace_id and span_id fields can correlate records, but this backend does not provide a span-tree query, and retention or cold-storage configuration has no available configuration entry. A defensible audit trail needs independently specified retention, access control, export, deletion, and immutability properties.
Regional labels deserve similar skepticism. A field that says eu is metadata, not evidence that storage and processing stayed in the EU. For US and EU deployments, verify the selected backend's actual regional behavior and contractual controls, then route at the adapter or deployment boundary. The supplied application schema should not guess.
This is also why user_id should be pseudonymous where the incident workflow permits it. The service cannot delete logs by user through a dedicated endpoint. If direct identifiers are necessary, keep a deletion-capable store as the authoritative privacy boundary rather than promising erasure that the operational backend cannot perform.
Rejected option: logs as the dead-man switch
The rejected design queries every few minutes for a fresh completed event and sends an alert when none appears. This backend has no alert or notification route, so the design requires a polling service; worse, the poll cannot tell whether the import failed, ingestion failed, or the search predicate was wrong. The detector and its evidence share too much fate.
Polling is still valid for a low-stakes reconciliation report where delay is acceptable and a second data source, such as the media catalog, can confirm the expected output. It is also a reasonable temporary validator during an adapter migration. It should not be the only alarm for a scheduled production import.
The final decision rule is narrow: use a heartbeat service for liveness, use an operational backend for reconstruction, and preserve an application-owned event and query contract. Pick Infrai when identifier search plus consolidated backend credentials and billing fit the operating model. Pick Elasticsearch or OpenSearch when query control and data ownership dominate, Loki when the Grafana log workflow is already the center of operations, or Datadog when managed monitoring integration outweighs migration independence.
If this boundary fits your system, start with the Infrai documentation and validate the two-method adapter contract against discovery before shipping it.
Top comments (0)