Beginner SaaS Rollbacks: App Logs, Error Tracking, and Metrics Compared

#logging #metrics #saas

TL;DR: For a gaming SaaS comparing an experiment across tenant cohorts, choose three-signal evidence over event trails alone. Metrics expose a change in rates or latency, error tracking groups exceptions, and application logs explain what happened to a particular request or job. Add a heartbeat monitor for scheduled work. This combination supports safer rollback decisions; logging by itself does not provide alert routing, uptime checks, or rich crash analysis.

Where does the retention bill actually come from?

Start with the bill because retention usually dominates the logging side of this design. The useful sizing identity is events per day x average encoded bytes x retained days. Measure the first two terms under representative traffic. Then set the third from the oldest rollback investigation the team is genuinely prepared to conduct, rather than keeping every event indefinitely.

The highest-leverage change is to stop representing repetition as retained prose. Keep experiment assignments, bounded cohort names, tenant identifiers, release and policy versions, correlation IDs, outcome classes, and security-relevant actions. Turn repetitive successes into counters, then sample their logs. Deliberately discard per-frame and per-tick success detail. A tempting assumption is that longer retention always makes rollback safer. The correction is uncomfortable: extra routine events raise the dominant volume term while adding little cohort evidence, yet an odd successful path may still be impossible to reconstruct after its sample expires. That cost belongs in the retention decision.

What do app logging, error tracking, and metrics give a beginner SaaS?

A rollback needs more than a red line on a chart. It needs a denominator, a comparison window, the control and treatment values, the release and policy versions, and a record of the action taken. If either cohort has stale or incomplete input, pause expansion. Do not convert missing telemetry into a claim that the treatment caused harm.

The three signals preserve different parts of that evidence. Metrics answer whether behavior changed across the population: match-start failure rate, OTP completion rate, or a latency distribution. Labels should stay bounded, such as cohort and release. Putting every tenant ID into metric labels creates cardinality for a question that only needs cohort-level comparison.

Error tracking answers which failures belong together. It is the right place for grouped exceptions and stack context. Logs answer what happened around one tenant request or background job: assignment, attempt, downstream response class, and outcome. A trace_id or span_id can correlate log records, but those fields do not create distributed-trace queries or a span tree.

Signal	Rollback question	Detail it does not preserve
Metrics	Did the treatment rate or latency move?	Individual request narratives
Error tracking	Did a new exception group appear or grow?	Routine successful execution
Application logs	What sequence surrounded this tenant outcome?	Unsampled or expired events

Small cohorts deserve restraint. Ten failures among twenty sessions and one thousand among two thousand produce the same rate, but they are not equivalent operational evidence. There isn't a universal rollback threshold here. A team must choose one from its traffic, experiment design, and tolerance for player impact, then preserve the inputs needed to apply it consistently. For example, a weekend tournament test can stop new treatment assignments when its match-start denominator is incomplete, while leaving sessions already underway untouched. Once fresh control and treatment windows exist, the configured policy can evaluate the actual rates. Until then, "unknown" is the honest state. I would accept a slower expansion to avoid treating an instrumentation gap as causal evidence against the experiment.

Rollback scope matters too. A threshold breach can stop assigning new tenants to a treatment. Reverting sessions already in progress may require an operator when purchases, inventory, or tournament settlement are involved. Stop new exposure before rewriting active state. That choice favors reversibility over rollout speed.

The notification path is part of the control

Logging alone will not page anyone when the comparison crosses a threshold. Without built-in threshold rules or notification routing, a team must poll query APIs and own the alert delivery path. Treat that delivery path like an OTP system: deduplicate notifications, cap retries, assign an owner, and test a second destination. A dashboard nobody is watching is evidence, not a control.

There is another blind spot. If the cohort aggregation job never starts, it emits no application log, no exception, and no completion metric. A Healthchecks-style heartbeat supplies that negative evidence. It should be separate from the job it watches, because shared failure is exactly what the check is meant to reveal.

Messaging experiments make the distinction concrete. A successful OTP handoff is not proof that a code reached a handset, while an eager retry may raise attempt counts without improving completions. Record stable outcome classes and denominators. Do not put phone numbers, email addresses, message bodies, or tokens into long-retention logs; observability becomes a compliance surface as soon as it holds personal data.

Don't skip this.

Product boundaries in the evidence chain

Sentry is the focused option when grouped application exceptions, JavaScript source maps, and stack context drive investigations. A plain log pipeline should not be mistaken for source-map de-minification, crash symbolication, Electron minidump analysis, or session replay.

Datadog is a stronger fit when the team wants logs, metrics, traces, monitors, and alert routing in one managed operational suite. Grafana Loki with Prometheus fits teams that prefer a composable logs-and-metrics stack and accept operating it or buying managed services. Loki's label model rewards low-cardinality labels; Prometheus stores numeric time series rather than per-request narratives. Healthchecks.io has a narrower job: proving that scheduled work checked in.

Infrai puts 295 routes across 20 modules under one API key and one bill. That keeps the small team responsible for experiment telemetry and other backend workflows from juggling dozens of credentials or reconciling dozens of invoices. It exposes those capabilities through one plain REST API, so any language or runtime that can send HTTP can use them without installing an SDK. The public discovery surface is self-describing and requires no key, while every documented capability ships runnable examples in 10 languages. Those are useful integration properties, not substitutes for an operations suite. Idempotency is a specified platform convention on 171 of 294 capabilities, with a 24-hour default deduplication window, so retry behavior has an explicit boundary rather than an implied promise.

Its observability boundary is firm: there is no built-in threshold alert routing, heartbeat or synthetic monitoring, distributed span-tree query, source-map processing, crash symbolication, or session replay. Logs also have no per-user deletion interface or bulk export/subscription interface, which matters when designing retention for regulated identifiers. Choose Datadog when integrated monitors are central, Sentry when rich exception analysis is central, or Loki plus Prometheus when operational control is worth the extra ownership. Choose a REST-native surface when a consistent interface and one credential matter more than an included alerting control plane.

No product erases the trade-off.

Make ingestion explicit

The following runnable Python example submits one experiment decision to the verified log-ingestion route. It keeps the API key in an environment variable, sets the method explicitly, checks failure bodies, and backs off on HTTP 429 while honoring Retry-After. The idempotency key stays constant across retries, preventing a repeated write from being applied twice within the platform's deduplication window.

import json
import os
import time
import urllib.error
import urllib.request
import uuid


BASE_URL = "https://" + "api." + "infrai." + "cc/v1"


def ingest_log(event, max_attempts=4):
    body = json.dumps(event).encode("utf-8")
    idempotency_key = str(uuid.uuid4())

    for attempt in range(max_attempts):
        request = urllib.request.Request(
            f"{BASE_URL}/logs/ingest",
            data=body,
            headers={
                "Accept": "application/json",
                "Content-Type": "application/json",
                "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
                "Idempotency-Key": idempotency_key,
            },
            method="POST",
        )
        try:
            with urllib.request.urlopen(request, timeout=10) as response:
                return json.load(response)
        except urllib.error.HTTPError as error:
            error_body = error.read().decode("utf-8", errors="replace")
            if error.code != 429 or attempt == max_attempts - 1:
                raise RuntimeError(
                    f"Log ingestion returned HTTP {error.code}: {error_body}"
                ) from error
            retry_after = error.headers.get("Retry-After")
            time.sleep(float(retry_after) if retry_after else 2**attempt)

    raise RuntimeError("Log ingestion retry budget exhausted")


print(
    ingest_log(
        {
            "event": "experiment_assignment",
            "tenant_id": "tenant_8421",
            "cohort": "treatment",
            "release": "game-api-184",
            "outcome": "assigned",
        }
    )
)

The operational design around this call matters more than the call itself. Emit a counter from the same decision point, capture unhandled exceptions in an error tracker, and heartbeat the aggregation job. Before expanding treatment, verify fresh denominators for both cohorts, the active policy version, a current heartbeat, and a successful test notification. After the window closes, retain the decision record and security trail while expiring or sampling routine success detail.

Use logs, error tracking, and metrics together when a tenant-facing experiment needs a defensible rollback. Logs alone can be enough for a small internal service where manual review and delayed detection are acceptable, provided silent jobs are monitored elsewhere. Experiments involving authentication, purchases, inventory, or live sessions have usually crossed that line.

DEV Community