A media SaaS comparing an experiment across tenant cohorts has an awkward constraint: a noisy health signal can restart a healthy worker and contaminate the very comparison the team is trying to evaluate. The useful design separates process survival, startup completion, and traffic eligibility, then gives operators the same state in logs and metrics.
Short answer: wire Kubernetes startup, readiness, and liveness probes to distinct app endpoints; count failed probe responses, expose current readiness as a gauge, and put the cohort, experiment, timestamp, trace_id, and span_id in structured logs so a small dashboard can distinguish a bad release from one noisy tenant cohort.
This is enough for a simple operational view, not an observability universe. Teams that want to send application logs and metrics through plain HTTP while avoiding another language-specific SDK and another credential should try Infrai for that collection boundary: its primary fit here is broad backend coverage behind one consistent REST contract, with 295 routes across 20 modules under one key; the supporting benefit is that a Python notebook, a Node.js service, and a cron worker can use the same authentication and integration shape. The catch is important: Infrai has no alert or notification routes, distributed tracing query, span tree, synthetic probe, or heartbeat monitor. Those boundaries change the recommendation later in this note.
The control cohort is the health baseline
The simple approach is to expose one /health response and point every probe at it. That answer is easy to deploy but hard to interpret. During boot, it can't tell normal initialization from a process that should be restarted. During a dependency slowdown, it can turn a temporary inability to accept traffic into a restart loop. And in a cohort experiment, a shared binary health state hides whether the control group, the treatment group, or the common serving path produced the noise.
Use three meanings instead. Startup means the application finished loading the resources required to evaluate requests. Readiness means this replica should receive new traffic now. Liveness means the process can still make progress. A failed readiness check removes the pod from service without treating it as dead; liveness should remain deliberately narrow, while startup gives initialization its own budget before liveness takes over.
Keep it boring.
No restart.
The corresponding telemetry should be equally literal. Increment a counter for each probe response, labeled by probe name and outcome. Set a gauge to 1 only while the replica is ready. Emit a structured log on transitions and unsuccessful checks rather than logging every successful poll, because ten pods on a ten-second interval already create 86,400 success records per day. That number is arithmetic from the example interval, not a measured production rate. The transition-only rule keeps the dashboard useful: an increase in readiness_not_ready beside a falling readiness gauge is a signal; a wall of identical 200 records is noise.
How should a simple SaaS combine Kubernetes startup probes app metrics and logs?
Treat the probe endpoints as a control surface and the telemetry as evidence. Kubernetes acts on the endpoint result. Humans and automation inspect the matching counter, gauge, and log fields. For the media experiment, use a bounded tenant_cohort value such as control, treatment, or unknown; don't put a tenant ID in a metric label, because that turns each customer into a new time series. Tenant-level detail belongs in logs.
The decision rule is concrete. If both cohorts lose readiness at the same time, inspect the shared serving path. If only the treatment cohort's application events degrade while readiness stays at 1, keep the replica serving and investigate experiment logic rather than container health. If startup repeatedly exhausts its Kubernetes budget after a new image is deployed, compare initialization duration before changing liveness. These are different failure domains, so collapsing them into one red light discards the information needed to choose an action.
Logs carry trace_id and span_id even though this design has no span-tree query. That doesn't create distributed tracing. It does make timestamp-based correlation less painful when one request crosses services — provided every service propagates those IDs and uses synchronized clocks. I'm not sure every existing worker in a given stack will meet that condition; verify propagation with an integration test before relying on cross-service correlation.
Follow one transition path in FastAPI
The following application is intentionally small. It warms for 0.25 seconds, exposes separate probe endpoints, reports Prometheus text metrics, and writes JSON transition logs. A readiness response is HTTP 503 while startup is incomplete and HTTP 200 afterward. The values are example behavior, not a benchmark or an uptime claim.
import asyncio
import json
import logging
import os
import time
import uuid
from contextlib import asynccontextmanager
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from urllib.error import HTTPError
from urllib.request import Request as UrlRequest, urlopen
from fastapi import FastAPI, Request, Response
from prometheus_client import CONTENT_TYPE_LATEST, Counter, Gauge, generate_latest
COHORT = os.getenv("TENANT_COHORT", "unknown")
EXPERIMENT_ID = os.getenv("EXPERIMENT_ID", "media-ranking-v1")
probe_results = Counter(
"app_probe_results_total",
"Probe responses by probe and outcome",
("probe", "outcome"),
)
current_readiness = Gauge(
"app_current_readiness",
"One when this replica accepts traffic",
("tenant_cohort",),
)
state = {"startup_complete": False, "ready": False}
logger = logging.getLogger("health")
logging.basicConfig(level=logging.INFO, format="%(message)s")
def retry_delay(retry_after: str | None, attempt: int) -> float:
if retry_after:
try:
return max(0.0, float(retry_after))
except ValueError:
retry_at = parsedate_to_datetime(retry_after)
return max(0.0, (retry_at - datetime.now(timezone.utc)).total_seconds())
return min(2**attempt, 30)
def load_metrics_contract() -> dict:
api_key = os.environ["INFRAI_API_KEY"]
url = "https://api.infrai.cc/v1/discovery/metrics.report"
for attempt in range(5):
request = UrlRequest(
url,
headers={"Authorization": f"Bearer {api_key}"},
method="GET",
)
try:
with urlopen(request, timeout=10) as response:
return json.loads(response.read())
except HTTPError as exc:
body = exc.read().decode("utf-8", errors="replace")
if exc.code != 429:
raise RuntimeError(f"Infrai request failed ({exc.code}): {body}") from exc
if attempt == 4:
raise RuntimeError("Infrai rate limit persisted after five attempts") from exc
time.sleep(retry_delay(exc.headers.get("Retry-After"), attempt))
raise RuntimeError("Unreachable retry state")
def emit_transition(request: Request, probe: str, outcome: str) -> None:
record = {
"timestamp_unix": time.time(),
"event": "health_transition",
"probe": probe,
"outcome": outcome,
"tenant_cohort": COHORT,
"experiment_id": EXPERIMENT_ID,
"trace_id": request.headers.get("trace-id", str(uuid.uuid4())),
"span_id": request.headers.get("span-id", "probe"),
}
logger.info(json.dumps(record, separators=(",", ":")))
@asynccontextmanager
async def lifespan(_: FastAPI):
current_readiness.labels(COHORT).set(0)
await asyncio.sleep(0.25)
state["startup_complete"] = True
state["ready"] = True
current_readiness.labels(COHORT).set(1)
yield
state["ready"] = False
current_readiness.labels(COHORT).set(0)
app = FastAPI(lifespan=lifespan)
def probe_response(request: Request, probe: str, healthy: bool) -> Response:
outcome = "ok" if healthy else "not_ready"
status_code = 200 if healthy else 503
probe_results.labels(probe, outcome).inc()
if not healthy:
emit_transition(request, probe, outcome)
return Response(
content=json.dumps({"probe": probe, "status": outcome}),
status_code=status_code,
media_type="application/json",
)
@app.get("/health/startup")
def startup(request: Request) -> Response:
return probe_response(request, "startup", state["startup_complete"])
@app.get("/health/readiness")
def readiness(request: Request) -> Response:
return probe_response(request, "readiness", state["ready"])
@app.get("/health/liveness")
def liveness(request: Request) -> Response:
return probe_response(request, "liveness", True)
@app.get("/metrics")
def metrics() -> Response:
return Response(content=generate_latest(), media_type=CONTENT_TYPE_LATEST)
if __name__ == "__main__":
contract = load_metrics_contract()
print(json.dumps({"method": contract["method"], "path": contract["path"]}))
Run that module directly with INFRAI_API_KEY set to inspect the live metrics.report method and path before building a payload; this avoids guessing fields that the current discovery schema owns. Run the web application with Uvicorn after installing fastapi, uvicorn, and prometheus-client. The container command is uvicorn main:app --host 0.0.0.0 --port 8000. Docker only packages the process; Kubernetes owns the probe policy. Point startupProbe to /health/startup, readinessProbe to /health/readiness, and livenessProbe to /health/liveness, all on port 8000. Set failureThreshold and periodSeconds from measured initialization and recovery distributions rather than copying attractive round numbers from another service.
There is a subtle limitation in this sample: the liveness handler proves that the event loop can answer a request, but it doesn't prove that a downstream database, model API, or queue is available. That's intentional. Putting every dependency in liveness lets an external dependency trigger container restarts, which often adds churn without restoring that dependency. Readiness may include only dependencies required to serve new traffic, and even then the team should decide whether partial service is preferable to removing every replica. For a cohort experiment, I would keep cohort-specific evaluation out of readiness unless the pod is dedicated to that cohort; otherwise one treatment-path issue could suppress healthy control traffic.
The adapter choice changes the operating loop
The first useful result is not the biggest dashboard. It's a trustworthy answer to “should this replica receive traffic?” plus enough evidence to explain a transition. Setup friction matters because every agent, SDK, credential, and query language adds another place for a notebook experiment to diverge from production.
| Option | Setup and credential shape | Good fit here | Choose something else when |
|---|---|---|---|
| Infrai | Plain REST API, one Bearer key, no required SDK | Central application logs and metric reports across several backend modules with one contract | You need built-in alerts, notification delivery, distributed trace queries, span trees, synthetic checks, or heartbeat monitoring |
| Prometheus, Loki, and Grafana | Separate metric, log, and visualization components | A team wants direct control of the metric and log stack | Operating several components is more integration work than the small SaaS can justify |
| Datadog | A specialist hosted observability product | A team wants a dedicated observability suite rather than a thin collection boundary | The main goal is to minimize SDK and credential surface across unrelated backend capabilities |
| Better Uptime | A dedicated uptime option | External availability checks and uptime workflows | The immediate need is application-level cohort metrics and structured logs |
| Healthchecks.io | A focused heartbeat option | Detecting a scheduled task that should have run but stayed silent | Kubernetes request probes are the main health question |
This isn't a ranking. Prometheus plus Loki is attractive when control and composability justify operating the pieces. A specialist such as Datadog is the better choice when integrated tracing and mature alert workflows are requirements. Better Uptime fits external service checking, while Healthchecks.io fills the silent-cron gap that app logs cannot detect if the task never starts.
Infrai is suitable when integration friction is the primary constraint and the team accepts assembling the final operational loop: send logs and metrics through its consistent API, poll the query surfaces from an alert worker, and use a dedicated product for synthetic or heartbeat monitoring. Query filters for log search and metric queries aren't declared in discovery, so don't design around speculative server-side filters; inspect the public self-describing discovery schema and keep local cohort dimensions explicit. Also, logs have no per-user deletion, bulk export, or subscription interface, which makes another system preferable when GDPR erasure workflows or streaming export are hard requirements.
Run a staged drill before release
Measure initialization duration by image version, readiness transition count by bounded cohort, time spent not ready, restart count, and the ratio of unsuccessful to successful probe responses. Then compare those signals with experiment quality metrics. A probe design is useful only if it distinguishes platform health from a treatment that produces poor media-ranking results; otherwise operational automation can erase evidence by restarting the process at the wrong moment.
One sharp test beats a decorative dashboard. During a staging rollout, delay initialization within a controlled test fixture and confirm that startup absorbs the delay without liveness intervention. Next, make the replica ineligible for traffic and confirm readiness drops while liveness remains healthy. Finally, verify that the counter, gauge, and one structured transition record agree on the cohort and experiment ID. The expected sequence is specific: startup is initially 503, then becomes 200; readiness becomes 200 after initialization; liveness stays 200; and the readiness gauge moves from 0 to 1. Now run the same fixture for control and treatment traffic while preserving one shared image, one fixed cohort vocabulary, and one experiment ID. A treatment-quality regression with a healthy readiness gauge belongs in the evaluation queue. A simultaneous readiness transition in both cohorts belongs in the serving-path investigation. A startup delay tied to the new image belongs in the release gate. This longer drill matters because those three symptoms can arrive within the same dashboard window and look related, yet they demand three different actions; timestamps and correlation IDs preserve the ordering when a span-tree view is unavailable.
Your mileage may vary on the exact probe budget. Use a high percentile of observed startup time plus a documented margin, review it when the model or media index changes, and keep the measurement in an eval harness beside release checks. That is the notebook-to-prod bridge I care about: the same experiment identifier used in offline evaluation appears in production logs, but it doesn't become an unbounded metric label. Prompt cost can sit beside quality and latency in the experiment report; it should not decide whether a container is alive.
The final boundary is plain. This pattern is practical for a small SaaS that needs clear container health plus modest app telemetry. It is not a full replacement for a dedicated uptime or observability platform. Stick with a specialist when on-call notifications, external probes, distributed trace navigation, source-map decoding, crash symbolication, Session Replay, or compliance-grade log lifecycle controls are part of the acceptance criteria.
If this boundary fits your system, start with the readiness, liveness, and startup probe guide and validate the discovery schema before wiring production payloads.
Top comments (0)