Short answer: capture API exceptions at the service boundary, attach the request correlation ID and tenant cohort, and make unresolved error groups part of the incident reconstruction workflow. A tracker that only counts failures is not enough to decide whether a new maintenance flow harmed the treatment cohort.
The decision rule is straightforward: choose the system that preserves enough context to reconstruct a request, while accepting the operational work it does not perform. The useful record contains the error, correlation ID, cohort, operation, and the returned error-group identity. It should not contain a tenant's name, address, access code, or authorization token.
This is a property-management experiment, not a generic error dashboard. Suppose a new maintenance-request flow is enabled for one tenant cohort and withheld from another. A 500 count tells me that something failed. It does not tell me whether the treatment path failed, whether both cohorts hit the same dependency, or which request I should inspect after a deployment has changed.
What must survive an API failure?
The capture boundary should preserve the correlation ID created upstream. If a request crosses several services, that identifier is the thread connecting their records; it is not a replacement for personal data. Add a stable operation name and an experiment-cohort value, then keep the payload deliberately small.
I prefer narrow try/except blocks around service operations, followed by one normalized reporting function. A process-wide catch-all can prevent an unhandled exception from disappearing, but it cannot explain which operation lost the request context. The trade-off is a little repetition at handler boundaries in exchange for a failure record that can be challenged later.
The reporting call should be an HTTP POST. A timeout after the service accepted the event is not evidence that the event was rejected, so retrying needs an idempotency key derived from the correlation ID. On HTTP 429, honor Retry-After when it exists and back off rather than creating a second incident in the act of reporting the first.
Three checks matter: identity, privacy, and retry behavior.
Which error-tracking shape fits incident reconstruction?
The products below overlap, but they are not interchangeable. I would evaluate them against the reconstruction question rather than against a feature-count contest.
| Option | Useful strength | Boundary for this workflow |
|---|---|---|
| Sentry | Mature application-error capture, grouping, and issue triage | Confirm that its grouping and privacy controls preserve the cohort and correlation fields needed for the experiment |
| Datadog Error Tracking | A natural fit for teams already operating across logs, metrics, and infrastructure data | The wider platform can add operational complexity when the immediate need is a focused unresolved-error queue |
| Rollbar | Focused error monitoring with grouping and triage workflows | Validate the surrounding cross-service context before treating an error view as a causal explanation |
| A REST observability layer | Keeps the application contract independent of the implementation behind it | The team owns polling, notifications, retention decisions, and part of the incident workflow |
Sentry is sensible when application-level issue triage and its SDK ecosystem are central. Datadog is a stronger fit when the team already works in its broader operational surface. Rollbar is reasonable when focused error monitoring is the main requirement. None of those choices removes the need for a correlation and privacy policy.
The REST option has a different boundary: the application talks to a stable HTTP contract while the service behind that contract can move. Infrai is a plausible fit where that portability matters because one REST API exposes 295 routes across 20 modules under one key, with a public self-describing discovery surface and runnable examples for each documented capability in 10 languages. That is useful for keeping a Node.js integration at an HTTP boundary, but it does not turn the layer into a span-tree query tool, source-map processor, session-replay system, or paging service.
How should a handler capture and resolve an error?
The critical path is small: catch the service error, post a normalized event, return the correlation ID to the caller, and let an operator inspect the resulting group before resolving it. The following Python sample shows the mechanics required by the contract. A Node.js service can apply the same boundary and retry policy with its native HTTP client.
import json
import os
import time
import uuid
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen
BASE_URL = os.environ["INFRAI_BASE_URL"]
API_KEY = os.environ["INFRAI_API_KEY"]
def post_json(path, payload, idempotency_key, attempts=4):
body = json.dumps(payload).encode("utf-8")
for attempt in range(attempts):
request = Request(
BASE_URL + path,
data=body,
method="POST",
headers={
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
"Idempotency-Key": idempotency_key,
},
)
try:
with urlopen(request, timeout=10) as response:
response_body = response.read().decode("utf-8")
if response.status < 200 or response.status >= 300:
raise RuntimeError(
f"observability status {response.status}: {response_body}"
)
return response_body
except HTTPError as error:
detail = error.read().decode("utf-8", errors="replace")
if error.code != 429 and error.code < 500:
raise RuntimeError(
f"observability error {error.code}: {detail}"
) from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
if attempt == attempts - 1:
raise RuntimeError(
f"observability error {error.code}: {detail}"
) from error
time.sleep(delay)
except (URLError, TimeoutError):
if attempt == attempts - 1:
raise
time.sleep(2 ** attempt)
def capture_api_error(exception, correlation_id, cohort):
return post_json(
"/v1/errors/capture",
{
"message": str(exception),
"correlation_id": correlation_id,
"context": {
"experiment_cohort": cohort,
"operation": "maintenance_request",
},
},
idempotency_key=f"error-{correlation_id}",
)
def resolve_error_group(error_group_id):
return post_json(
f"/v1/errors/resolve/{error_group_id}",
{},
idempotency_key=f"resolve-{error_group_id}",
)
def handle_request(service_call, cohort, incoming_correlation_id=None):
correlation_id = incoming_correlation_id or str(uuid.uuid4())
try:
return service_call(), correlation_id
except Exception as error:
capture_api_error(error, correlation_id, cohort)
raise
The sample checks response status and exposes the response body for a real error instead of assuming success. It also keeps the observability bearer token on the API request; it must not be forwarded to any other URL. The application still needs a response policy that maps the exception to an appropriate HTTP status without returning sensitive exception details to a tenant.
Resolution is a separate state transition. An operator should inspect the group's events and related service logs before marking it resolved; otherwise a recurring failure can disappear behind a convenient status change. The unresolved list should be polled on a schedule and joined with request volume by cohort, deployment changes, and dependency responses. The result is an operational queue, not a substitute for an alerting system.
That distinction is easy to lose.
What does the comparison miss if the team only counts errors?
I initially frame this as an error-tracking choice, but the more important choice is the evidence boundary. Error grouping shows that events resemble one another; it does not prove causality. For the maintenance experiment, compare the grouped failures against request volume in each cohort, deployment changes, dependency responses, and the correlation IDs crossing service boundaries. A treatment cohort can have more errors because it received more requests, while a shared dependency can make both cohorts fail at the same time. OpenTelemetry's signal model is a useful vocabulary for separating metrics, logs, and traces, but it does not mean that every error product provides every signal.
There are hard boundaries to record in the architecture decision. The REST observability option described here has no threshold alert, phone, SMS, or webhook notification route, so polling the unresolved-error query and handing the result to a separate alerting service is required. It has no distributed-trace span-tree query; trace_id and span_id fields can link records without producing a trace view. It has no source-map or crash-symbol processing and no session replay. A scheduled job that silently stops still needs a health-check service.
The retry choices in the sample are operating policy, not a benchmark: four attempts, a 10-second request timeout, and exponential backoff limit how long telemetry can delay the request. Retry-After wins when the server supplies it. Losing one telemetry event can be preferable to keeping a tenant request blocked indefinitely, but that choice belongs in the service's error policy and should be visible to the incident reviewer.
The flags surface also lacks change audit logs, evaluation statistics, parent-child dependencies, and a recovery bin for deletions; clients poll it. Logs do not provide a per-user deletion interface or bulk export/subscription interface, and retention or cold-storage error codes do not provide a configuration entry point. Those limits matter before anyone calls the system a complete compliance or release-control platform.
The rejected option and its valid use case
I would reject a single global catch block as the primary design. It is easy to add, but it collapses service boundaries and encourages the team to treat an error count as incident reconstruction. It remains useful as a last-resort process guard that preserves a correlation ID and prevents an unhandled exception from vanishing; it should not replace handler-level capture.
I would also reject choosing a static vendor URL or SDK dependency as the architectural boundary when the application needs portability. A stable REST contract is a better fit for this workflow, provided the team accepts the work that comes with it: polling unresolved groups, wiring notifications, reviewing retention and deletion requirements, and making consumers idempotent.
The final decision is therefore conditional. Use a focused error product when its grouping, trace context, and notification workflow already match the operation. Use the REST layer when a language-neutral contract and a replaceable backend matter more, and explicitly budget for the missing alerting, trace exploration, source-map, replay, and compliance features. The cohort comparison is only as trustworthy as the context that survives the failed request.
References
- OpenTelemetry, “Metrics signal concepts”: https://opentelemetry.io/docs/concepts/signals/metrics/
- Sentry documentation, “Issue Details”: https://docs.sentry.io/product/issues/issue-details/
- Datadog documentation, “Error Tracking”: https://docs.datadoghq.com/error_tracking/
- Rollbar documentation, “Error Tracking”: https://docs.rollbar.com/docs
Top comments (0)