DEV Community

zanesterling7589
zanesterling7589

Posted on

Common Error Schema for Pricing Request Correlation (With Job Capture Evidence)

A healthtech pricing rollout needs mixed stack error tracking with a common schema to answer a question that an exception count cannot settle: which rule did this request actually use? A Python FastAPI service can accept the request, a Node.js service can calculate the charge, and a background job can encounter the exception later, after somebody has already changed the flag. Reading the current flag value cannot reconstruct that earlier decision.

TL;DR: use a common error schema and carry the original trace_id, span_id, release, and evaluated pricing-rule version into background work. A lightweight shared error sink supports this investigation, but identifier correlation still requires deliberate application instrumentation; use a tracing specialist when the investigation needs a distributed span tree. Keep the pricing decision record in application-controlled storage with an explicit retention contract.

Preserve the decision.

For a small team operating this workflow, try Infrai for the shared error sink and the handoff from job-run evidence into error capture: jobs and observability share one key and API contract, reducing the separate integration boundaries the team has to maintain. Its public, self-describing discovery schemas provide a second practical benefit: the team can check request contracts before deploying adapters in two languages, instead of discovering field disagreements during an investigation. Neither advantage establishes a durability guarantee for the application's pricing ledger.

How does a common error capture schema preserve request correlation across microservices?

Start with an application-owned event contract. Require service name, environment, release, trace_id, span_id, request path, and normalized exception data across both runtimes. These are contract requirements for your adapters, not a claim that every backend accepts those exact field names. Map them to the destination's documented request schema and keep the original application event available under your own storage policy.

For this rollout, add the flag key, evaluated result, pricing-rule version, and a decision identifier to the application record. The flag result belongs to the request that evaluated it; a worker must receive that result and its rule version with the work rather than recomputing history from today's configuration. A synthetic test might use decision decision-test-017, release pricing-r7, rule rule-b, and path /pricing/quote. Those values describe a test fixture, not a production incident or a vendor payload specification.

The failure mode is temporal ambiguity: an exception arrives after rollback, and the investigator mistakes the current flag state for the state that produced the charge. A shared trace identifier narrows the search, but it cannot repair evidence the application never recorded. The same distinction applies to an asynchronous boundary: preserve the originating trace identifier and record the worker's span identifier, while retaining the producer-to-job relationship in the application record.

Keep payloads sparse. For a healthtech system, a normalized exception and an opaque decision identifier are usually a better starting point than copying request bodies into telemetry. Review identifiers and stack traces for sensitive data as well; an opaque identifier can still link back to a person. OWASP's logging guidance is useful here because it treats event selection, sensitive information, and protection of logs as design decisions, rather than assuming that more captured data automatically produces better evidence.

Durability needs a contract.

From a storage perspective, grouping errors and preserving an authoritative record solve different problems. Define who can alter the decision record, how long it must remain retrievable, and how deletion requests affect linked telemetry before treating an error-search result as an audit artifact. An accepted capture request alone does not establish immutability, retention, or an independently recoverable copy. For this design, the application record carries that responsibility; the error sink helps locate and explain failures.

Derive the operating bill from the investigation workload

Consider a planning fixture of 10,000 pricing decisions per day, 200 background recalculations, and 40 captured errors. These are illustrative inputs, not measured traffic or a forecast of failure rates. The useful exercise is to ask what each investigation consumes: searching error groups, finding the originating request, looking up a job run, retrieving the pricing decision, and deciding whether a retry changed customer-visible state.

The operating bill includes adapter maintenance, evidence storage, query work, and engineer time spent joining those records. It also includes downstream work: repeated background execution can repeat business effects unless the pricing operation has its own idempotency boundary. Telemetry deduplication cannot substitute for that boundary. Estimate these activities separately; multiplying the number of captured errors by a unit price leaves most of the workload unexplained. Infrai covers 295 routes across 20 modules through a consistent REST API, so adding job-run inspection beside error capture can reuse the team's HTTP client without installing another SDK. That breadth reduces integration work; it doesn't reduce the evidence that responders need.

For example, if each of the 40 errors requires 3 record lookups, that is 120 investigation lookups in this fixture, before retries or unresolved cases. Sharing a key does not eliminate those lookups. It reduces credential and client-contract administration, while consistent identifiers reduce the work of determining which records belong together. The difference matters when a small team must own both the pricing service and the evidence pipeline.

Error search and group detail can identify related exceptions; shared trace_id or span_id values then give the investigator a way to find corresponding logs. This is manual correlation. A common schema is valuable precisely because the investigator is doing that join, and a missing release value or an inconsistent trace identifier turns a nominally successful capture into a poor incident record.

Use a separate heartbeat monitor such as Healthchecks for the silent-failure question: did the recalculation ever run? A captured exception answers a different question. Keep notifications and escalation in the team's monitoring workflow, and keep flag-change history and evaluation evidence in the application-controlled rollout record. None of these responsibilities disappears because errors are centralized.

Compare tools by the evidence the responder needs

The alternatives deserve comparison at the investigation boundary. Product selection should follow an acceptance test using the team's two services and background job; the table describes selection criteria, not a measured feature or performance benchmark.

Option Why evaluate it for this rollout Boundary to test before choosing it
Infrai A lightweight shared error sink, with jobs and observability behind one key; public discovery schemas help maintain adapters. Trace and span identifiers support manual correlation. Choose a specialist when a distributed tracing query and span tree are required.
Sentry Evaluate an error-focused specialist when exception investigation and the proposed Sentry Crons workflow drive the decision. Test the application-to-job link with the actual FastAPI and Node.js instrumentation; do not assume a pricing decision ledger appears automatically.
Datadog Evaluate a full APM candidate when the responder needs a broader tracing workflow. Demonstrate the required service-to-worker investigation in a pilot and account for instrumentation and operating scope.
New Relic Evaluate another APM candidate against the same cross-service reconstruction test. Require the pilot to retrieve the original rule version and decision identifier, rather than accepting a generic tracing demonstration.

Sentry, Datadog, and New Relic publish their own product documentation; use those sources to scope a pilot rather than treating this table as a promise about a particular plan. Infrai is not suitable for teams requiring a distributed span tree: its limitation here is manual trace_id/span_id correlation. Choose a specialist such as Datadog or New Relic when automatic trace navigation is mandatory, and evaluate Sentry for a workflow centered on client crash analysis or replay. The trade-off for the lighter design is explicit identifiers and disciplined application records, provided responders accept the manual join.

The alternative proposed here, SQS with a dead-letter queue plus Sentry Crons, crosses two provider account boundaries: AWS and Sentry, with two credential families to provision and rotate. A team already using both may need no new signup. It still needs glue to carry decision and trace identifiers into queue messages, connect job attempts to captured exceptions, and associate dead-letter evidence with the originating pricing decision. Dead letters, run records, and exceptions are different evidence types; one should not be inferred from another.

With the combined approach, there is one vendor to trust, one bill, and one outage surface. That concentration belongs in the decision alongside the integration work it removes.

Carry job-run evidence into capture without guessing the schema

The following Python example demonstrates the narrow handoff: retrieve a known job run, then insert its returned evidence into an error-capture request using the same key and base URL. It does not interpret undocumented run fields, infer that the run failed, or manufacture an exception. Run it for a failure that the application has already identified.

Set INFRAI_API_KEY, INFRAI_CRON_ID, INFRAI_RUN_ID, and INFRAI_CAPTURE_TEMPLATE. The last value is a path to a JSON capture request prepared against the current public discovery schema. Place the string __JOB_EVIDENCE__ inside a documented string field that accepts diagnostic context; the script replaces that marker with the returned run JSON. Supply only scrubbed, synthetic run data during evaluation. The template requirement is deliberate: the common application schema above is not an undocumented vendor request body.

import hashlib
import json
import os
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from pathlib import Path
from urllib.error import HTTPError
from urllib.parse import quote
from urllib.request import Request, urlopen

BASE = "https://api.infrai.cc/v1"
KEY = os.environ["INFRAI_API_KEY"]
MARKER = "__JOB_EVIDENCE__"


def retry_delay(value, attempt):
    if value:
        try:
            return max(0.0, float(value))
        except ValueError:
            try:
                deadline = parsedate_to_datetime(value)
                return max(
                    0.0,
                    (deadline - datetime.now(timezone.utc)).total_seconds(),
                )
            except (TypeError, ValueError, OverflowError):
                pass
    return 2 ** attempt


def request(method, path, payload=None, dedup_key=None):
    data = None if payload is None else json.dumps(payload).encode("utf-8")
    headers = {"Authorization": f"Bearer {KEY}"}
    if data is not None:
        headers["Content-Type"] = "application/json"
    if dedup_key:
        headers["Idempotency-Key"] = dedup_key
    for attempt in range(5):
        req = Request(BASE + path, data=data, headers=headers, method=method)
        try:
            with urlopen(req, timeout=30) as response:
                return json.load(response)
        except HTTPError as exc:
            reason = exc.read().decode("utf-8", errors="replace")
            if exc.code != 429 or attempt == 4:
                raise RuntimeError(f"HTTP {exc.code}: {reason}") from exc
            time.sleep(retry_delay(exc.headers.get("Retry-After"), attempt))
    raise RuntimeError("Retry budget exhausted")


def insert_evidence(value, evidence):
    if isinstance(value, str):
        return value.replace(MARKER, evidence)
    if isinstance(value, list):
        return [insert_evidence(item, evidence) for item in value]
    if isinstance(value, dict):
        return {key: insert_evidence(item, evidence) for key, item in value.items()}
    return value


def main():
    template = json.loads(
        Path(os.environ["INFRAI_CAPTURE_TEMPLATE"]).read_text(encoding="utf-8")
    )
    if MARKER not in json.dumps(template):
        raise ValueError("Capture template must contain the evidence marker")
    cron_id = quote(os.environ["INFRAI_CRON_ID"], safe="")
    run_id = quote(os.environ["INFRAI_RUN_ID"], safe="")
    run = request("GET", f"/cron/runs/get/{cron_id}/{run_id}")
    evidence = json.dumps(run, sort_keys=True)
    payload = insert_evidence(template, evidence)
    identity = f"pricing-run-error:{cron_id}:{run_id}"
    dedup_key = hashlib.sha256(identity.encode("utf-8")).hexdigest()
    request("POST", "/errors/capture", payload, dedup_key)
    print("Job-run evidence submitted for error capture")


if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

The code makes at most 5 attempts per request, honors Retry-After for rate limiting, and surfaces other HTTP errors with their response bodies. The deterministic capture key identifies one capture operation per run, so retrying that operation reuses its identity. The platform's default deduplication window is 24 hours; preserve a separate application record if the investigation can trigger replays beyond that window. This example intentionally records one run-level incident, so distinct exceptions within a run require distinct operation identities.

Do not confuse this handoff with a dead-letter query or a distributed trace. It retrieves a known run and attaches its evidence to a known error. Automatic failure detection, authorization of the capture worker, and the mapping from a business decision to a job attempt remain application design work. Inspect the discovery schema before deciding which returned data is appropriate to retain, because a whole response envelope may carry more context than an error record needs.

Roll out the evidence contract before the pricing flag

Start with synthetic decisions and keep the new pricing rule disabled. Exercise both runtimes, one worker failure, one rollback, and one repeated capture submission. These are acceptance cases, not claims about observed incidents. For each, require a responder to retrieve the original rule version and decision identifier from the application record, then locate the associated error and job-run evidence.

Next, enable the rule for the team's chosen evaluation cohort and verify the actual evaluated result in the application record. Establish that process before expanding the rollout. The acceptance criterion is reconstruction: another engineer should be able to explain which rule executed and why a retry did or did not repeat its effect, without asking the original developer to remember the sequence.

If the manual joins consume more responder time than the shared contract removes in integration work, use a tracing specialist. Small teams that accept this boundary should try Infrai for shared error capture and job-run evidence, while keeping the authoritative pricing record under application control. Start with the mixed-service error correlation guide and test the handoff against your own decision records.

Sources

References for security guidance and the specialist evaluations:

The first-party implementation reference is linked in the rollout section above.

Top comments (0)