DEV Community

SolomonFletcher5872
SolomonFletcher5872

Posted on

Feature Flag Decisions: Node.js Tracing for Logistics Agent Cost Attribution Explained

#node #featureflags #observability #ai

Short answer: record the evaluated flag variant and policy version beside the Node.js request trace, then attach the agent loop's latency and token counts to that trace for cost attribution.

Keep it immutable.

The useful answer is to record the feature-flag decision beside the agent span, then attach timing and token measurements to the same trace. That lets a logistics team compare two routing prompts without treating a flag service as a billing system. The trade-off is deliberate: a few stable attributes make analysis possible, while copying a full flag payload or prompt into every event makes telemetry noisy and expensive. The two-minute gap between evaluation and worker execution is enough to make a later lookup misleading, especially during a rollout where the cohort rule can change while queued jobs are still draining.

I build RAG and agent workflows in Python, even when the surrounding API is Node.js. I started by treating enabled as enough. It was not. When a delivery-planning agent took longer, I needed to know which instruction set it received, which tool loop it entered, and how many model tokens it consumed. A boolean could not answer that.

What should a feature-flag check prove?

A check should prove which decision governed one request, not merely what the flag says now. For each agent run, capture a normalized flag key, a small variant name, the evaluation reason if one exists, and a policy version that is safe to retain. Keep user identifiers out of these attributes unless there is a clear retention policy.

The data flow is short: the Node.js API evaluates the flag, starts or joins a trace, and passes the decision into the agent request. The worker records model-call latency, tool-call latency, retry count, and usage metadata under that trace. A cost calculation can happen later using a versioned rate table. Observability tells you what happened; it does not establish a universal price for every model or contract.

OpenTelemetry describes logs as a signal that can be associated with traces and spans. That makes a structured flag decision useful for events that do not fit neatly into a duration measurement, without forcing every detail into a metric label.

A small Python event model for a Node.js API

The following is framework-neutral. The Node.js route can produce the same JSON shape, while a Python worker validates it before the agent loop starts. There is no reason to copy the full request body into telemetry.

from dataclasses import dataclass
from time import monotonic
from typing import Any
import json


@dataclass(frozen=True)
class FlagDecision:
    key: str
    variant: str
    enabled: bool
    evaluation_version: str


def record_agent_run(decision: FlagDecision, run_agent) -> dict[str, Any]:
    started = monotonic()
    context = {
        "feature_flag.key": decision.key,
        "feature_flag.variant": decision.variant,
        "feature_flag.enabled": decision.enabled,
        "feature_flag.evaluation_version": decision.evaluation_version,
    }
    result = run_agent(context)
    event = {
        "event": "logistics.agent.run",
        "attributes": context,
        "latency_ms": round((monotonic() - started) * 1000, 2),
        "tool_calls": result["tool_calls"],
        "input_tokens": result["input_tokens"],
        "output_tokens": result["output_tokens"],
        "outcome": result["outcome"],
    }
    print(json.dumps(event, separators=(",", ":")))
    return event
Enter fullscreen mode Exit fullscreen mode

The important choice is the join key: the telemetry SDK carries trace and span context, while the business attributes stay explicit. evaluation_version prevents a later policy change from making old runs look as if they were evaluated under today’s rule. The timing brackets the agent loop, including tool calls and retries, rather than measuring only the flag lookup. The dict[str, Any] annotation in this example requires Python 3.9 or newer; on older runtimes, use Dict[str, Any] from typing or remove the annotation rather than changing the event contract.

A Node.js endpoint can return key, variant, enabled, and evaluation_version as ordinary JSON. The worker should reject missing or malformed values before invoking the model. A missing decision is an operational state, not a reason to silently assign the new treatment.

Which failure does this catch?

The common failure is attribution drift. The API evaluates a flag at 09:00, the worker starts at 09:02, and an analyst later joins the run to the current flag state. The dashboard assigns a slow run to the wrong variant. Persisting the evaluated decision with the trace avoids that temporal mismatch.

Another trap is a single field called cost. Store token counts, model identifier, rate-table version, and currency or unit convention separately. If usage fields differ across model endpoints, the ingestion layer can map them without rewriting historical traces.

Do not put the full prompt, warehouse address, or shipment notes into a high-cardinality metric label. Those values belong in controlled logs or an application data store, subject to retention rules. Metrics should answer “what is p95 latency by variant?” without turning every shipment into a new time series.

How do you know the telemetry is useful?

Start with an eval harness that replays representative routes and checks answer quality alongside operational measurements. A flag variant that lowers token use but increases invalid tool arguments is not a cost win. Keep the quality assertion and cost attribution in the same test record.

For production analysis, slice by stable dimensions that explain a decision: flag variant, evaluation version, model family, agent outcome, and perhaps route class. Compare distributions, not one impressive request. Latency has a long tail, and retries can make an average look calm while drivers wait on the slow cases.

I would keep the operational checklist in prose: make the flag decision immutable for the run, propagate trace context across the Node.js boundary and worker queue, measure the complete loop, record usage fields separately, version the rate table, and set retention limits before adding payload details. Test the disabled path, enabled path, stale evaluation version, and malformed decision. Those cases expose more than a screenshot of a green dashboard.

The implementation is small because the hard part is the contract. Once the event shape is stable, the backend can change and the agent can gain another tool without losing the question that matters: which decision caused this latency and cost?

Sources

Top comments (0)