Short answer: use backend custom metrics as the cost ledger for an edtech experiment, use feature flag statistics to explain assignment and exposure, and join the two only after preserving their different denominators. A flag dashboard can tell you which tenant cohort was offered a variant. It can't, by itself, attribute model tokens, retries, or completed tutoring sessions to that cohort.
This split matters when a team compares an AI tutoring experiment across school and district tenants. The backend sees the billable work and the learning workflow outcome; the flag system sees rollout decisions. Put those records into one dashboard, but don't pretend they are one measurement stream.
Keep the first version boring.
How should a product analytics dashboard combine backend custom metrics and feature flag stats?
Treat the dashboard as a projection over two append-only event families. An assignment or exposure event answers, "Which variant could this tenant use?" A backend operation event answers, "What work actually ran, what did it cost, and what outcome followed?" Both need an experiment ID, variant, tenant cohort, and a time bucket. A request ID can connect one exposure to one backend operation when that relationship exists, but cohort-level aggregation must still work when a learner sees a flag and never starts the AI workflow.
That last case is the trap. Suppose 1,000 eligible sessions receive a guided-tutor variant, 760 open the tutor, 702 trigger a model request, and 511 reach the defined completion event. A flag-only chart may use 1,000 as its denominator. A backend-only chart may start at 702. Neither denominator is wrong; each answers a different question. Mixing them without labels produces a conversion rate that looks precise while hiding where 489 sessions disappeared.
For cost attribution, the backend record should carry the smallest useful business context: experiment_id, variant, tenant_cohort, a pseudonymous request identifier, the measured usage, the attributed cost, and an outcome from a small controlled vocabulary. Don't attach prompt text, learner names, email addresses, or arbitrary metadata just because the analytics store accepts them. Article 17 of the GDPR describes a right to erasure under specified conditions, so data deletion cannot be an afterthought when identifiers can point back to a person. This isn't legal advice; it is a strong reason to make the event contract deliberately sparse.
The data flow is straightforward: the assignment path emits exposure counts, the application backend emits operation events after usage and outcome are known, a scheduled aggregation groups both streams by experiment, variant, cohort, and time window, and the dashboard renders the measures side by side. The join belongs in the aggregation layer. Doing it in browser code makes definitions hard to test and invites every chart to invent a slightly different metric.
Build the cost ledger before polishing the dashboard
Here is a runnable, standard-library Python example. The rows are deliberately small and illustrative; the dollar values are sample event data, not vendor prices or a benchmark. In production, the same aggregation should run against durable event storage rather than an in-memory list.
from collections import defaultdict
from decimal import Decimal
backend_events = [
{
"request_id": "req-101",
"experiment_id": "tutor-answer-v3",
"tenant_cohort": "district",
"variant": "guided",
"token_cost_usd": "0.0180",
"outcome": "completed",
},
{
"request_id": "req-102",
"experiment_id": "tutor-answer-v3",
"tenant_cohort": "district",
"variant": "guided",
"token_cost_usd": "0.0210",
"outcome": "abandoned",
},
{
"request_id": "req-103",
"experiment_id": "tutor-answer-v3",
"tenant_cohort": "district",
"variant": "concise",
"token_cost_usd": "0.0110",
"outcome": "completed",
},
{
"request_id": "req-104",
"experiment_id": "tutor-answer-v3",
"tenant_cohort": "school",
"variant": "guided",
"token_cost_usd": "0.0160",
"outcome": "completed",
},
{
"request_id": "req-105",
"experiment_id": "tutor-answer-v3",
"tenant_cohort": "school",
"variant": "concise",
"token_cost_usd": "0.0090",
"outcome": "abandoned",
},
]
# These counts come from the assignment/exposure stream, not backend requests.
flag_exposures = {
("district", "guided"): 3,
("district", "concise"): 2,
("school", "guided"): 2,
("school", "concise"): 2,
}
ledger = defaultdict(
lambda: {
"backend_requests": 0,
"completed": 0,
"cost_usd": Decimal("0"),
}
)
for event in backend_events:
key = (event["tenant_cohort"], event["variant"])
row = ledger[key]
row["backend_requests"] += 1
row["completed"] += event["outcome"] == "completed"
row["cost_usd"] += Decimal(event["token_cost_usd"])
for key in sorted(flag_exposures):
cohort, variant = key
row = ledger[key]
exposures = flag_exposures[key]
cost_per_completion = (
row["cost_usd"] / row["completed"]
if row["completed"]
else None
)
print(
{
"tenant_cohort": cohort,
"variant": variant,
"flag_exposures": exposures,
"backend_requests": row["backend_requests"],
"completed": row["completed"],
"total_cost_usd": str(row["cost_usd"]),
"cost_per_completion_usd": (
str(cost_per_completion) if cost_per_completion else None
),
}
)
Decimal is intentional because binary floating-point rounding is a distraction in a cost ledger. More important, the code never divides cost by flag exposures and calls the result "cost per completion." It reports exposure, backend request, completion, total cost, and cost per completion as separate measures. That makes the funnel inspectable.
Read the tiny district rows as a rehearsal for the real decision. The guided variant has three exposures but only two backend requests, so the dashboard must leave a visible gap between those stages; filling that gap with zeros would imply that an exposed session incurred work when it did not. One of the two requests completed, which makes the sample's total backend cost the numerator for one completion. The concise variant also has an exposure-to-request gap, but its single recorded request completed. Those observations aren't evidence that one prompt is better: five backend events are fixture data, evaluation quality is absent, and the assignment counts are too small for an experiment claim. They do prove that the schema can express the question without changing denominators halfway through the calculation. Run this fixture in a test, assert every displayed field, and then replace the list with the production event reader. The chart comes last.
The sample also keeps tenants in cohorts rather than turning each tenant into a dashboard series. Per-tenant drill-down can be useful for support or contract analysis, but it raises access-control, cardinality, and deletion questions. Start with the cohort needed for the experiment decision, then add narrower dimensions only when somebody can name the decision they enable.
One more rule: define completion outside the prompt. A model response with a successful application status is not necessarily a completed tutoring session. The business event might be "learner submits the next exercise" or "teacher accepts generated feedback." Whichever event the product team chooses, version that definition alongside the experiment. Prompt changes are frequent; outcome definitions should not drift silently with them.
The two data sources fail in different ways
Feature flag statistics are attractive because assignment and variant breakdowns arrive with little application code. The catch is that rollout telemetry is not a financial subledger. It may count evaluation, assignment, or exposure depending on the event contract, while token usage and retry work happen later in the backend. Stick with flag statistics alone when the decision is limited to rollout health, audience allocation, or whether clients are receiving the intended variant. They are not suitable as the sole source for cost per successful AI interaction.
Backend custom metrics have the opposite strength and weakness. They can observe model calls, token usage, cache behavior, retries, and domain outcomes at the point where those facts become known. They also demand an explicit schema, deduplication policy, retention policy, and tested aggregation. A dashboard built only from backend requests misses eligible sessions that never invoked the backend, so it can't explain the full exposure-to-action funnel without the flag-side denominator.
| Decision | Primary evidence | Why | Main limitation |
|---|---|---|---|
| Did rollout allocation match the plan? | Flag assignment or exposure counts | The decision occurs in the flag path | It does not measure downstream compute work |
| Which cohort incurred AI workload? | Backend operation events | The backend observes usage after execution | It excludes exposed sessions with no request |
| What did one completion cost? | Backend cost plus a versioned outcome | Numerator and outcome share an auditable event path | Late outcomes require reconciliation |
| Where did the funnel lose sessions? | Both streams, shown as separate stages | Each stream owns a distinct denominator | Identity and time-window rules must align |
Retries deserve special treatment. If request req-101 is retried twice, three raw operation attempts may be operationally useful while only one logical tutoring interaction belongs in product analytics. Keep an attempt identifier for reliability analysis and an idempotency or logical request identifier for product counting. Then specify whether cost includes every attempt. Usually it should, because the spend happened, even when the user-facing interaction is counted once.
Late events create a quieter failure mode. A tutoring session can start near midnight, incur model cost immediately, and complete in the next reporting window. If a chart attributes cost by request time but completion by completion time, a daily cost-per-completion ratio will wobble at the boundary. Pick a cohorting clock, publish it in the metric definition, and allow a reconciliation window before treating the number as final. I'm not sure one window length fits every learning workflow; observed completion latency should settle that choice.
Time is a dimension, too.
Then there is missingness. Don't convert an absent outcome to abandoned during ingestion. "No outcome received yet," "explicitly abandoned," and "outcome event failed validation" are different states, with different fixes. A compact dead-letter count beside the business chart is often more valuable than another colorful segmentation control.
Make the experiment decision reproducible
The dashboard should answer a prewritten decision rule, not invite a tour through charts. For this edtech scenario, a useful rule might compare cost per completed tutoring session between guided and concise within each tenant cohort, while also checking the exposure-to-request and request-to-completion stages. The team can then see a variant that improves completion but costs more, a variant that looks efficient only because many exposures never reach the backend, or a cohort whose behavior differs enough to reject a global rollout decision.
Cost alone is insufficient.
A cheaper completion may come with a lower evaluation score, a worse teacher-review rate, or a prompt that fails a safety test. Keep offline evaluation results versioned by prompt and model configuration, but don't stuff every evaluator trace into the product dashboard. The dashboard needs a small release gate such as "evaluation suite passed for configuration X" and a linkable internal record of the detailed run. This preserves the notebook-to-production chain: the same configuration identifier used during evaluation appears in backend events after deployment.
Before release, test the aggregation with fixtures for duplicate requests, zero completions, late outcomes, unknown variants, and a tenant moving between cohorts. Test authorization separately: a district operator must not gain visibility into another district through a filter parameter. Also test deletion. If the event design retains an identifier that falls within an erasure request, the deletion path has to reach raw events, derived tables, and caches according to the organization's applicable policy.
Don't let the dashboard query raw high-cardinality events on every page load. Produce bounded aggregates for the common cohort comparison, retain a controlled drill-down path for debugging, and put freshness and last-successful-aggregation time next to the results. A stale number with an honest timestamp is actionable. A stale number presented as live isn't.
The operational checklist is short enough to keep in prose. On every deployment, verify that experiment and configuration IDs are present, unknown outcome counts remain visible, and exposure totals reconcile with the flag stream for the same time window. On a schedule, inspect duplicate logical request rates, late-event volume, and deletion coverage. Before making the rollout call, freeze the metric definition and export the exact cohort window, configuration version, and aggregation version used for the decision. Re-running that tuple should reproduce the table.
References
- GDPR Article 17, "Right to erasure ('right to be forgotten')": https://gdpr-info.eu/art-17-gdpr/
Further reading
- GDPR Article 17, "Right to erasure ('right to be forgotten')": https://gdpr-info.eu/art-17-gdpr/
Top comments (0)