Short answer: treat application logs as an append-only experiment ledger, not a diary. For a small Node.js SaaS comparing an AI tutoring experiment across school-tenant cohorts, emit one JSON event per model attempt with opaque request and user references, source-reported usage units, assignment version, outcome, and a deliberate level; redact PII before the logger receives it. Recompute cost later against a versioned rate table. That preserves the evidence needed for attribution without turning logs into a second student-record system.
The key decision is event grain. One browser request can trigger retrieval, a model call, an eval rejection, and a retry. Charging the request once hides work; adding usage again on the final response double-counts it. I want the production stream to answer the same question as the Python eval notebook: for each tenant and cohort, what usage did completed attempts consume, and what outcome did the learner see?
Make those two grains explicit.
How can a small Node.js SaaS evaluate structured JSON logging across cohorts?
The data flow is short enough to sketch in prose. Node.js creates a request_id at the trusted ingress, reads a previously assigned experiment cohort, and emits an attempt event after each model call. The event carries opaque tenant_ref and user_ref values, but no prompt, answer, name, email, authorization header, or raw body. A separate attribution job reads those events, joins a versioned rate card, and produces tenant-cohort totals. The terminal request event records the learner-visible outcome; it does not repeat attempt usage.
Here is a runnable Python version of the attribution step. The sample events are illustrative fixtures, not benchmark results. Keeping this beside the eval harness makes notebook-to-prod review concrete: a schema change must still produce the expected cohort totals before deployment.
from collections import defaultdict
from decimal import Decimal
events = [
{
"schema_version": 1,
"event": "model_attempt_completed",
"level": "info",
"request_id": "req_a91f",
"attempt_id": "att_01",
"tenant_ref": "tenant_7c2a",
"user_ref": "user_18d4",
"cohort": "baseline",
"assignment_version": "exp_math_03",
"model_class": "tutor_standard",
"input_units": 620,
"output_units": 140,
"outcome": "accepted",
},
{
"schema_version": 1,
"event": "model_attempt_completed",
"level": "info",
"request_id": "req_b04e",
"attempt_id": "att_01",
"tenant_ref": "tenant_7c2a",
"user_ref": "user_b118",
"cohort": "retrieval_a",
"assignment_version": "exp_math_03",
"model_class": "tutor_standard",
"input_units": 810,
"output_units": 175,
"outcome": "accepted",
},
]
rates = {
"tutor_standard": {
"input_per_unit": Decimal("0.000001"),
"output_per_unit": Decimal("0.000002"),
}
}
def attribute_cost(rows, rate_table):
totals = defaultdict(lambda: {"attempts": 0, "cost": Decimal("0")})
seen_attempts = set()
for row in rows:
attempt_key = (row["request_id"], row["attempt_id"])
if attempt_key in seen_attempts:
raise ValueError(f"duplicate attempt: {attempt_key}")
seen_attempts.add(attempt_key)
rate = rate_table[row["model_class"]]
cost = (
Decimal(row["input_units"]) * rate["input_per_unit"]
+ Decimal(row["output_units"]) * rate["output_per_unit"]
)
group = (row["tenant_ref"], row["cohort"], row["assignment_version"])
totals[group]["attempts"] += 1
totals[group]["cost"] += cost
return totals
cohort_totals = attribute_cost(events, rates)
assert cohort_totals[("tenant_7c2a", "baseline", "exp_math_03")] == {
"attempts": 1,
"cost": Decimal("0.000900"),
}
The rate values exist only to make the fixture executable. Production rates need their own effective date and review path; don't freeze currency inside an application event. Usage units are evidence from the attempted operation, while money is an interpretation that can change with a contract, model class, or rounding rule. Keeping them apart also lets an analyst replay old events after correcting an attribution rule without rewriting the log archive.
This is the first hard boundary: attempt events carry usage; terminal request events carry user-visible outcomes. The request ID joins the story, but it isn't the accounting grain.
Implement the event contract around invariants
An application logger should receive a small allowlisted dictionary. Passing a request object and asking a formatter to clean it later is risky because serializers, exception handlers, and fallback sinks may observe the original object. OWASP advises excluding or masking data such as access tokens, passwords, connection strings, and sensitive personal data. In an edtech application, I would extend that deny category to student names, email addresses, free-form questions, generated answers, and uploaded lesson content. The allowlist is stronger than a growing collection of regexes because an unrecognized field stays out by default.
The minimum useful attempt event has a timestamp, stable event name, schema version, level, request ID, attempt ID, opaque tenant and user references, cohort, assignment version, operation, model class, outcome, latency, and raw usage dimensions. Each field earns its place by supporting a join, an aggregation, or an operational decision.
| Contract area | Required fields | Invariant |
|---|---|---|
| Identity |
request_id, attempt_id
|
The pair is unique per attempt |
| Assignment |
tenant_ref, cohort, assignment_version
|
Values stay fixed throughout one request |
| Usage |
model_class, input_units, output_units
|
Appears on attempt events only |
| Result |
outcome, latency_ms
|
Describes this attempt, not a later retry |
| Control |
event, level, schema_version
|
Uses typed, reviewable values |
assignment_version matters because a label like retrieval_a can silently acquire new meaning after experiment logic changes. attempt_id matters because retries share a request ID. schema_version makes a breaking change visible instead of leaving an analyst to infer it from missing columns.
No prose parsing.
Use event="model_attempt_completed" and outcome="accepted", for example, rather than hiding both facts in message="Tutor call was fine". A human-readable message can help during an incident, but the cost query should group typed fields. Keep trace identifiers separate from request IDs when distributed tracing exists: a request ID names the application interaction, while trace context can cross service boundaries. Conflating them works in a single process and becomes ambiguous as soon as a queue or background evaluator enters the path.
User IDs deserve the same precision. user_ref should be an application-generated opaque reference, not an email address and not a hash of an email address. A deterministic hash still supports correlation and may be guessable when the original value comes from a small, known domain. Store the mapping in the system that already governs identity, apply its access and deletion controls there, and let the log retain only the reference for the period the experiment actually needs.
The tempting design is one giant context object. It feels flexible in a notebook, then becomes impossible to review: cardinality grows, field meanings drift, and a prompt can arrive under a tenant-defined key nobody anticipated. A flat, versioned measurement contract is less clever. Good.
Operate log levels as action contracts
Levels aren't a measure of how surprising a line of code felt. debug is disposable diagnostic detail that can be disabled or sampled. info records expected lifecycle facts such as an assignment or completed attempt. warn marks a recoverable condition that deserves attention, and error records an operation that failed and requires an owner, retry policy, or alert decision.
Log a failure once at the boundary that owns its outcome. Repeating the same exception in a model adapter, service layer, and route handler inflates cohort failure counts and makes alerts noisy. Expected input rejection should follow the application's response policy rather than automatically becoming error; the operational question is whether the system failed to do its job, not whether a branch returned early.
Keep stack traces away from the accounting event. They can reveal arguments, paths, query strings, or serialized state, so send a sanitized error class and internal error code with the same request ID to a separately controlled diagnostic event. I'm not sure one universal redaction expression can cover tenant-defined school data — the field names and free-form values vary too much. Resolve that uncertainty with representative fixtures from the actual data classification, then fail the contract test when a forbidden key or value pattern appears.
Govern user IDs and PII before serialization
Start with a constructor that copies known-safe values into a new object. Validate types and bounds there. Then serialize. This order means a raw authorization header or student response never enters the logger's buffer, child context, transport fan-out, or exception fallback. Sink-side masking remains useful as defense in depth, but it cannot prove that an earlier sink did not receive the unredacted event.
Test the negative space — fields that must never appear — as aggressively as required fields. A useful fixture set includes an accepted first attempt, a retry with two distinct attempt IDs, an input rejection, and an event containing every forbidden key the application knows about. Parse each emitted line as JSON; reject duplicate attempt keys; require the current schema version; reject direct identifiers and content fields at any nesting depth; and confirm that a terminal event contains no usage dimensions. That last assertion prevents double attribution even when both events are individually valid.
Cost-aware logging also needs a field budget. High-cardinality identifiers are justified when they enable a controlled investigation or cohort join, but putting entire prompts into message adds storage, privacy exposure, and token-sized payloads without improving the aggregate. Sample debug events, set retention by event class, and restrict access to identity references. The exact retention window depends on school agreements and applicable policy, so the engineering contract should name an owner and configuration point rather than inventing a universal number.
There is a subtle eval failure worth guarding against. If cohort assignment is looked up after a model attempt, a concurrent experiment update can label the usage with a different variant from the one that shaped the request. Capture cohort and assignment version at request start, carry them unchanged to every attempt, and assert their equality during aggregation. The logger isn't the experiment allocator; it records the allocator's decision.
Carry the contract through the deployment pipeline
This ledger is not suitable for full conversation replay, safeguarding review, legal discovery, or detailed learning analytics. Those jobs need a purpose-built content store with stricter access, deletion, and audit controls. Link to such a record only when policy permits, and don't copy its payload into observability data. Likewise, stick with metrics rather than per-user logs when the team cannot operate access reviews, retention, schema ownership, and deletion handling for high-cardinality references.
The catch is that privacy and attribution pull in opposite directions. Remove attempt identity and retries become hard to count; retain personal content and an operational store acquires obligations it was never designed to carry. Opaque references, narrow event grains, and short purpose-based retention don't eliminate that trade-off. They make it reviewable.
Before release, run the JSON fixtures through the same Python evaluator used by the notebook, replay one request with two attempts, and confirm it yields two attempt charges but one terminal outcome. Verify that changing a rate table changes the computed total without changing stored events. Inspect rejected events for PII, check that every error has an owner-facing action, confirm debug sampling cannot remove accounting events, and have someone outside the feature branch explain the grain back to you. If that explanation is fuzzy, the dashboard will be worse.
Ship the contract first.
Top comments (0)