Short answer: for an edtech code-review assistant, build a structured answer with citations, validate it against a JSON Schema, and keep the contract independent of the model provider. Store the document identifier, chunk version, quoted span, retrieval score, and validation result. Expire the candidate set and verbose telemetry on a schedule. This preserves an auditor's path from finding to source without turning every prompt, trace label, and token into a permanent liability.
The important calculation is volume multiplied by retention. Imagine 200 student changes per day, 12 findings per change, and 4 KB for each normalized finding plus its evidence pointer. That is 9.6 MB of answer records per day, or 288 MB for 30 days before indexes and operational logs. The arithmetic is intentionally plain: the dominant term is often how many artifacts are retained and for how long, not the price of one generation request.
That is the first lever.
For this workflow, provider portability means more than swapping an HTTP base URL. A retrieval adapter should return a stable evidence shape; a generation adapter should accept a stable prompt and schema; the application should reject incomplete or unsupported evidence before a finding reaches a teacher.
How should a Node.js answer use JSON Schema and citations?
A code-review finding should point to the material that justified it, not reproduce an entire search session. I retain a document ID, an immutable chunk or source revision, a character range or quoted span, the retrieval and reranking decisions, and the schema-validation result. I do not need to retain every low-scoring candidate forever.
That distinction matters in an ask-your-docs system for coursework. A policy document may explain that a dependency must be pinned. A repository excerpt may show the unpinned declaration. The final finding needs both references, with enough text to let a reviewer verify the claim. It does not need the full contents of every unrelated document returned during retrieval.
A practical record can be expressed as JSON Schema so that the Node.js application validates the same contract regardless of which search or generation adapter is behind it. The schema should require an empty findings array when the evidence is insufficient. “No finding” is a valid result; a confident sentence without a source is not.
The evidence is the product.
Retention is a deliberate loss function. Keeping full prompts helps reproduce a disputed answer, but prompts can contain student code and private course material. Keeping only hashes and evidence pointers protects less data, but it makes some investigations harder. I would keep full evidence excerpts for the review window, a compact decision record for the audit window, and aggregate counts for longer trend analysis. The exact windows belong to the institution's policy; the shape of the decision should be explicit.
How does the pipeline keep search and generation interchangeable?
Use three boundaries. The ingestion boundary produces versioned chunks with stable source IDs. The retrieval boundary accepts a query and returns candidates with scores and source metadata. The answer boundary receives only the selected evidence and returns a schema-checked object. Each boundary can be tested with fixtures, which is more useful for portability than testing a particular provider's response prose.
Reranking is useful where the first retrieval pass returns several plausible passages, but its score is a selection signal, not evidence. The application should preserve the selected passage and its source location. A score can help diagnose a missed document; it cannot justify a finding by itself.
The same rule applies to prompt instructions. Ask the model to use only supplied evidence, distinguish a direct violation from a suggestion, and emit source references for every finding. Then enforce those requirements in code. Prompt wording is a control; validation is the boundary.
Here is the kind of request an adapter can send to a chat-completions-compatible gateway. The endpoint is intentionally pseudonymous, and the application-owned schema is the portable part. The gateway adapter may translate the response-format fields for a different provider, while the rest of the review service remains unchanged.
curl https://llm-gateway.example/v1/chat/completions \\
-H 'content-type: application/json' \\
-d '{
"messages": [
{"role": "system", "content": "Return only JSON matching the supplied schema. Use only the evidence."},
{"role": "user", "content": "Review the change against the supplied course policy. Evidence: ..."}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "review_result",
"strict": true,
"schema": {
"type": "object",
"required": ["findings"],
"properties": {
"findings": {
"type": "array",
"items": {
"type": "object",
"required": ["severity", "message", "evidence"],
"properties": {
"severity": {"enum": ["error", "warning", "note"]},
"message": {"type": "string"},
"evidence": {
"type": "array",
"items": {
"type": "object",
"required": ["source_id", "quote"],
"properties": {
"source_id": {"type": "string"},
"quote": {"type": "string"}
}
}
}
}
}
}
}
}
}
}
}'
The response still needs application-side parsing and validation. Reject malformed JSON, unknown severity values, missing evidence, and quotes that do not match the retrieved source revision. A retry can be appropriate for a transient protocol failure, but it should not silently convert missing evidence into an accepted finding.
Validate twice.
Where does retention change the operational design?
Telemetry should answer bounded questions: which retrieval stage lost the relevant policy, how often validation failed, and how many findings had no usable evidence? It does not need the full prompt in every span. Log a request identifier, schema version, source revision, candidate count, selected count, latency buckets, and outcome. Avoid putting raw student code into labels. Labels become dimensions in a metrics system; a course ID, repository ID, or document ID can create a cardinality problem quickly.
I treat every log line as bytes stored and every label as a future index. A useful event can therefore contain a short reason code such as evidence_missing or schema_rejected, while the protected review record holds the detail. This separation keeps dashboards cheap to aggregate and investigations possible through an access-controlled lookup.
The retention table below is a starting policy, not a universal number.
| Artifact | Useful during | Keep after expiry | Why |
|---|---|---|---|
| Retrieved candidate text | Active review and dispute window | Source ID and revision | Reconstruct selection without duplicating documents |
| Final evidence excerpt | Review and audit window | Hash, source pointer, outcome | Verify the claim while limiting copied code |
| Full prompt and response | Short debugging window | Request ID and schema result | Debug protocol issues without long-lived student data |
| Metrics and reason codes | Trend analysis | Aggregated counts | Measure quality without high-cardinality payloads |
Retention also changes retry behavior. A retry should reuse the same evidence snapshot when investigating a disputed finding; otherwise a moving index can produce a different answer and obscure the original failure. Store the snapshot identifier, not an unbounded copy of every retrieval response.
What should portability mean in a real decision?
Portability is testable when the application owns the contract. Build a conformance suite with fixed document chunks and expected classes of output: a finding with two citations, no finding when the policy is absent, rejection when a quote is not present, and a recoverable protocol error. Run each adapter against those fixtures. Compare semantic outcomes and validation results, not prose similarity.
There are trade-offs. A provider-native schema feature may reduce parsing work but increase adapter code. A generic JSON mode may work across more gateways but still permit unsupported values or weak evidence. A managed reranker may improve candidate ordering while adding a dependency and another retention boundary. A self-hosted index offers control over records and deletion, but shifts upgrades, capacity planning, and on-call work to the team. None of these is a universal winner.
The decision rule I use is narrow: choose the smallest boundary that can prove why a finding exists, reject outputs the boundary cannot prove, and retain only the artifacts needed for the next legitimate question. For an education product, that makes provider substitution a controlled test rather than a rewrite, and it keeps the observability bill attached to decisions the team can actually act on.
Further reading
- JSON Schema specification: https://json-schema.org/specification
- Node.js HTTP and fetch documentation: https://nodejs.org/api/globals.html#fetch
- Cohere Rerank documentation: https://docs.cohere.com/docs/rerank-overview
- Prompt Engineering Guide: https://www.promptingguide.ai
Top comments (0)