Short answer: for fintech code-review extraction, put a token budget and a deadline around every document chunk, retrieve evidence per finding type, and merge validated JSON in a worker that records tenant ownership. Do not make one model request responsible for reading the entire change history.
This is the least complex shape that stays inspectable: parse, count, split, retrieve, extract, validate, and merge. A long pull request or audit attachment then becomes a set of bounded jobs instead of one request that can consume an unknown amount of context and leave an opaque timeout behind.
How can a fintech text-to-JSON worker keep long documents inside token limits?
Start with the output schema, not the document. For a code-review assistant, useful fields might be severity, file, line, finding, and evidence. Each field group should have a retrieval question. “Find a possible authorization bypass” is a better query than “summarize this repository.” A single document can contain the changed code, a ticket, a policy excerpt, and an old discussion; they should not all receive equal space in the extraction prompt.
Count before inference. A tokenizer-aware count is preferable to a word count, but the control flow is the same: reject or split an input that exceeds the model and job budgets before sending it. Split on file, function, heading, or paragraph boundaries first. Apply a hard ceiling second. Overlap can preserve a cross-boundary condition, yet it also duplicates evidence and increases tenant usage, so measure it instead of choosing a large overlap by habit. In a fintech review, that accounting detail has a practical consequence: if a single changed file is retrieved for five finding types, the worker must record five stage decisions even when the source chunk is reused, otherwise the tenant report cannot explain why the job consumed its allowance.
For each chunk, retain a stable document version, tenant ID, chunk ID, and source location. Embeddings are useful for a broad candidate pass; a rerank step can then compare a small candidate set with the finding-specific query. The final extraction call should see only the selected passages and the schema. Retrieval recall and JSON validity are different metrics. Keep them separate.
I’m not sure a universal chunk size exists. The right value depends on code density, comments, schema breadth, and whether a finding needs two adjacent functions. A small labeled fixture set settles that question: sweep sizes, overlap, candidate count, and deadline together, then keep the smallest context that preserves the required evidence.
Build the bounded worker before tuning retrieval
The example below is deliberately deterministic. It is runnable in a notebook, uses no credentials, and makes the accounting and merge rules testable before a model is added. In production, replace the lexical scorer with embeddings plus rerank and replace extract_findings with a schema-constrained model call. Keep the job identity, tenant attribution, and conflict behavior.
import json
import re
from dataclasses import dataclass
@dataclass(frozen=True)
class Chunk:
chunk_id: str
file_name: str
start_line: int
text: str
def split_text(file_name: str, text: str, lines_per_chunk: int = 8) -> list[Chunk]:
lines = text.splitlines()
return [
Chunk(
chunk_id=f"{file_name}:{start + 1}",
file_name=file_name,
start_line=start + 1,
text="\n".join(lines[start:start + lines_per_chunk]),
)
for start in range(0, len(lines), lines_per_chunk)
]
def approximate_tokens(text: str) -> int:
return max(1, len(re.findall(r"\w+|[^\w\s]", text)))
def retrieve(chunks: list[Chunk], query: str, limit: int = 2) -> list[Chunk]:
terms = set(re.findall(r"[a-z0-9_]+", query.lower()))
def score(chunk: Chunk) -> tuple[int, str]:
words = set(re.findall(r"[a-z0-9_]+", chunk.text.lower()))
return len(terms & words), chunk.chunk_id
return sorted(chunks, key=score, reverse=True)[:limit]
def extract_findings(chunk: Chunk) -> list[dict]:
findings = []
for line_number, line in enumerate(chunk.text.splitlines(), chunk.start_line):
if "TODO_SECURITY_REVIEW" in line:
findings.append({
"severity": "high",
"file": chunk.file_name,
"line": line_number,
"finding": "Review the security-sensitive change",
"evidence": line.strip(),
"evidence_chunk": chunk.chunk_id,
})
return findings
def merge_findings(results: list[dict]) -> list[dict]:
seen = set()
merged = []
for result in results:
identity = (result["file"], result["line"], result["finding"])
if identity not in seen:
seen.add(identity)
merged.append(result)
return merged
def review_document(tenant_id: str, job_id: str, files: dict[str, str]) -> dict:
chunks = [
chunk
for file_name, text in files.items()
for chunk in split_text(file_name, text)
]
queries = ["authorization access security", "input validation injection"]
selected = {
chunk.chunk_id: chunk
for query in queries
for chunk in retrieve(chunks, query)
}
input_tokens = sum(approximate_tokens(chunk.text) for chunk in selected.values())
findings = merge_findings([
finding
for chunk in selected.values()
for finding in extract_findings(chunk)
])
return {
"tenant_id": tenant_id,
"job_id": job_id,
"input_tokens": input_tokens,
"findings": findings,
}
files = {
"payments.py": """
def approve_payment(user, payment):
# TODO_SECURITY_REVIEW: authorization check belongs here
return process(payment)
""".strip()
}
print(json.dumps(review_document("tenant-acme", "review-042", files), indent=2))
The word-based counter above is a test double. A production worker should use the selected model’s tokenizer or a trusted counting endpoint, reserve room for instructions and output, and enforce a separate wall-clock deadline. The important part is that the count is recorded with the job. A timeout without the input-token count, candidate IDs, and stage name is barely an incident report.
Validation belongs at the chunk boundary. Reject unknown fields, enforce enum values for severity, and require evidence for every non-empty finding. A malformed response should be retried or quarantined as one chunk result. It should not force a replay of the entire document, and it should not reach the final review record merely because it was valid JSON.
Why per-tenant accounting changes the design
Per-tenant cost visibility is an architectural requirement, not a dashboard decoration. Carry the tenant ID through ingestion, retrieval, rerank, extraction, retries, and storage. Record prompt tokens, completion tokens, model identifier, latency, retry count, and outcome for each stage. If a shared queue reports only a daily total, a large customer can quietly consume the budget intended for smaller tenants.
Use two budgets. The first is a hard job allowance: maximum selected tokens, maximum chunks, and maximum elapsed time. The second is an alert threshold that leaves room for retries and documents that contain unusually dense code. When a tenant crosses the alert threshold, reduce candidate count or move the job to a batch lane according to a documented policy. Preserve enough evidence to explain that decision.
This also exposes a common accounting mistake: charging only successful extraction. Failed retrieval, validation retries, and abandoned requests still use resources. Attribute them to the same tenant and document version. For a fintech review workflow, an empty result and a failed result must remain distinct; otherwise a cost report can look healthy while recall is collapsing.
Three words: measure the stages.
Make it boring.
What should happen when extraction times out or evidence conflicts?
Interactive requests should enqueue large documents and return a job identity. A worker can retry a single chunk, while the caller polls or receives a completion event. Make the retry replace the result for the same (tenant_id, document_version, chunk_id, stage) key. This prevents a late response from being appended as a second finding.
Retry semantics need a deliberate boundary. RFC 9110 describes idempotency and method semantics for HTTP; the application still has to provide an idempotency key and make its storage operation conditional. A retry is safe only when the worker can identify the same logical operation. Backoff should respond to rate limiting, while a token-budget rejection should be routed to splitting or a batch policy rather than retried unchanged.
Conflicts require evidence, too. If two chunks produce different severities for the same line, retain both candidates and run a narrow adjudication pass over the relevant excerpts. Never let “last response wins” decide a compliance-relevant finding. If the source is missing or contradictory, return an explicit review state that a human can inspect.
The catch is latency. Per-field retrieval, reranking, validation, and conflict passes add work, so this design is unsuitable when the product promises an immediate answer for every document. Use a smaller schema and a direct bounded request for short diffs; use the queue for long audits, multi-file changes, and any workflow where evidence traceability matters more than instant display.
Production checks for a notebook-to-prod handoff
Before rollout, build fixtures from the shapes that matter: a finding split across functions, repeated identifiers, generated files, a large diff, missing evidence, and a tenant with a strict allowance. Pin expected fields and acceptable source locations. Evaluate retrieval recall before measuring end-to-end JSON accuracy, then add timeout rate, p95 stage latency, duplicate rate, and tokens per tenant. I would also replay one document version through the worker after changing only the candidate limit. If the output changes, the fixture should show whether the new passage added evidence or merely displaced a better passage; without that comparison, a lower token count can look like an improvement while a high-severity finding has quietly lost its source line. This is where a notebook-to-prod workflow earns its keep: the experiment is small enough to inspect, but the same identifiers and metrics survive the move into the queue.
| Workload shape | First choice | Watch closely |
|---|---|---|
| Short, known-size diff | One bounded request | Output tokens and schema validity |
| Long multi-file audit | Chunk jobs with retrieval | Evidence recall and duplicate findings |
| Strict tenant allowance | Queue with per-tenant budgets | Retries and shared-queue fairness |
Keep sensitive source text out of ordinary logs. Store references and hashes where possible, cap concurrency per tenant, and make cancellation visible to the worker. Alert on the stage that exceeded its budget: “chunk 17 exceeded extraction time” is actionable; “the AI request timed out” is not.
The decision rule is plain. Choose bounded chunk jobs when documents are long, evidence must be cited, or tenant usage must be explained. Choose a direct request when the diff is small and its token count is known. In both cases, schema validation, evidence retention, and per-tenant metrics are part of the extraction system itself.
Top comments (0)