A legal contract review workspace should fail toward a smaller, cited answer, not a confident guess: keep a lexical search path, a vector path, and a human-review state, then expose which path produced each citation. The deciding constraint is grounding. A fast answer that cannot point to the exact clause is a failed answer.
Short answer: use a two-stage retrieval architecture with explicit fallbacks, preserve page and clause coordinates in every chunk, and abstain when evidence does not meet a citation threshold.
Grounding wins.
The concrete job here is watching a web page for changes and alerting a reviewer about a contract clause or policy page. That sounds like a diff problem until the first layout redesign turns every line into noise. Retrieval must identify the same legal provision across HTML versions, then let a reviewer inspect the source span that triggered the alert.
What should a fallback prove before an alert is trusted?
Start with an evidence contract. Every candidate passage carries a document ID, source URL, retrieval timestamp, page number when available, clause heading, character offsets, and a content hash. The generator receives those fields alongside text. The UI can then render a citation that survives re-ranking and makes an audit trail possible.
A useful decision rule is deliberately boring:
- Run lexical retrieval against normalized clause text and headings.
- Run vector retrieval only when the query has enough semantic content and the index is fresh.
- Merge and re-rank candidates, deduplicating by clause ID rather than by similar wording.
- Require two independent signals, or a high lexical score plus a direct quote, before creating an alert.
- If the threshold is missed, return
needs_reviewwith the best candidates and no generated legal conclusion.
The RAG paper describes retrieval as a way to connect generation to an external memory. In a contract workspace, that memory is not an abstract embedding store; it is versioned source material. The fallback is successful only if it keeps provenance intact.
How can Node.js keep lexical, vector, and human fallbacks consistent?
Use one result shape at the boundary. The underlying engines can differ, but the rest of the application should not branch on vendor-specific response fields. This TypeScript sketch keeps the policy visible and makes an abstention a first-class outcome.
type Evidence = {
clauseId: string;
text: string;
sourceUrl: string;
hash: string;
page?: number;
heading?: string;
};
type Hit = Evidence & {
lexical: number;
semantic: number;
source: "lexical" | "vector";
};
type Retrieval =
| { status: "grounded"; hits: Hit[] }
| { status: "needs_review"; hits: Hit[]; reason: string };
interface Searcher {
search(query: string, limit: number): Promise<Hit[]>;
}
export async function retrieve(
query: string,
lexical: Searcher,
vector: Searcher | null,
): Promise<Retrieval> {
const lexicalHits = await lexical.search(query, 8);
const semanticHits = vector ? await vector.search(query, 8) : [];
const merged = [...lexicalHits, ...semanticHits]
.sort((a, b) => (b.lexical + b.semantic) - (a.lexical + a.semantic))
.filter((hit, index, all) =>
all.findIndex((other) => other.clauseId === hit.clauseId) === index,
)
.slice(0, 5);
const top = merged[0];
const hasQuote = Boolean(top?.text && top.text.length >= 40);
const hasTwoSignals = Boolean(top && top.lexical > 0 && top.semantic > 0);
if (!top || (!hasQuote && !hasTwoSignals)) {
return {
status: "needs_review",
hits: merged,
reason: "Evidence did not meet the citation policy",
};
}
return { status: "grounded", hits: merged };
}
The exact score cutoffs belong in evaluation configuration, not in a blog post or a prompt. Tune them against a labeled set of clause changes. Your mileage may vary: a short indemnity clause and a long privacy policy need different minimum quote lengths, even when they share a vocabulary.
Where do retrieval fallbacks fail in contract change detection?
The common failure is not a missing model. It is a broken identity map. If a parser keys chunks by character offset, a one-line insertion shifts every later chunk and produces a false alert. Imagine a page whose termination clause starts at offset 18,400 on Monday; a cookie-banner sentence inserted near the top moves it to 18,447 on Tuesday, so an offset-based diff reports a change even when the clause text is byte-for-byte identical. Stable clause IDs should come from heading paths and normalized anchors; when those are unavailable, map old and new passages with a similarity step and mark the mapping as uncertain. Keep both the proposed mapping and its confidence so a reviewer can see why an alert was grouped.
Another trap is embedding drift. Re-indexing with a new model can change nearest neighbors without any source edit. Store the embedding model ID, index build timestamp, and chunking configuration with each retrieval run. During a rebuild, compare alerts from the old and new index before switching traffic. A lexical path remains valuable because exact terms such as “governing law” and “termination for convenience” are often the strongest signal.
Parsing errors deserve their own state. An empty extraction, a truncated PDF, or a page that requires authentication should produce source_unavailable, not a zero-result answer that looks like “no change.” The alert should include the last successful hash and ask for review. Three words: preserve the last truth.
For a web page watcher, normalize boilerplate, navigation, and timestamps before diffing. Keep the raw response for audit, but retrieve from the cleaned representation. Record HTTP validators such as ETag and Last-Modified when available; they reduce needless parsing, while the content hash remains the final identity check.
Which implementation trade-offs matter more than the model name?
A small team can combine PostgreSQL full-text search with a vector extension, Elasticsearch with a dense-vector field, or a hosted vector database. These are evidence about implementation styles, not rankings. PostgreSQL keeps transactional metadata beside clauses; Elasticsearch offers mature text analysis and distributed indexing; a hosted vector database can reduce operational work but adds a network and policy boundary. None removes the need for clause IDs, freshness checks, or abstention.
The catch is operational. A vector-only design is not suitable when exact legal terms, citations, or newly added clauses dominate the query. A lexical-only design is a poor fit for paraphrased obligations across templates. Stick with a single lexical index when the corpus is small, terms are stable, and a reviewer can inspect every hit; add semantic retrieval when labeled evaluation shows it recovers misses without increasing unsupported alerts.
Measure before copying the architecture: citation precision, citation recall, unsupported-answer rate, stale-index rate, p95 retrieval latency, and cost per reviewed document. Split the test set by document type and by edit pattern, including heading changes, inserted paragraphs, and reordered lists. A single aggregate score hides the failure that legal teams notice first: a plausible alert attached to the wrong clause.
How should the workspace expose uncertainty to reviewers?
Make the fallback visible in the event record. Store retrieval_path, candidate scores, source hashes, and the policy decision next to the generated summary. Show the changed quote, its before-and-after context, and a link that resolves to the exact source version. A reviewer should be able to reject an alert without deleting the evidence that caused it.
Retries need boundaries. Retry transient fetches with capped exponential backoff, but do not retry a deterministic parse failure forever. Queue re-indexing separately from alert generation so a delayed index cannot silently suppress a change. Emit metrics for each transition: fetched, normalized, indexed, retrieved, grounded, and needs review.
I would ship the lexical path first, then add vectors behind the same interface and compare decisions in shadow mode for one release cycle. That keeps the blast radius small and gives the team a real error budget. The fastest route to trust is a clear abstention, backed by a citation, when the system is unsure.
References
- https://arxiv.org/abs/2005.11401
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/ETag
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Last-Modified
- https://www.postgresql.org/docs/current/textsearch.html
- https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html
Top comments (0)