A legal contract review workspace should fall back only when the current retrieval stage cannot support a precise citation. For a developer tool that watches a contract web page, detects a change, and alerts on the diff, the practical design is a four-trigger ladder: missing coverage, weak clause identity, version conflict, and insufficient citation context. Each step may widen recall, but it must preserve the exact page version and text span that justify the alert.
That is the result. The evaluation constraint matters more than the retrieval brand: a reviewer must be able to open an alert, locate the changed language, and tell which document version produced the answer. An empty result is obvious. A fluent answer grounded in the wrong revision is much worse.
The tempting implementation is one search call followed by a larger top-k retry. Don't ship that as a fallback policy. More candidates can hide the failure signal, increase prompt cost, and mix nearby clauses that use similar language for different obligations. The useful fallback changes a retrieval assumption, records why it changed, and refuses to turn weak evidence into a strong claim.
How should a legal contract review workspace trigger retrieval fallbacks?
Use observable evidence conditions, not a generic confidence number. A single score rarely tells you whether the problem is vocabulary mismatch, a missing page snapshot, a bad clause boundary, or two revisions competing for the same citation. Those failures require different actions.
The four triggers below work as an explicit state machine. Their thresholds are examples to calibrate with a labeled set; they are not universal quality guarantees.
| Trigger | What it means | Next retrieval action | Citation rule |
|---|---|---|---|
| Coverage gap | No candidate overlaps the changed span | Search the surrounding section in the same snapshot | Cite the diff span plus its section heading |
| Identity gap | Candidates lack a stable clause or section key | Add lexical matching over headings and normalized text | Mark the clause identity as inferred |
| Version conflict | Top candidates come from different snapshots | Filter to the alert's captured version, then retrieve again | Never merge versions into one citation |
| Context gap | The matched span omits a definition or exception | Expand to adjacent blocks within the same section | Cite both the changed clause and supporting block |
Start narrow. Stage zero should use information the watcher already has: canonical page URL, captured version, changed character range, nearby heading, and content hash. If that produces a citation-ready passage, stop. Stage one adds clause-aware lexical retrieval inside the same version. Stage two can add semantic retrieval, still version-filtered. Stage three expands context around the winning passage.
Order is important. Semantic similarity is useful when wording drifts, but it should not silently outrank direct evidence from the observed diff. The ladder spends uncertainty in small increments — one changed assumption at a time — so a failed alert is diagnosable later.
Make the diff the retrieval anchor
A page watcher gives the retrieval system something ordinary document chat does not have: a before-and-after boundary. Keep it. The diff is the query seed, the evaluation target, and the first citation candidate. Re-embedding an entire contract and asking a broad question throws away that advantage.
Represent every retrievable block with lineage fields before generating embeddings: document_id, snapshot_id, source_url, captured_at, section_path, start and end offsets, text, and a content hash. A generated answer should carry the candidate IDs it used. This is plumbing, but it is the plumbing that lets an evaluator distinguish “the search found a related indemnity clause” from “the search found the indemnity sentence that changed on this page version.”
A focused Python policy can stay pleasantly small:
from dataclasses import dataclass
from enum import Enum
class Trigger(str, Enum):
READY = "ready"
COVERAGE_GAP = "coverage_gap"
IDENTITY_GAP = "identity_gap"
VERSION_CONFLICT = "version_conflict"
CONTEXT_GAP = "context_gap"
@dataclass(frozen=True)
class Candidate:
candidate_id: str
snapshot_id: str
section_path: tuple[str, ...]
start: int
end: int
text: str
overlaps_diff: bool
has_stable_section: bool
needs_definition: bool
def choose_trigger(
candidates: list[Candidate], expected_snapshot: str
) -> Trigger:
if not candidates or not any(item.overlaps_diff for item in candidates):
return Trigger.COVERAGE_GAP
matching = [
item for item in candidates if item.snapshot_id == expected_snapshot
]
if len(matching) != len(candidates):
return Trigger.VERSION_CONFLICT
if not all(item.has_stable_section for item in matching):
return Trigger.IDENTITY_GAP
if any(item.needs_definition for item in matching):
return Trigger.CONTEXT_GAP
return Trigger.READY
This function does not pretend to retrieve or generate. It makes the routing decision testable. In a notebook, feed it hand-labeled candidates from real page diffs. In production, log the trigger, stage, candidate IDs, snapshot filter, query text, and citation spans as one trace. Same contract, different scale.
There is a subtle trap here. Suppose the watched page changes “within 30 days” to “within 15 days,” while an older snapshot contains the same section heading and scores slightly higher. A broad retry may return both. The correct behavior is not to average their relevance or let the model reconcile them; it is to constrain retrieval to the captured snapshot associated with the alert, cite the new text, and retain the prior snapshot only as explicitly labeled comparison evidence. That single distinction prevents a polished summary from laundering stale language into a current review.
Small boundary, big payoff.
Preserve citations while recall widens
A fallback should return an evidence envelope, not a loose string. At minimum, that envelope needs the source URL, snapshot ID, section path, offsets, excerpt, retrieval stage, and trigger. Generation receives envelopes and may cite only those envelopes. If a supporting definition is added, it gets its own locator rather than being pasted into an untraceable mega-chunk.
The research formulation of retrieval-augmented generation combines parametric and non-parametric memory. For this application, the operational implication is straightforward: retrieved material must remain inspectable after generation. The model output is not the evidence record; the versioned passage is.
Keep abstention in the contract. When no stage can produce a passage from the expected snapshot, emit an alert that says the change was detected but no reviewable explanation was generated. That's a successful safety outcome, not an excuse to search unrelated versions. I'm not sure one global context-expansion limit will fit both short click-through terms and long negotiated agreements; section length and the rate of cross-references should resolve that during evaluation.
The catch is that strict snapshot filtering is not suitable when the actual task is historical comparison across amendments. In that workflow, make version comparison a first-class mode with paired citations. Likewise, stick with deterministic diff presentation when counsel only needs exact additions and removals; retrieval and generation add cost without improving that narrow job.
Test the ladder before tuning the model
Build the eval set from page-change events, not generic legal questions. Each case should include the old snapshot, new snapshot, diff range, expected clause or section, required supporting definitions, and a valid abstention outcome. Include easy wording edits, moved clauses, reused headings, changed definitions, deleted sections, and pages whose navigation or footer also changed.
Then score stages separately. Useful measures are diff coverage, snapshot accuracy, locator accuracy, citation entailment, abstention correctness, fallback frequency, candidates inspected, retrieved characters, prompt tokens, and end-to-end latency. Report the distribution by trigger. One average can conceal a version-conflict path that is rare and disastrous.
Do this early.
A notebook-to-production workflow should replay the same frozen cases against every change to chunking, embeddings, reranking, prompt construction, or context expansion. Store the retrieval trace beside the expected evidence, then compare both answer quality and prompt cost. If stage three fires on half the corpus, the problem may sit in section parsing rather than model quality. If citation accuracy falls when recall rises, the fallback is too permissive.
Don't tune all four triggers at once. First require the expected snapshot. Then validate overlap and section identity. Add context expansion last, because it changes both token usage and the surface area from which a plausible but irrelevant sentence can be selected. Your mileage may vary on the exact thresholds, but the sequence keeps the experiment legible.
What should be measured before copying this design?
Measure whether your source can support its assumptions. Stable snapshots and offsets are easy when the watcher stores normalized page content; they are harder when pages render personalized text or reorder blocks. Clause identity also varies: a numbered agreement has a better structural key than a short terms page with visual headings only.
Before adopting the ladder, answer four questions with data: How often does the diff map to one section? How often do definitions outside that section change the meaning? How often do duplicate clauses occur across versions? How frequently is abstention acceptable to reviewers? Those answers determine whether you need semantic retrieval at all and how much surrounding text belongs in a citation envelope.
The recommendation is deliberately conditional. Use this architecture when a changed web page, a captured version, and reviewable citations define success. Do not use it as-is for open-ended discovery across a contract repository, where cross-document recall is the main goal; that needs a separate retrieval policy and an evaluation set built around repository questions. Also avoid generated explanations when exact redlines already satisfy the user.
A good fallback is boring under inspection: one trigger, one relaxed assumption, one traceable result. Copy the design only after your eval set shows that each added stage recovers relevant evidence without weakening version accuracy or citation entailment.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
Top comments (0)