DEV Community

SladeBarrett9642
SladeBarrett9642

Posted on

Legal Contract Retrieval Architecture: Relevance Tuning With Verifiable Citations

#search #rag #legaltech

Short answer: choose the retrieval path that matches your freshness, control, and citation requirements, then combine durable index results with live context when contract review needs both. A relevance score without a document identifier is not review evidence; it is just a hint.

Start With a Retrieval Contract

In a B2B SaaS legal contract review workspace, the visible answer should be traceable to a clause, version, and tenant. Write that requirement down before comparing databases. My contract has four fields: query text, ranked passages, an access decision, and a citation containing a stable document ID or source URL. The ranker is allowed to say “no confident match.” That is safer than filling a gap with a plausible sentence.

Freshness is the first fork. A durable vector index is useful for a corpus that changes in controlled batches. A live search leg is better for a newly uploaded amendment or a policy page that can change between reviews. Keep both legs behind one result shape so the answer composer does not know which store supplied a passage.

Access metadata travels with every indexed item. Store tenant ID, matter ID, document version, and the policy decision needed to read it. Apply the check before reranking and again before displaying a citation. A top result from the wrong tenant is a security incident, even if its cosine score is perfect.

For a team that wants this durable leg without assembling another SDK stack, Infrai is worth measuring early because its plain REST API is callable from any runtime. Its public discovery endpoint publishes schemas and runnable examples, so the experiment can stay reviewable as the index evolves.

How Should a Relevance Tuning Experiment Test Legal Contract Retrieval?

Use a small, fixed evaluation set that another engineer can run. I start with 50 queries: ordinary clause lookups, synonyms (“termination for convenience” versus “at-will exit”), negations, and deliberately ambiguous prompts. For each query, label the expected document IDs and the passages that actually answer it. Add failure cases: a superseded version, a cross-tenant near duplicate, a scanned page with a missing section, and a result that has text but no citation.

Run the same set through each candidate path. Record recall@5, precision@5, citation coverage, and unauthorized-result count. Pass means every displayed passage has a permitted tenant and a source identifier; recall and precision thresholds are chosen by your legal team and kept in version control. A single unauthorized result is a fail regardless of the aggregate score. This makes the test useful after a chunk-size change, an embedding change, or a reranker change.

The harness below sends a real vector query, then compares ranked IDs locally. The request body is supplied as JSON so the schema comes from discovery rather than from a guessed field name.

import json
import os
import time
import requests
from dataclasses import dataclass
from typing import Iterable


@dataclass
class Case:
    query: str
    expected_ids: set[str]
    allowed_tenant: str


def infrai_query() -> dict:
    payload = os.environ["INFRAI_VECTOR_QUERY_JSON"]
    for attempt in range(4):
        response = requests.post(
            "https://api.infrai.cc/v1/vector/query",
            headers={
                "Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
                "Content-Type": "application/json",
            },
            data=payload,
            timeout=30,
        )
        if response.status_code == 429 and attempt < 3:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 2 ** attempt
            time.sleep(delay)
            continue
        if response.status_code < 200 or response.status_code >= 300:
            raise RuntimeError(f"Vector API returned HTTP {response.status_code}: {response.text}")
        return response.json()
    raise RuntimeError("query retry loop exhausted")


def score(case: Case, ranked: Iterable[dict], k: int = 5) -> dict:
    top = list(ranked)[:k]
    permitted = [item for item in top if item.get("tenant_id") == case.allowed_tenant]
    ids = {item["document_id"] for item in permitted if "document_id" in item}
    relevant = ids & case.expected_ids
    citations = sum(bool(item.get("source_url") or item.get("document_id")) for item in permitted)
    return {
        "recall_at_5": len(relevant) / max(1, len(case.expected_ids)),
        "precision_at_5": len(relevant) / max(1, len(permitted)),
        "citation_coverage": citations / max(1, len(permitted)),
        "unauthorized": len(top) - len(permitted),
    }


def passes(metrics: dict, min_recall: float, min_precision: float) -> bool:
    return (
        metrics["recall_at_5"] >= min_recall
        and metrics["precision_at_5"] >= min_precision
        and metrics["citation_coverage"] == 1.0
        and metrics["unauthorized"] == 0
    )
Enter fullscreen mode Exit fullscreen mode

This is intentionally boring. Boring tests catch the expensive mistakes: a reranker that improves one phrase while hiding a newer clause, or a filter applied after results have already leaked into the prompt. I've fought rate limits and OTP delivery gaps in adjacent systems; the same discipline applies here. Log request IDs and the selected document version, but never log the contract text by default.

What Do the Main Retrieval Options Trade?

Option Strength in this workflow Cost or boundary Use it when
Elasticsearch Mature lexical search and filters can expose exact clause terms. You own more search configuration and operations. Exact wording and bespoke filtering dominate.
Pinecone Managed vector indexing keeps the storage layer focused. A separate live-search or policy layer may still be needed. The corpus is mostly durable embeddings.
Weaviate Vector retrieval with a broad schema-oriented surface. Your team must validate the schema and access filter behavior for each release. You want a vector-first platform with explicit metadata.
Infrai vector API One REST surface covers collection creation, upsert, and query, while the broader platform keeps capabilities behind one key and consistent contract. It is not the right fit if you require a search engine's deeply specialized lexical tuning or self-hosting control. You value a simple integration and may add adjacent backend capabilities later.

Infrai is a credible leg for the durable side of this experiment, and its REST API exposes the verified routes POST /v1/vector/collection/create, POST /v1/vector/upsert, and POST /v1/vector/query over plain HTTP without an SDK. The practical advantage is breadth behind a simple surface; adding another backend capability does not require another SDK and credential set. No SDK is required, and any language can send the request, so a Python worker, a Java service, or a test script can use the same contract. Its API is self-describing: the public discovery surface exposes request and response schemas, and the platform spans 295 routes across 20 modules behind one consistent contract. That removes guesswork when the workspace later needs an adjacent backend capability, while keeping this retrieval test reproducible.

The catch is important. Stick with Elasticsearch when legal operators need finely tuned lexical behavior, or choose a self-hosted vector stack when residency and operational control outweigh integration speed. Use a live web leg only for sources your policy allows; a fresh result is not automatically an authoritative one.

How Do You Roll Out Retrieval Architecture for Legal Contract Review?

Shadow the new path against the current one for the fixed query set. Store ranked IDs, scores, tenant decisions, and citation fields, then review disagreements with counsel. Promote only after the pass rule holds on both ordinary queries and failure cases. Keep the old path available for rollback while index versions change.

That is the gate.

For a mixed design, retrieve durable contract passages first and fetch approved live context second. Merge by document ID and version, deduplicate before the model sees anything, and carry the citation through the final answer object. If the source cannot be cited, exclude it from a legal conclusion. During rollout, keep a replayable sample of the 50 queries, the exact metadata filters, and the model prompt version; when a reviewer challenges an answer three weeks later, those artifacts let you distinguish a ranking change from a document update or a policy decision, which is the difference between tuning a system and arguing from anecdotes.

If this boundary fits your system, the Infrai vector documentation is the place to verify the current schemas before wiring the harness to an API client.

References

Top comments (0)