We are building a contract risk scanner that runs the same legal clause through multiple LLMs and surfaces disagreements in semantic interpretation. Legal language is full of hidden obligations and ambiguous qualifiers, so this tool is useful for legal ops teams and developers automating pre-signature review. By comparing outputs from several Oxlo.ai models, we can spot low-confidence risks before they become liabilities.
What you'll need
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai - An Oxlo.ai API key from https://portal.oxlo.ai
Step 1: Set up the Oxlo.ai client
I initialize the OpenAI-compatible client pointing at Oxlo.ai. I keep the API key in an environment variable so I do not accidentally commit it.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)Step 2: Define the evaluation schema
To make the model output comparable, I force a JSON structure that captures risk type, severity, and the exact phrase that triggered it. I will enable JSON mode in the API call later.
import json
# Expected response shape (enforced via response_format in the API call)
EVAL_SCHEMA = {
"risks": [
{
"risk_type": "string",
"severity": "low|medium|high|critical",
"trigger_phrase": "string",
"reasoning": "string"
}
],
"overall_summary": "string"
}Step 3: Write the system prompt
The prompt instructs the model to perform deep semantic analysis, not keyword matching. It must distinguish between terms like "reasonable efforts" and "best efforts" because those carry different legal obligations.
SYSTEM_PROMPT = """You are a senior legal analyst reviewing contract clauses for hidden risks and semantic ambiguity.
Analyze the user-supplied clause and return JSON matching this schema:
{
"risks": [
{
"risk_type": "string",
"severity": "low|medium|high|critical",
"trigger_phrase": "string",
"reasoning": "string"
}
],
"overall_summary": "string"
}
Guidelines:
- Identify semantic traps: unlimited liability, ambiguous termination triggers, asymmetric indemnity, vague service levels, and words like "reasonable" or "best efforts" without definitions.
- Explain the real-world obligation created by each trigger phrase.
- If a phrase is standard and low risk, omit it.
- Return only valid JSON."""Step 4: Build the single-model evaluator
I write a function that sends the clause to one Oxlo.ai model and parses the JSON response. I use the exact client pattern and enable JSON mode so the output is machine readable.
def analyze_clause(model_id: str, clause: str) -> dict:
response = client.chat.completions.create(
model=model_id,
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": clause},
],
response_format={"type": "json_object"},
temperature=0.2,
)
raw = response.choices[0].message.content
return json.loads(raw)Step 5: Compare across models
Now I run the same clause through four Oxlo.ai models that excel at reasoning. Each model sees identical context, so differences in output reflect genuine variance in semantic interpretation.
MODELS = [
"llama-3.3-70b",
"qwen-3-32b",
"deepseek-v3.2",
"kimi-k2.6",
]
def compare_models(clause: str) -> dict:
results = {}
for m in MODELS:
print(f"Running {m} ...")
results[m] = analyze_clause(m, clause)
return resultsStep 6: Surface disagreements with consensus scoring
I add a lightweight scorer that flags risks mentioned by only one model. A lone high-severity finding usually means the model is either hallucinating or catching something others missed, both of which deserve human review.
from collections import Counter
def consensus_report(results: dict) -> None:
all_risks = []
for model, data in results.items():
for risk in data.get("risks", []):
all_risks.append({
"model": model,
"type": risk["risk_type"],
"severity": risk["severity"],
"phrase": risk["trigger_phrase"],
})
by_phrase = {}
for r in all_risks:
by_phrase.setdefault(r["phrase"], []).append(r)
print("\n=== Consensus Report ===\n")
for phrase, hits in by_phrase.items():
sevs = [h["severity"] for h in hits]
count = len(hits)
print(f"Phrase: '{phrase}' | Mentioned by {count}/4 models")
print(f" Severities: {sevs}")
if count == 1 and hits[0]["severity"] in ("high", "critical"):
print(" WARNING: Lone high-severity finding. Review manually.")
print()Run it
I test with a clause that looks benign but contains a subtle unlimited liability hook.
CLAUSE = """
Vendor shall indemnify and hold harmless Client against any and all claims, losses,
liabilities, damages, and expenses arising out of or relating to the Services,
including but not limited to third-party claims, regardless of whether such claims
are based on negligence, breach of contract, or any other theory of liability,
and regardless of whether Vendor has been advised of the possibility of such damages.
"""
if __name__ == "__main__":
results = compare_models(CLAUSE)
consensus_report(results)
print("\n=== Raw Summaries ===\n")
for model, data in results.items():
print(f"{model}: {data['overall_summary']}")Example output:
Running llama-3.3-70b ...
Running qwen-3-32b ...
Running deepseek-v3.2 ...
Running kimi-k2.6 ...
=== Consensus Report ===
Phrase: 'any and all claims' | Mentioned by 4/4 models
Severities: ['high', 'high', 'high', 'critical']
Phrase: 'regardless of whether such claims are based on negligence' | Mentioned by 3/4 models
Severities: ['high', 'medium', 'high']
Phrase: 'regardless of whether Vendor has been advised of the possibility of such damages' | Mentioned by 1/4 models
Severities: ['critical']
WARNING: Lone high-severity finding. Review manually.
=== Raw Summaries ===
llama-3.3-70b: Broad indemnity clause with uncapped exposure; negligence carve-out is missing.
qwen-3-32b: High-risk unlimited indemnity lacking liability caps or negligence exclusions.
deepseek-v3.2: Unlimited liability exposure via sweeping indemnity and omission of foreseeability limits.
kimi-k2.6: Critical risk: vendor assumes liability without cap, including unforeseeable damages.Next steps
Wire this harness into a CI job that flags contract pull requests automatically, or extend it to diff two versions of a clause and highlight shifting semantic risk. Because Oxlo.ai uses flat per-request pricing, running four models on every clause costs the same regardless of whether you pass a single paragraph or a fifty-page agreement. See https://oxlo.ai/pricing for details.
Top comments (0)