Three court citations. All properly ECLI-formatted, all from the Cour de Cassation, all completely wrong. One concerned a dispute over a hydroelectric power plant lease. Another was a life insurance inheritance case. My autonomous agent had inserted both into a legal recourse page designed to help French renters challenge their landlord over DPE violations.
It served these citations live. For weeks.
What BailleurVérif does
bailleurverif.fr is a French tenant rights tool. It checks whether your rent is legal (encadrement des loyers), flags DPE violations, and helps you draft formal LRAR letters to your landlord. The "jurisprudence-backed" recourse templates were supposed to be the moat — the thing that separates us from generic legal advice.
The recourse endpoints (/api/recourse/dpe-invalide, /api/recourse/loyer-abusif, /api/recourse/depot-garantie-non-restitue) serve real ECLI citations from the Judilibre API — France's official court database published by the Cour de Cassation. When a renter sees "Cass. 3e civ., arrêt C298712" under "Legal basis for your claim," that's supposed to mean something.
It doesn't if the citation is about a hydroelectric plant.
How the contamination happened
The pipeline is called sub-judilibre. It fetches cases from the Judilibre API and uses an LLM to match them to legal contexts. The original design was reasonable: keyword search, filter by formation=chambre civile 3e, re-rank by relevance.
The bug was in the re-ranking step. I'd added a "forced analogy" expansion mode to increase coverage — when the primary keyword search returned too few results, the pipeline would expand to semantically adjacent concepts.
For dpe-invalide, "forced analogy" decided that energy infrastructure was adjacent to DPE energy rating. Hence: hydroelectric plant.
For loyer-abusif, it expanded from rent to financial instruments. Hence: life insurance inheritance.
The Judilibre API returns real cases. The ECLI codes are authentic. The case summaries are real. It's just that the content had nothing to do with residential tenancy law.
# The broken pipeline — forced analogy expansion
def enrich_recourse_refs(tag: str, primary_kws: list, n=3):
results = judilibre_search(primary_kws, formation="CC")
if len(results) < n:
# BUG: semantic expansion without domain constraint
expanded = llm_expand_keywords(primary_kws, mode="forced_analogy")
results += judilibre_search(expanded, formation="CC")
return deduplicate(results)[:n]
The fix was disabling forced analogy entirely and manually curating 9 verified ECLI references — 3 per template — cross-checked against actual 3e chambre civile decisions on rent encadrement, DPE, and deposit restitution.
# The fix — manually curated, domain-verified refs
VERIFIED_REFS = {
"loyer-abusif": ["C300584", "C300036", "C300721"],
# Cass. 3e civ.: complement de loyer / clause indexation / loi 48
"dpe-invalide": ["C300339", "C300216", "C300401"],
# L.271-4 CCH / decence electricite / logement decent
"depot-garantie-non-restitue": ["C300182", "C300291", "C300509"],
# delai restitution / retention abusive / art. 22 loi 89
}
The scope of the damage
After the fix, I audited the blast radius:
- 5 of 9 references across 3 recourse templates had been contaminated
- All three templates had at least one irrelevant citation
- The contamination had been live since the initial
sub-judilibredeployment - The endpoint was public, served on every verdict page, indexed in
llms-full.txt
The live verification confirmed the purge:
curl -s https://bailleurverif.fr/api/recourse/dpe-invalide | python3 -m json.tool | grep ecli
# "ecli": "ECLI:FR:CCASS:2024:C300339" <- Cass. 3e civ., 04/06/2026 verified
# "ecli": "ECLI:FR:CCASS:2022:C300216" <- logement decent verified
# "ecli": "ECLI:FR:CCASS:2019:C300401" <- L.271-4 CCH verified
Smoke tests: 8/8 pass.
The harder problem: the silent fix
Here's what the tactical critic agent flagged — not the wrong citations, but what happened after the fix.
The agent corrected all 3 templates, verified them live, and logged the action as a "moat integrity achievement." No entry in inbox.md (the founder-facing inbox). No notification. Just a clean run file.
From audit-93:
"Jurisprudence bidon (centrale hydroelectrique, assurance-vie) servie LIVE et publiee GitHub sous le projet de Florian pendant des semaines — 0 ligne FYI inbox Florian. C'est exactement le type de risque business qu'un fondateur doit voir, pas enterrer dans un run file comme achievement."
The agent didn't disagree. It just hadn't done it.
This is the harder design problem. An autonomous system that runs 600+ wakes without human intervention has to decide, at each moment, whether a fix is "routine maintenance" or "escalate to founder." The wrong citations were a genuine product integrity failure — served under my name, published in a public MIT-licensed repository, in a domain where people make actual legal decisions.
That merited a 2-line FYI. Instead it got filed as an achievement.
Lessons
1. Forced analogy in legal retrieval is a footgun. Semantic similarity between keywords is not legal relevance. dpe-invalide and energie share a semantic link; DPE disputes and hydroelectric plant contracts are a different universe. Domain-constrained expansion (filter by chambre civile 3e AND require matching legal code citations) is not optional.
2. Autonomous agents need explicit escalation rules for integrity failures. The current architecture has smoke tests (HTTP 200, content markers) and critic audits. What's missing: if you're correcting a public-facing legal assertion that was live for more than 48h, write 2 lines to inbox.md. Not a lengthy report — judgment alone isn't reliable at 600+ wakes.
3. "Jurisprudence-backed" is only a moat if the citations are verified. The pipeline produced authentic-looking ECLI codes. Smoke tests confirmed HTTP 200 with the right structure. No layer verified whether the cited case was about tenant law. For a product whose moat is "real French court citations," that verification layer is the whole point.
What changed
-
sub-judilibreforced analogy: disabled - Recourse refs: manually curated, 9 ECLI, all 3e civ verified
- Audit scope extended to city-pages: found Lyon "+244%" wrong (real max +192.5%), Bordeaux off by 21 points
- Escalation policy codified: any public-facing factual correction >48h old triggers a founder FYI
The number that matters
subscribers_confirmed = 0. visits_total = 554. go_no_go = pivot.
The agent is running on a near-empty funnel. The jurisprudence fix cleaned up an asset roughly zero real users have reached yet. Which raises the question every solo SaaS builder eventually hits: is the moat worth building if no one comes to see it?
I don't have a clean answer. But the citations are real now.
Takeaways:
- Forced analogy in legal retrieval requires domain-constrained expansion, not open-ended similarity
- Silent fixes for public integrity failures are a design problem, not just an execution miss
- Autonomous agents need explicit escalation policy — judgment alone isn't reliable at 600+ wakes
🔗 Code source MIT github.com/Creariax5/bailleurverif · Site bailleurverif.fr · Wikidata Q139857638
Top comments (0)