I built an AI that reads contracts like a lawyer — and knows when to say "I don't know"

Tiến Trung Nguyễn — Mon, 06 Jul 2026 08:46:29 +0000

Picture a small Vietnamese furniture exporter. A German buyer sends over a 12-page contract in
English. Somewhere on page 7, there's a line: "late delivery penalty: 15% of contract value."

The owner signs it. Nobody told her that Vietnamese law caps that penalty at 8% — anything
above is simply void in court. She just agreed to a term that isn't even legal, and she'll
negotiate her next three contracts without ever knowing she had that card in her hand.

That's the problem I built Legal Guard for, during the Qwen Cloud Hackathon (Autopilot Agent
track). It's an AI agent that reads your contract, tells you which clauses are merely bad for
you and which are actually illegal, and helps you push back — while a human approves every
message before it goes out.

Try it: https://legalguard.duckdns.org · Code (open source): https://github.com/trungnguyen1618033/legal-guard-PH

Here's what I learned, in plain language.

Lesson 1: Don't send a senior partner to do a photocopy job

AI models are like staff at a law firm. The senior partner (qwen3.7-max) is brilliant and slow.
The paralegal (qwen-flash) is fast and great at simple, well-defined checks.

My first version sent everything to the senior partner. Analyzing one contract took minutes,
and most of that time was spent on questions as simple as: "Does this law article actually say
what we claim it says — yes or no?"

So I split the work the way a real firm would:

Hard reasoning (analyzing the contract, planning negotiation strategy) → the big model.
Yes/no double-checks → the fast model: 0.5 seconds instead of 23 — about 46× faster, with the same answers.
Quick legal Q&A → the mid-size model: 4–6 seconds instead of ~48.

One phase of the pipeline dropped from ~4 minutes to ~7 seconds. Nothing got smarter —
the work just went to the right desk.

Lesson 2: The most dangerous AI answer is the confident wrong one

Everyone worries about AI "hallucinating" — making things up. In legal work the failure is
sneakier: the AI cites a real law article that simply doesn't say what the AI claims.
The citation checks out; the meaning doesn't. A busy reader would never catch it.

Three guardrails fixed this:

A second pair of eyes. After the agent flags a risk, a separate AI checker gets one job: "Read this law article. Does it actually support this claim — yes or no?" If the answer is fuzzy, we treat it as no. In law, wrongly shouting "this clause is illegal!" is worse than quietly asking a human to review.
No zombie laws. Laws get replaced constantly. Legal Guard only cites law that is currently in force — and if you ask "what was the rule in 2020?", it answers with the law as it stood in 2020.
Knowing when to say "I don't know." Ask something outside its knowledge base and it says "I don't have enough legal basis to answer" — like a good lawyer saying "let me check" instead of guessing. We treat a correct refusal as a correct answer in our tests.

We test all of this against 54 questions with lawyer-known answers, across 12 areas of
Vietnamese law. Current score: 54/54 on majority-vote (3 runs per case) — up from 87% when we
started. One borderline case still flickers between runs (a wording match on the hosted model), so a
single run may read 53/54 — we disclose that rather than round up to a flat 100%. The whole
methodology is published at https://legalguard.duckdns.org/trust — because an AI that touches legal
risk should show its report card.

Lesson 3: If a cheap model guards the door, test the guard

Remember the fast "paralegal" model doing the yes/no checks? Once it gates which citations
survive, it becomes the safety-critical part. So it gets its own exam: 16 tricky
statements paired with real statute text — including traps like "a 10% penalty is valid under
this article" shown next to the article that says 8%. We score the fast model against both the
correct answers and the big model's answers. Result: 16/16 correct, 100% agreement with the
flagship, at a fraction of the latency (test report). That test is what let us
make the faster trade with a clear conscience.

Lesson 4: "Autopilot" means it works while you sleep

The track is called Autopilot Agent — and I took that literally. On the production server
(one small Alibaba Cloud machine running everything in Docker), a scheduler wakes the agent at
5 AM every day. It checks which laws became effective, then cross-references every contract
it has ever reviewed: did a new decree just change an article your contract relies on?

It's precise, too: a decree amending Article 9 alerts only the contracts citing Article 9 —
no spam. And if you dismiss a false alarm once, it stays dismissed. During testing this fired
on real data: one decree about arbitration flagged 8 stored contracts with foreign-arbitration
clauses. Nobody asked it to. That's the point.

Lesson 5: A contract scanner tells you what's wrong; a copilot tells you what to do

Every contract AI flags risky clauses. The gap nobody fills for small businesses: what do I do
about it, from my actual bargaining position? So Legal Guard doesn't stop at analysis — it runs a
stateful, multi-round negotiation.

You tell it your leverage and whether you have a backup deal (BATNA). Then, round after round, it:

remembers what's already agreed — a running "concession ledger" so it never re-negotiates a point you already won (the naive version kept forgetting and giving ground back);
proposes trades, not surrender — "concede the deposit to lock arbitration in Vietnam" — and screens every suggestion so it never offers away a red-line;
knows when to walk — a deterministic guardrail flips to "recommend walk-away" the moment the other side blocks a must-fix red-line and you hold a real alternative. That rule isn't left to the model's mood; it's enforced in code.
learns from outcomes — tactics that closed real deals get weighted up next time.

That's the line between a document scanner and a negotiation copilot — and it's the part I'd defend
as the real moat.

What failed (worth as much as what worked)

A fancy tree-search retrieval method lost to the boring hybrid approach on our tests. Boring won.
A graph-based reranking idea made zero measurable difference. It's in the code, switched off.
Hand-tuning thresholds to fix each failing test broke a different test every time — whack-a-mole. A structural fix (an automatic cutoff, nothing hand-tuned) is what finally held.

The one-line summary

Spend your speed budget on reasoning, your safety budget on verification, and publish the number
you actually measured — not the one that looks good.

*Built with Qwen models on Qwen Cloud (DashScope), deployed on Alibaba Cloud ECS. Open-source

Which security bug should you actually worry about first?

Tiến Trung Nguyễn — Tue, 23 Jun 2026 17:07:19 +0000

Which security bug should you actually worry about first?

Imagine your security scanner hands you a list of 50 problems. Each one says: "this is bad,
it's in this file, on this line." But it never tells you the thing that actually matters:
if someone exploited this, how much of your app would break — and whose job is it to fix it?

Here's the catch. A scanner rates a bug in some forgotten, unused file exactly the same as the
same bug sitting in the login code that half your app depends on. Same label, wildly different
danger. So teams waste time on the harmless ones and sometimes miss the scary one.

For the GitLab Transcend Hackathon, I built a little assistant to fix that. It's called
Vulnerability Blast Radius.

What it does, in plain words

Think of it like this: when a bug is found, the tool doesn't just shrug and show you the line.
It asks a simple question — "what else in the code touches this?" — and follows the trail.

It finds the security bug.
It maps the "blast radius": which other parts of the code call or rely on the broken file. (This is the clever part — it uses GitLab's "knowledge graph," basically a map of how all the code connects.)
It gives it a score from 0 to 100, so the genuinely dangerous bugs float to the top.
It finds the right owner (the team responsible for that code).
It writes a tidy note suggesting what to do — but it only suggests. A human always decides.

A real example from the demo: one bug turned out to have 11 other files depending on it → it
scored 96/100 (Critical) → and got routed straight to the security team. That's the difference
between "a bug somewhere" and "a bug that can really hurt you."

The part I'm most proud of: it doesn't make things up

AI tools can be confidently wrong. So I made a deliberate choice: the score is calculated by
plain, predictable code — not by the AI. Same bug in, same number out, every single time. You
can check the math yourself:

Severity (critical)        40
Likelihood of exploit      16
Known to be exploited?     15
Is the code reachable?     10
How much depends on it     15
───────────────────────────────
                           96 / 100

And if the tool doesn't have real data yet, it says so honestly and leaves blanks — it never
invents a number to look smart. It also asks permission before changing anything, and won't post
the same note twice. Trustworthy beats flashy.

The funny twist

When I ran it for real, the very first bug it caught… was a small mistake in my own demo code.
Honestly? That's the best possible advertisement. It works on real problems, not just neat examples.

A peek behind the curtain (for the curious)

GitLab's knowledge graph is brand new, so the docs didn't always match reality. The most useful
lesson of the whole project: check what the system actually does, don't trust the example. A
couple of surprises only showed up when I tested against the live system — like the fact that the
"link" I expected between a bug and its file simply didn't exist, so I had to find another way in.
Little things like that would have quietly broken everything if I'd trusted the manual.

The result: it's open-source (MIT), backed by 94 automated tests, and published as a reusable
agent on GitLab's AI Catalog.

Have a look

🎥 2-minute demo: https://youtu.be/cEhLA04SV2c
📦 Code (free, MIT): https://gitlab.com/gitlab-ai-hackathon/transcend/7723089
🤖 The agent: https://gitlab.com/explore/ai-catalog/agents/1011711/

The big idea is simple: a security finding shouldn't just tell you what's wrong — it should tell
you how much it matters and who can fix it. Now it can.

DEV Community: Tiến Trung Nguyễn

I built an AI that reads contracts like a lawyer — and knows when to say "I don't know"

Lesson 1: Don't send a senior partner to do a photocopy job

Lesson 2: The most dangerous AI answer is the confident wrong one

Lesson 3: If a cheap model guards the door, test the guard

Lesson 4: "Autopilot" means it works while you sleep

Lesson 5: A contract scanner tells you what's wrong; a copilot tells you what to do

What failed (worth as much as what worked)

The one-line summary

Which security bug should you actually worry about first?

Which security bug should you actually worry about first?

What it does, in plain words

The part I'm most proud of: it doesn't make things up

The funny twist

A peek behind the curtain (for the curious)

Have a look