Gatekeeper: Building an AI-Powered Pull Request Review Desk for Azure DevOps

How a single HTML file turned our messy PR hygiene into a stamped, auditable review pipeline.

---

The problem: merges that nobody could trace

Every engineering team has a version of this story. A bug ships to production. Someone asks, "which change caused this?" You find the pull request — and it has no linked work item, an empty description, and zero reviewer approvals. It was merged on a Friday evening by someone in a hurry.

Our team is small. There is no army of release managers enforcing process. Azure DevOps can enforce branch policies, but policies are blunt instruments: they block or they don't. What we actually wanted was something in between — a review desk that could look at any PR on demand and answer three questions instantly:

1. Is this PR mapped to a ticket? If code can't be traced back to a work item, our audit trail is broken.
2. Has a human actually reviewed it? Assigned reviewers who never voted don't count.
3. Is the code itself any good? Bugs, secrets in config files, missing error handling — the things a tired reviewer skims past.

The first two are governance questions with deterministic answers. The third is a judgment call — and that's exactly where an LLM earns its keep.

The constraint that shaped everything: one HTML file

We deliberately built this as a single, self-contained HTML file. No backend, no deployment pipeline, no database, no npm build. You open the file, paste four values — organization, project, a personal access token (PAT), and a PR ID — and press one button.

This constraint bought us three things:

• Zero infrastructure. Nothing to host, patch, or pay for.
• Zero credential storage. Every secret lives only in the page's memory and dies on refresh. There is nothing to leak because nothing is persisted.
• Instant adoption. "Here's a file, open it" is the shortest onboarding document ever written.

Architecture: two layers of review

The interesting design decision was splitting the review into two layers with very different trust models.

Layer 1 — Deterministic compliance gates

Facts should not be outsourced to a language model. Whether a work item is linked to a PR is not a matter of opinion — it's a field in an API response. So the first layer is plain JavaScript running four checks against the Azure DevOps REST API:

Gate	How it's checked	Verdict
Ticket mapping	GET .../pullRequests/{id}/workitems — if empty, a regex scans the title, description and branch name for ticket-shaped references (AB#123, PROJ-456)	MAPPED / LOOSE REF / NOT MAPPED
Code review sign-off	The reviewers array on the PR object. Azure DevOps encodes votes numerically: 10 = approved, 5 = approved with suggestions, −5 = waiting, −10 = rejected	REVIEWED / NOT REVIEWED / REJECTED
Description quality	Character count of the PR description	DOCUMENTED / THIN / EMPTY
Open discussions	GET .../threads, counting any with active or pending status	RESOLVED / N OPEN

Each gate renders as a card with a rubber-stamp verdict that slams onto the screen — a small theatrical touch, but it does real work: a big red NOT MAPPED stamp is impossible to ignore in a way that a table row never is. Process failures should feel like failures.

The "loose ref" state deserves a mention. Teams often think they've linked a ticket because the branch is named feature/PROJ-456-fix-login. That's a convention, not a link — the work item relationship in Azure DevOps is what makes traceability queryable. Gatekeeper distinguishes the two: a formal link passes, a naming-convention-only reference gets a yellow warning telling you to link it properly.

Layer 2 — The AI deep review

Once the facts are stamped, the judgment layer kicks in:

1. Gather the evidence. The app fetches the PR's latest iteration, lists every changed file, and downloads the actual contents of up to eight text files from the source branch tip (filtered by extension, capped per file to keep the prompt sane).
2. Build one structured prompt. Title, description, branch, the linked-ticket list (or an explicit "NONE — no tickets mapped"), reviewer votes, unresolved thread count, the full file list, and the raw code.
3. Demand structured output. The model is instructed to act as a strict senior reviewer and respond only in JSON matching a fixed schema: a plain-language summary, an overall risk level (low → critical), a recommendation (approve / request changes / block), and up to eight findings, each with severity, category, file, an explanation of what's wrong, and a concrete suggested fix.
4. Treat governance as a code smell. The prompt explicitly tells the model that missing tickets or missing sign-off are high-severity findings in their own right — so the AI verdict and the compliance stamps reinforce each other.

The response is parsed defensively (with a fallback that extracts the outermost JSON object if the model wraps it in prose), findings are sorted critical-first, and the whole thing renders as a risk dial plus expandable finding cards.

Why not just one layer?

Because the failure modes are opposite. Deterministic checks never hallucinate but can't read code. LLMs read code brilliantly but shouldn't be trusted to count reviewers. Splitting the layers means the compliance verdicts are always correct even if the AI call fails entirely — and the AI gets to spend its entire token budget on the thing only it can do.

Bring your own AI

The first version only worked inside a hosted AI environment where the model call was authenticated invisibly. That was fine until someone asked the obvious question: "can I run this on my own machine?"

So the sidebar grew an AI engine selector with three modes:

• Built-in — no key, works when the page runs inside a platform that proxies model calls.
• Anthropic API key — paste your own key; the page calls the Claude API directly from the browser (Anthropic supports direct browser access with an explicit opt-in header). This makes the file fully standalone.
• Azure OpenAI — point it at your own endpoint, deployment name and key. One caveat: Azure OpenAI doesn't allow browser calls by default, so in practice you route the endpoint through an API gateway with a CORS policy. The app detects this failure and tells you exactly that.

Credential handling is identical across all three: page memory only, sent directly from the browser to the respective service, never written to storage of any kind.

What we learned

Make process failures loud. The single most effective feature isn't the AI — it's the stamp. Nobody argues with a giant red NOT MAPPED.

Cap the evidence, not the honesty. On a 60-file PR, the AI reviews a sample (eight files, ~45K characters). We surface that limitation instead of hiding it. A review tool that pretends to be exhaustive is worse than one that's honest about its window.

Structured output is a contract. Forcing the model into a strict JSON schema — and salvage-parsing when it drifts — turned "chat about my code" into a repeatable pipeline stage with a predictable shape.

Validation order matters. Credentials are checked before any API calls fire, so you never burn a full fetch pipeline just to hit a missing-key error at the last step.

Honest limitations

• The AI sees the current version of changed files, not a line-by-line diff — so it reviews the code as it stands, not just the delta. (Real diff support is the top roadmap item.)
• Very large PRs are sampled, not exhaustively read.
• Browser-based means CORS is a fact of life: some enterprise network policies will require routing through a gateway.

What's next

The natural evolution is closing the loop: a button that posts the AI findings back to the PR as comment threads, batch mode for reviewing every open PR at once, and eventually wiring the same two-layer review into the intake automation pipeline — so by the time a human looks at a work item, the PR attached to it has already been stamped.

---

Gatekeeper is a single HTML file: one input panel, four compliance stamps, one AI verdict. Sometimes the best architecture is the one you can attach to an email.