Put a 0.25 second judge in front of your AI agents

Omer Saleem — Mon, 02 Nov 2026 13:00:00 +0000

Originally published at omerbinsaleem.com by Omer Bin Saleem (Omer Saleem).

Big models do the work. Around that work sit hundreds of small yes-or-no questions a day. A small judge model answered 116,097 of them in 20 days, in a median of 245 ms, and only acts alone above a confidence gate.

What changed

Model routing: the judge picks the model per message. The same report went from 137 s to 45 s.
Merge request reviewer: findings on code the MR never touched score 0.19 to 0.40, real ones 0.94. 16 of 26 false majors became follow-ups.
QA: it reads the page and judges 57 acceptance criteria. 94 passed, 38 went to a human, none guessed.
17% of judge calls failed in 20 days and nothing broke, because the old rule decides whenever the judge is down or unsure.

How to do it

List the small questions your agents answer with a full model run today: is this message asking me to stop, did this change cause this bug, which model should answer this. Each is a yes or no, a pick, or a score.
Put a small judge model in front of them. It answers a typed question with a probability in about a quarter of a second, instead of 5 to 14 seconds for a big model.
Set a confidence gate. Above 0.9 the judge acts alone. Below it, or when it is down, the old rule decides. Nothing new can break.
Keep the dangerous words on the strong model. Merge, prod, deploy and release always go to the big model, after the judge once sent a merge-to-prod to the small one and missed that the work was already merged.
Log every decision with its probability for 20 days, then read where it paid and where it was wrong before widening it.

Prompt to copy

Read my agent code and list every place a large model answers a yes/no, a pick-one, or a score question. For each: the question, how often it runs per day, and what a wrong answer would cost.
Propose which of them a small judge model could answer with a probability, the confidence gate above which it may act alone, and the fallback rule below the gate. Keep merge, deploy, production and release decisions on the strong model, always.

I am Omer Bin Saleem, a frontend chapter lead who builds with AI every day. One practical AI hack a day at omerbinsaleem.com. LinkedIn: omer-bin-saleem · Instagram: @iomerbinsaleem.

Lead a team of AI agents, not one chatbot

Omer Saleem — Sat, 31 Oct 2026 17:09:22 +0000

Originally published at omerbinsaleem.com by Omer Bin Saleem (Omer Saleem).

One lead you talk to, narrow agents that each do one job, and a gate in front of anything you cannot undo. This week the team got its own lanes, a QA runner, a rule book and a voice.

What changed

Lanes: every ticket gets its own worktree and its own run, up to three at once, so two tasks never mix.
A one-second brief: a small fast model (Jev) triages each new message for priority, size and owner before the big model starts, so clear work starts at once.
Live telemetry on the phone: who is working, which model each agent runs on, and a progress line with an estimate that says when it runs over.
QA without scripts: the tester writes a short steps file, one runner records the video and screenshots, and a playbook plus a lessons file make her better every run.
A librarian keeps every story's material and rule book, and stops work that conflicts with what the designer said.
Guards that cannot be talked out of: no push without my tap, Slack messages always in English, no secret hunting.
A voice: I speak Roman Urdu with English words mixed in, and Whisper on my Mac writes it down exactly.

How to do it

Split the work into roles: watch, investigate, doubt, build, review, test, guard.
Give every role exactly one job and a short prompt. A reviewer never writes code.
Put one lead on top that you talk to. It routes work and reports back.
Gate everything irreversible: pushes, merges and messages wait for your yes.

Prompt to copy

You are my team lead. You route work, own the gates and report; you never write code yourself.
Reply in the language I write in. Nothing is pushed, merged or sent to another person without my explicit yes.
When I ask what to focus on, read the watchers' notes and my open tickets and answer in five lines or fewer.

I am Omer Bin Saleem, a frontend chapter lead who builds with AI every day. One practical AI hack a day at omerbinsaleem.com. LinkedIn: omer-bin-saleem · Instagram: @iomerbinsaleem.

DEV Community: Omer Saleem

Put a 0.25 second judge in front of your AI agents

What changed

How to do it

Prompt to copy

Lead a team of AI agents, not one chatbot

What changed

How to do it

Prompt to copy