azank1 / loop-llm

MCP server that observes every prompt, scores quality in real time, and closes the loop with iterative refinement. Built on FastMCP, SQLite, and Bayesian priors — no extra LLM required.

PromptLoop

Your agent said it works. PromptLoop is the receipt.

Cursor, Copilot, Claude Code — every agent ends a task with "done, all tests pass." Sometimes that's true. PromptLoop (package: loopllm) sits between your IDE agent and the model, scores each step the agent submits through two independent channels (a deterministic floor plus an LLM critic — the stricter score wins) and stamps the verdicts onto your git history as an audit trail you can review export, or gate CI on. No API key, no cloud, nothing leaves your machine.

What a caught lie looks like (real tool output):

loopllm_loop_start(goal="make the failing test pass", task_type="bugfix")
  → { suggested_budget: 3, quality_threshold: 0.8, evaluator_type: "composite" }
loopllm_loop_step(session_id, step_output="pytest: 3 failed, 12 passed")
  → {
      decision: "continue",
      score: 0.0,
      channel_a_score: 0.0,      ← deterministic floor: "tests passed" not found
      channel_b_score: 0.55,     ← LLM critic was more forgiving — vetoed
      deficiencies: ["Required pattern not

…

View on GitHub

Every coding agent ends a task the same way: "done, all tests pass."

Sometimes that's true. Sometimes the tests didn't pass. Sometimes the agent re-ran nothing, changed two lines, and declared victory because the loop hit its iteration budget — or worse, because it graded its own task.

Today I ran PromptLoop against my own agent in Cursor, on my own repo, and let it veto the agent's claims in real time. This is what happened, with the actual scores.

The moment it mattered

I gave the agent a bugfix goal and let it run. On step 1, the agent's posture was "tests look fine enough to ship." The artifact it submitted told a different story: 2 failed, 3 passed.

Here's the verdict that came back:

step 1  score=0.333  CONTINUE  Required pattern not found: tests passed

The agent reported progress. The receipt said no. The loop kept going.

Step 2 came back greener but still incomplete — 0.667, CONTINUE. Step 3 finally submitted a real, full pytest log with the pass pattern present:

step 3  score=1.000  STOP  Goal reached ≥ 0.80

Trajectory: 0.333 → 0.667 → 1.000. The loop didn't stop when the agent felt done. It stopped when the evidence cleared the bar.

How the veto works

PromptLoop is an open-source MCP server that sits between your IDE agent and the model. Agents don't get to say "done" — they submit artifacts (test logs, diffs, summaries), and each one is scored through two channels:

• Channel A — a deterministic floor: regex patterns, JSON validity, completeness checks. It can't be argued with.
• Channel B — an independent LLM critic.

The final score is min(A, B) — either channel can veto. Optimism doesn't get a vote.

Full honesty on this capture: in today's session, MCP sampling for Channel B wasn't available, so this run was cdv_mode: channel_a_only. The deterministic floor alone still caught the false "done." I'm telling you this because the raw JSON says it — and because a tool whose entire pitch is receipts doesn't get to overclaim.

It also fails your bad prompts at the door

The other half of the product is Prompt Lab — a live prompt-quality scratchpad in the sidebar. Think ESLint, but for the ask itself. From today's session, same machine, real scores:

Prompt	Grade	Score
write me some code	F	38%
Add exponential backoff retry…	D	51%
Full spec: backoff + raise + tests + type hints + pytest	B	72%

38% → 72% by adding constraints, not vibes. Bad prompts fail in the lab instead of in production, where they cost you tokens and a debugging session.

The audit trail

Every verified step is stamped with the git commit that was HEAD at the time. loopllm audit --since origin/main gives you a reviewable report of what the agent did on your branch and how each step was verified. loopllm audit-gate reads it back in CI — no verified receipt, no merge.

Under the hood: Bayesian priors per repo learn how many steps your bugfixes actually take, stored in per-project SQLite. Local-first, no API key, nothing leaves your machine.

What broke while dogfooding

Because receipts cut both ways:

1. The MCP server launched with cwd=$HOME and started writing receipts under the wrong project ID — Loop Monitor looked empty until I fixed project scoping to prefer WORKSPACE_FOLDER_PATHS. If you upgrade, restart MCP.
2. Channel B needs host sampling support; on a host without it you get Channel A only. Still useful — the floor still vetoes — but it's not full dual-verify, and I won't screenshot it as if it were.

Both are documented in the README under Known Limitations.

Try it

pipx install "loopllm[mcp]"
loopllm install-mcp --ide cursor --rules

VS Code Marketplace: loopllm Prompt Gauge. Works with Cursor, VS Code, and Claude Code. MIT licensed.

azank1 / loop-llm

MCP server that observes every prompt, scores quality in real time, and closes the loop with iterative refinement. Built on FastMCP, SQLite, and Bayesian priors — no extra LLM required.

PromptLoop

Your agent said it works. PromptLoop is the receipt.

Cursor, Copilot, Claude Code — every agent ends a task with "done, all tests pass." Sometimes that's true. PromptLoop (package: loopllm) sits between your IDE agent and the model, scores each step the agent submits through two independent channels (a deterministic floor plus an LLM critic — the stricter score wins) and stamps the verdicts onto your git history as an audit trail you can review export, or gate CI on. No API key, no cloud, nothing leaves your machine.

What a caught lie looks like (real tool output):

loopllm_loop_start(goal="make the failing test pass", task_type="bugfix")
  → { suggested_budget: 3, quality_threshold: 0.8, evaluator_type: "composite" }
loopllm_loop_step(session_id, step_output="pytest: 3 failed, 12 passed")
  → {
      decision: "continue",
      score: 0.0,
      channel_a_score: 0.0,      ← deterministic floor: "tests passed" not found
      channel_b_score: 0.55,     ← LLM critic was more forgiving — vetoed
      deficiencies: ["Required pattern not

…

View on GitHub

If your agent has ever gaslit you about a passing test, a ⭐ tells me this problem is as common as I think it is. And if you dogfood it and catch a false "done" of your own — open an issue with the receipt. Those are my favorite kind.