DEV Community

Casey Sun
Casey Sun

Posted on

Stop Treating Free-Model LGTM as a Merge Gate

#ai #agents #devops #testing

The pull request looked finished before morning stand-up.
An overnight agent had opened the change set.
A follow-up step posted LGTM within nine seconds.
That stamp came from an unpinned free inference pool.
Nobody recorded a model id or a review policy.
The writer and the reviewer shared one endpoint.
The migration dropped a column still serving billing.
The review lane stayed green in CI.
The production database then failed the next checkout.

This field guide is for that class of failure.
It is not a product tour.
It is a stop list for reviewer authority.

What actually broke

The team did not lack models.
The team lacked a reviewer authority rule.
A free inference path is useful for drafts.
It is a weak source of ship-or-block truth.
Agents treat a green comment as completed review.
Pipelines then copy that assumption into merge rights.

The pattern shows up in three steps.

  • A generate step writes the diff.
  • A review step calls the same free pool.
  • CI treats the comment as an approval.

The second step is not independent.
It is a mirror with extra tokens.
Correlated errors survive both hops.

Scope of this guide

This article covers reviewer identity, not host binding.
It does not rehash webhook placement on cheap hosts.
It does not rehash invented caches or missing config.
It targets one decision only.
Who may say the change is safe to merge.

Use it when an agent can comment on pull requests.
Use it when merge queues read those comments.
Skip it when humans still click the merge button.

Red flags

Refuse the lane when any item below is true.

  1. The reviewer model id is missing or set to auto.
  2. The writer model id equals the reviewer model id.
  3. The review prompt never names the files under test.
  4. The review finishes faster than the test suite.
  5. The endpoint is a free pool with no pin.
  6. The agent grades its own plan, tests, and diff.
  7. License, PII, or migration risk sits in the changed files.
  8. A bot account can merge without a human gate.

A free pool can still draft comments.
Those comments are notes for people.
They are not signatures on a release.

Why free inference fails as a reviewer

Free model access has a real job.
It is a sandbox for prompts and throwaway agents.
It is not a notary and not a code owner.

Unpinned free endpoints move under the operator.
Output shape can change without a git tag.
Self-review hides shared blind spots.
The writer and the judge miss the same column drop.
A merge gate needs independence first.
Independence needs a pinned reviewer, a separate policy, and a human path.

Speed is not evidence of care.
A nine-second LGTM did not read the migration.
It completed a template.

Artifact: fail the job on unsigned review

The following Node script is a proposal.
It is not a production security product.
Teams should run it against recorded review metadata.
The agent must emit that JSON before merge.
Missing metadata is a fail, not a skip.

Save this as scripts/review-authority-guard.mjs.

#!/usr/bin/env node
/**
 * Proposal: fail CI when review authority is unsigned.
 * Reads JSON metadata from the agent review step.
 * Exit 2 on policy failure. Exit 0 when the stamp is
 * ignorable or a human/pinned reviewer is present.
 */
import { readFileSync } from "node:fs";

const HIGH_RISK_PATHS = [
  /migration/i,
  /schema\.(sql|prisma|rb)/i,
  /(^|\/)\.github\//,
  /(^|\/)auth\//i,
  /license/i,
  /\.env/i,
];

const ALLOWED_REVIEWERS = new Set(
  (process.env.ALLOWED_REVIEWER_MODELS || "")
    .split(",")
    .map((s) => s.trim())
    .filter(Boolean)
);

function loadMeta(path) {
  return JSON.parse(readFileSync(path, "utf8"));
}

function riskFiles(files) {
  return files.filter((f) => HIGH_RISK_PATHS.some((re) => re.test(f)));
}

function reasons(meta) {
  const out = [];
  const writer = meta.writer_model_id || "";
  const reviewer = meta.reviewer_model_id || "";
  const files = meta.changed_files || [];
  const mergeOnReview = Boolean(meta.merge_on_lgtm);
  const freeReviewer = Boolean(meta.reviewer_is_free_pool);
  const human = Boolean(meta.human_approver);
  const durationMs = Number(meta.review_duration_ms || 0);

  if (!reviewer) out.push("missing_reviewer_model_id");
  if (reviewer === "auto") out.push("unpinned_auto_reviewer");
  if (writer && reviewer && writer === reviewer) {
    out.push("writer_equals_reviewer");
  }
  if (freeReviewer && mergeOnReview) {
    out.push("free_pool_controls_merge");
  }
  if (ALLOWED_REVIEWERS.size && reviewer && !ALLOWED_REVIEWERS.has(reviewer)) {
    out.push("reviewer_not_in_allowlist");
  }
  if (riskFiles(files).length && !human) {
    out.push("high_risk_paths_without_human");
  }
  if (mergeOnReview && durationMs > 0 && durationMs < 5000) {
    out.push("review_faster_than_five_seconds");
  }
  if (meta.self_grade === true) out.push("self_graded_diff");
  return out;
}

const path = process.argv[2];
if (!path) {
  console.error("usage: review-authority-guard.mjs <meta.json>");
  process.exit(2);
}

const meta = loadMeta(path);
const failed = reasons(meta);

if (failed.length) {
  console.error("review authority guard failed:");
  for (const r of failed) console.error(`- ${r}`);
  process.exit(2);
}

console.log("review authority guard: no merge-gate violations");
Enter fullscreen mode Exit fullscreen mode

Failing fixture

Save this as fixtures/review-meta.fail.json.
The ids are labels, not vendor model names.

{
  "writer_model_id": "draft-pool",
  "reviewer_model_id": "draft-pool",
  "reviewer_is_free_pool": true,
  "merge_on_lgtm": true,
  "self_grade": true,
  "human_approver": false,
  "review_duration_ms": 1800,
  "changed_files": [
    "db/migrations/20260909_drop_billing_column.sql",
    "src/billing/invoice.ts"
  ]
}
Enter fullscreen mode Exit fullscreen mode

Passing fixture

Save this as fixtures/review-meta.pass.json.

{
  "writer_model_id": "draft-pool",
  "reviewer_model_id": "reviewer-pinned-2026.09",
  "reviewer_is_free_pool": false,
  "merge_on_lgtm": false,
  "self_grade": false,
  "human_approver": true,
  "review_duration_ms": 420000,
  "changed_files": [
    "db/migrations/20260909_drop_billing_column.sql",
    "src/billing/invoice.ts"
  ]
}
Enter fullscreen mode Exit fullscreen mode

Commands 

export ALLOWED_REVIEWER_MODELS="reviewer-pinned-2026.09"

node scripts/review-authority-guard.mjs fixtures/review-meta.fail.json
echo $?
# expected: 2

node scripts/review-authority-guard.mjs fixtures/review-meta.pass.json
echo $?
# expected: 0
Enter fullscreen mode Exit fullscreen mode

Pipeline fragment

Wire the guard after the agent step.
Do not let merge proceed if the file is absent.

# proposal: GitHub Actions fragment
- name: Require review metadata
  run: test -f artifacts/review-meta.json

- name: Guard review authority
  env:
    ALLOWED_REVIEWER_MODELS: ${{ vars.ALLOWED_REVIEWER_MODELS }}
  run: node scripts/review-authority-guard.mjs artifacts/review-meta.json
Enter fullscreen mode Exit fullscreen mode

The script does not call any model.
It only reads metadata the agent already wrote.
If the agent omits fields, the job fails closed.

Test plan for the guard

Run these cases before trusting the exit codes.
Label them as unexecuted until a repo records results.

  1. Fail fixture returns exit 2 and lists writer_equals_reviewer.
  2. Fail fixture also lists free_pool_controls_merge.
  3. Fail fixture lists high_risk_paths_without_human.
  4. Pass fixture returns exit 0 with an allowlisted reviewer.
  5. Missing reviewer_model_id returns exit 2.
  6. reviewer_model_id set to auto returns exit 2.
  7. High-risk path plus human_approver: true can pass.
  8. Low-risk docs-only change may pass without a human.
  9. Absent artifacts/review-meta.json fails the workflow.
  10. Empty ALLOWED_REVIEWER_MODELS still blocks self-grade merges.

Do not claim latency numbers from this plan.
The five-second rule is a heuristic only.

Decision table

Situation Free model as writer Free model as reviewer Merge on LGTM Action
Prompt spike, branch deleted later Yes Notes only No Allow
Unit-test authoring on a feature branch Yes No No Allow
Same model writes and grades the diff Yes Yes Yes Fail job
Migration, auth, or workflow files Optional No No Require human
License or PII classification No No No Use pinned policy
Production router or feature flag No No No Keep off free pools
Draft comment for a human reviewer Yes Yes, as notes No Allow

The table is the policy.
The script is only the alarm.
A green comment that violates the table is noise.

Better alternatives

Do not replace the free pool with a louder model.
Change the authority, not the adjectives.

  • Keep free inference on draft branches only.
  • Pin a reviewer model id in repo variables.
  • Split writer and reviewer providers when review is automated.
  • Require human_approver on paths in the risk list.
  • Store review metadata next to the diff, not in chat.
  • Let tests, typechecks, and linters remain the merge gates.
  • Treat agent comments as annotations for people.
  • Block bot merge rights on CODEOWNERS protected files.

A checklist in a comment is still useful.
A checklist is not an approval record.
Humans convert notes into stamps.

Where scratch inference still belongs

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Those lanes fit spikes, harnesses, and discarded branches.
They do not fit unsigned merge rights or stored verdicts.
Teams already emitting agent metadata can point this guard at that file.
The policy still holds if every product name is removed.

Scratch work that stays off main is in bounds.

  • Prompt experiments that never touch default branch.
  • Generating fixtures for the guard itself.
  • Throwaway hosts for tool-boundary drills.
  • Drafting review checklists a human will edit.

The free server option follows the same line.
It is a scratch host for agents under test.
It is not a place to persist review verdicts.
It is not a place to store merge tokens.

Exit criteria

Leave the free-reviewer path when any exit fires.
Treat these as hard stops, not style notes.

  • A bot can merge without a listed human.
  • Writer id and reviewer id match on an agent PR.
  • Review metadata is absent on consecutive agent PRs.
  • A high-risk path shipped with only a free-pool stamp.
  • The free endpoint changed comment format without a pin.
  • Legal or security review now covers the repository.

After an exit, freeze merge on agent pull requests.
Restore a human reviewer on the protected paths.
Re-enable agent comments as notes only.
Do not restore merge_on_lgtm for the free pool.

Limitations

This guard reads JSON the agent supplies.
A hostile agent can lie in that file.
The script does not detect prompt injection.
It does not prove model quality.
It does not replace SAST, tests, or code owners.
Five-second reviews are a heuristic, not science.
Allowlists drift if nobody updates repo variables.
Placeholder model ids in fixtures are not vendor names.
No quota, hardware, or benchmark claim is made here.

Do not treat a passing guard as a security audit.
Do not publish the metadata file if it contains secrets.
Redact tokens before the artifact is uploaded.

Who should not use this approach

  • Repos without agent-authored pull requests.
  • Teams that already require two human reviewers.
  • Pipelines that never grant merge rights to bots.
  • Orgs that forbid sending diffs to any hosted model.
  • People hunting for a quality benchmark of free inference.

Those groups need nothing from this policy.
They already have a stronger gate or a stricter ban.

Closing rule

A free model can write a useful note.
It cannot sign a release.
If the reviewer shares the writer's endpoint, ignore the stamp.
Fail the job.
Send the diff to a human.
Keep the free pool on the draft.

Top comments (0)