Here is a scenario that happens constantly on teams using AI coding tools.
A developer is wrapping up before a standup. GitHub shows a PR open for 18 hours — 340 lines added to src/routes/auth.py, mostly AI-generated by Copilot. They open it on their second monitor, scroll through to the bottom in about four seconds, and click Approve. "Looks fine." PR merges. Status: approved.
Two weeks later someone asks: was this auth refactor actually reviewed?
The answer stored in every system in your stack is: yes.
The real answer is: not in any meaningful sense.
The gap between "approved" and "reviewed"
The problem is structural, not behavioral. Most teams do not have negligent developers — they have too many PRs, too many AI-generated diffs that look syntactically correct at a glance, and no system that distinguishes a 30-second rubber-stamp from a 20-minute line-by-line review. Both produce the same approved status.
For non-AI-generated code, this has always been a problem. For AI-generated code, it is a materially different one. Human-written code embeds the author's context — you can often infer the intent from the surrounding code, the variable names, the structure. AI-generated code can look completely idiomatic while doing something the developer who triggered it did not fully anticipate. The code is a downstream artifact of a prompt. Without the prompt context and a real review of what the model produced, "approved" is a label on a box you did not open.
What LineageLens captures when a review happens
LineageLens's backend has a POST /review/attest endpoint that records a human review as a signed attestation. The payload includes four fields that go beyond a simple verdict:
{
"scopeRef": "pr/12345",
"linesReviewed": 340,
"secondsOnDiff": 42,
"commentCount": 0,
"verdict": "approved"
}
linesReviewed, secondsOnDiff, and commentCount are the raw behavioral signals from the code review session. The endpoint then computes a depth_signal from these — not a replacement for the verdict, but an additional classification layer that runs on top of it.
The depth_signal formula
The full formula is documented in lineagelens-backend/app/services/human_review_service.py. It is intentionally transparent — if you are going to use a score to block merges, the thresholds should be auditable:
# Input signals
time_per_line = seconds_on_diff / max(lines_reviewed, 1)
comment_count = inline or PR comments left by the reviewer
lines_reviewed = AI-flagged lines the reviewer claims to have seen
# Scoring (0–100):
time_score = min(time_per_line / 5.0, 1.0) × 40 → max 40 pts at ≥ 5 s/line
comment_score = min(comment_count / 3.0, 1.0) × 30 → max 30 pts at ≥ 3 comments
coverage_score = min(lines_reviewed / 50.0, 1.0) × 30 → max 30 pts at ≥ 50 lines
# Bands:
# shallow raw_score < 35
# adequate 35 ≤ raw_score < 70
# deep raw_score ≥ 70
The time signal gets the most weight (40 points) because time-per-line is the hardest signal to fake without actually reading. A developer who spent 30 seconds on a 340-line diff clocked 0.088 seconds per line. To score maximum time points on that diff, they would need to spend 28 minutes on it — roughly 5 seconds per line.
The comment signal (30 points) captures engagement. A reviewer who left three or more inline comments clearly engaged with the diff content. Zero comments on 340 AI-generated lines in an auth file is a meaningful absence.
The coverage signal (30 points) is the least informative in isolation — linesReviewed is self-reported. But combined with the time signal, it creates a consistency check: if you claim to have reviewed 340 lines in 42 seconds, the time_per_line is 0.12 seconds. That floors the time_score and caps the achievable total regardless of coverage.
The rubber-stamp override
The most important rule in the formula is not the scoring — it is this:
# Implausibly-fast override: time_per_line < 1.0 s → always "shallow"
# (flags rubber-stamp approvals of large diffs, e.g. 3 s for 400 lines).
if time_per_line < 1.0:
return "shallow", 0.0
If the reviewer spent less than one second per line — regardless of comment count, regardless of lines_reviewed, regardless of verdict — the result is shallow with a raw score of zero.
This catches the scenario at the top of this article. 4 seconds on 340 lines = 0.012 seconds per line. The gate returns shallow. The signed attestation records it. A merge gate configured to require adequate or higher rejects the PR.
The threshold of 1.0 second per line is a judgment call, not a scientific constant. A genuinely fast reader who knows the codebase well might clock 2 seconds per line. A reviewer skimming for obvious errors might spend 1.5 seconds. The override is calibrated to catch clearly impossible review speeds — not to penalize fast reviewers who know what they are looking at.
What the attestation record looks like
After compute_depth_signal() runs, record_review() signs the result and persists two rows:
Attestation row:
subject_type: "review"
subject_id: "pr/12345"
statement_json: {
"reviewer": "user-uuid",
"lines_reviewed": 340,
"seconds_on_diff": 42,
"comment_count": 0,
"depth_signal": "shallow",
"depth_score": 0.0,
"verdict": "approved"
}
signature: <Ed25519 signature over canonical JSON>
public_key_id: <16-hex fingerprint>
HumanReviewAttestation row:
scope_ref: "pr/12345"
depth_signal: "shallow"
verdict: "approved"
attestation_id: <FK to Attestation>
The signed statement means the depth classification cannot be retroactively altered without invalidating the signature. The audit trail is not just what someone claimed — it is what the behavioral signals said, cryptographically bound to the review event.
This is a different kind of evidence than a closed PR. A closed PR tells you someone clicked approve. The attestation tells you how long they spent, what depth that corresponded to, and whether the classification was shallow, adequate, or deep.
The CI merge gate
The gate endpoint sits at POST /review/gate/{pr_ref}?min_depth=adequate. It is designed to be called by CI (X-API-Key auth, not JWT) at the point a PR is ready to merge:
GET /review/gate/pr/12345?min_depth=adequate
→ 200 OK
{
"prRef": "pr/12345",
"passed": false,
"reason": "Review depth 'shallow' is below minimum required 'adequate'.",
"depthSignal": "shallow",
"verdict": "approved",
"minDepthRequired": "adequate"
}
The gate passes only when depth_signal >= min_depth AND verdict == "approved". Three valid values for min_depth: shallow, adequate, deep. The depth ranking is ordinal: shallow=0, adequate=1, deep=2.
The practical effect: for high-sensitivity paths (auth, payments, security), you set min_depth=deep and require at least 70 points — which means roughly 5 seconds per line minimum, 3+ comments, and 50+ lines reviewed. That is a real review. For standard paths, min_depth=adequate at 35+ points is a reasonable floor that blocks 3-second approvals while allowing fast but engaged reviewers through.
ASCII flow diagram
Developer reviews AI-generated PR
│
▼
POST /review/attest
{ linesReviewed, secondsOnDiff, commentCount, verdict }
│
▼
compute_depth_signal()
┌─────────────────────────────────────────────────────┐
│ time_per_line = seconds / lines │
│ if time_per_line < 1.0 → return ("shallow", 0.0) │
│ time_score = min(tpl/5.0, 1.0) × 40 │
│ comment_score = min(cc/3.0, 1.0) × 30 │
│ coverage_score = min(lr/50.0, 1.0) × 30 │
│ raw = sum → band: shallow / adequate / deep │
└─────────────────────────────────────────────────────┘
│
▼
sign_attestation() [Ed25519]
persist Attestation + HumanReviewAttestation
│
▼
CI calls POST /review/gate/{pr_ref}?min_depth=adequate
│
┌────┴─────┐
│ │
passed blocked
(merge) (reason: "Review depth 'shallow' is below 'adequate'")
What this does not solve
The depth signal is a behavioral proxy, not a comprehension test. A developer who knows how to game the system can sit on a diff for 30 minutes, leave three generic comments, and achieve deep classification without actually understanding what the AI produced. This is not a flaw unique to this design — any measurable signal can be gamed. The value is not that the system is ungameable; it is that it sets a minimum bar that catches the most common failure mode (pure rubber-stamping) and creates an auditable record of the behavioral signals.
The formula is also calibrated for AI-generated code review, not general PR review. The 5-seconds-per-line maximum for time_score reflects the assumption that AI-generated code may require more careful reading than human-written code where you know the author's patterns. Teams reviewing dense algorithm implementations might reasonably argue the bar should be higher. The comment threshold of 3 might be too low for a 340-line diff. These are parameters worth arguing about — and they should be.
The connection to Risk Discovery
The depth_signal system exists because reviewStatus alone is not a sufficient filter. The full Risk Discovery query in LineageLens is:
lineagelens report . --unreviewed --category auth
Without the depth classification, "unreviewed" only catches code with no review record at all. With it, you can extend the definition: code where the only review on record is shallow is operationally closer to unreviewed than to reviewed. The filter becomes meaningfully stricter.
More at lineage-website.vercel.app. The Hashnode post goes deeper on the design tradeoffs I considered before settling on this formula.
One question for the comments: Is 1 second per line the right floor for the rubber-stamp override? What threshold would you set for auth-path code — and what signals am I missing that would make this more reliable?
Top comments (0)