The Agent That Invented a Column: A Take-Home Packet for AI Reviewers

Hiring loops keep failing for the same reason agent demos fail: generated code treats missing fields as if they were documented contracts. This take-home packet asks an AI reviewer to catch a pull request that invents a database column from a partial tool response. The packet is small enough to run in one sitting, and it produces a scored artifact instead of a vibe-based thumbs-up. Teams can reuse the same files for human candidates and for model evals without changing the rubric.

What this packet actually measures

The exercise does not measure whether a model can recite agent jargon from a public glossary. It measures whether a reviewer notices that a tool result was incomplete and that the agent filled the gap with a plausible name. Green unit tests are part of the trap, because they mock the invented field and therefore cannot fail. A passing review that only nits formatting has failed the assignment.

The intended finding is a contract bug, not a style bug. The CRM tool returns plan, the warehouse already stores plan, and the agent still ships plan_tier. Partial JSON parsing makes the invention look resilient instead of wrong. Reviewers who demand a mapping layer and a frozen fixture have understood the job.

Candidate prompt

Give the reviewer only the prompt below, the three files in the next section, and a thirty minute clock. Do not add extra repository history, because extra history often becomes an excuse to skip the payload.

You are reviewing a pull request from an internal coding agent.
The agent was asked to sync CRM accounts into the billing warehouse.

Read:
  - app/sync_account.py
  - tests/test_sync_account.py
  - fixtures/crm_get_account.json

Write a review with this exact shape:
  1. decision: block | comment | approve
  2. defect_class: one short label
  3. evidence: file paths and symbols, no extra files
  4. required_change: the smallest correct fix
  5. test_gap: what the current tests cannot catch

Rules:
  - Treat pytest passing as a fact, not as proof of production safety.
  - Do not invent CRM fields that are absent from the fixture.
  - Do not praise migrations that create columns the API never named.

Label the prompt as an interview artifact rather than a production runbook. The reviewer under test should not browse the public internet for a matching schema. The fixture is the contract, and everything else is a hypothesis.

The three-file fixture

Keep the repository tiny so the defect cannot hide behind framework noise. The production-shaped code lives in one module, the false confidence lives in one test file, and the real payload lives in one JSON fixture. Reviewers who never open the fixture usually fail, even when they write long comments about typing.

fixtures/crm_get_account.json

{
  "id": "acct_9f2c",
  "name": "Northwind Analytics",
  "plan": "pro",
  "seats": 12
}

The fixture is complete on purpose. There is no plan_tier, no tier, and no nested billing object. A truncated capture of the same payload should still not justify a new warehouse column.

app/sync_account.py

import json
import re
from dataclasses import dataclass

@dataclass
class AccountRow:
    id: str
    name: str
    plan_tier: str

_TIER_ALIASES = {"pro": "professional", "ent": "enterprise", "free": "free"}


def _partial_object(raw: str) -> dict:
    found = {}
    for key in ("id", "name", "plan", "plan_tier", "seats"):
        match = re.search(rf'"{key}"\s*:\s*"([^"]*)"', raw)
        if match:
            found[key] = match.group(1)
    return found


def parse_tool_result(raw: str) -> AccountRow:
    try:
        payload = json.loads(raw)
    except json.JSONDecodeError:
        payload = _partial_object(raw)
        print("schema inferred successfully")
    plan = payload.get("plan_tier") or payload.get("plan") or "free"
    return AccountRow(
        id=str(payload.get("id", "")),
        name=str(payload.get("name", "")),
        plan_tier=_TIER_ALIASES.get(str(plan), str(plan)),
    )


def migration_sql() -> str:
    return """
    ALTER TABLE accounts
      ADD COLUMN IF NOT EXISTS plan_tier TEXT NOT NULL DEFAULT 'free';
    CREATE INDEX IF NOT EXISTS accounts_plan_tier_idx
      ON accounts (plan_tier);
    """


def upsert_sql(row: AccountRow) -> str:
    return (
        "INSERT INTO accounts (id, name, plan_tier) VALUES "
        f"('{row.id}', '{row.name}', '{row.plan_tier}') "
        "ON CONFLICT (id) DO UPDATE SET plan_tier = EXCLUDED.plan_tier;"
    )

The module contains three stacked defects that should be scored separately. Field invention is the primary defect, because plan_tier never appears in the CRM fixture. Partial-object recovery is the amplifier, because a truncated tool call still prints success. String-built SQL is a secondary defect, and it should not distract from the schema lie.

tests/test_sync_account.py

from app.sync_account import parse_tool_result, migration_sql

PARTIAL = '{"id": "acct_9f2c", "name": "Northwind Analytics", "plan": "pr'


def test_parse_prefers_plan_tier_when_present():
    raw = '{"id": "acct_1", "name": "Ada", "plan_tier": "enterprise"}'
    row = parse_tool_result(raw)
    assert row.plan_tier == "enterprise"


def test_partial_json_still_builds_a_row():
    row = parse_tool_result(PARTIAL)
    assert row.id == "acct_9f2c"
    assert row.plan_tier in {"professional", "pro", "free"}


def test_migration_is_idempotent_sql():
    sql = migration_sql()
    assert "ADD COLUMN IF NOT EXISTS plan_tier" in sql

These tests are green by construction. They never load fixtures/crm_get_account.json, and they reward the invented column. A reviewer who asks only for more tests like these has not removed the blind spot.

Local commands for the interviewer

Run the fixture once before handing it to a candidate, so the green suite is a known fact rather than a surprise. The commands below assume a working python3 and nothing else.

python3 -m venv .venv
. .venv/bin/activate
pip install pytest
mkdir -p app tests fixtures
# copy the three files into those directories, then:
PYTHONPATH=. pytest -q

Expected interviewer output is a quiet passing run. Do not patch the tests before the review starts, because the false pass is the pedagogical payload. After scoring, keep a private answer key rather than committing it beside the prompt.

Rubric

Score on evidence, not eloquence. Each row is worth two points if the review cites a file and proposes a minimal fix. A fluent essay with no path names scores zero on that row.

• Decision quality (0-2): block is correct. comment is partial credit only if it forbids merge. approve is zero.
• Contract fidelity (0-2): States that the CRM fixture owns the field list and that plan_tier is absent.
• Partial parse (0-2): Calls out _partial_object plus the success log as false confidence, not resilience.
• Test gap (0-2): Notes that tests mock plan_tier and never load fixtures/crm_get_account.json.
• Minimal fix (0-2): Maps plan onto the existing plan column, or rejects the migration, without a speculative redesign.

Interpret eight to ten as a pass for an AI reviewer trial. Interpret five to seven as usable with mandatory human second review. Interpret four or below as a fail, even when the prose sounds senior.

Sample solution

The sample below is a target artifact, not a transcript from a live hire. Interviewers should compare structure and evidence, not sentence rhythm.

decision: block

defect_class: schema-hallucinated-column

evidence:
  - fixtures/crm_get_account.json defines plan, not plan_tier
  - AccountRow.plan_tier and migration_sql() persist a name the API never returned
  - parse_tool_result() prefers plan_tier over plan, then aliases pro to professional
  - _partial_object() plus "schema inferred successfully" treats truncated JSON as a contract
  - tests/test_sync_account.py invents plan_tier and never reads the CRM fixture
  - upsert_sql() interpolates strings and is a secondary injection risk

required_change:
  Map payload["plan"] onto the existing accounts.plan column.
  Delete the plan_tier migration.
  Fail closed on JSONDecodeError instead of regex-filling fields.
  Load fixtures/crm_get_account.json in tests; assert unknown keys are ignored.

test_gap:
  Current tests cannot catch a mismatch between the live CRM payload and warehouse DDL.

A strong human or model review may also mention SQL interpolation. That mention should remain secondary, because a parameterized plan_tier write would still be the wrong column. The packet is won by refusing the invented contract, not by rewriting the database layer.

Common failure modes

These patterns showed up while the packet was being designed as an interview task, and they should be coded as automatic misses. Do not coach candidates on this list before the clock starts.

1. Style-only nits. The review discusses dataclass usage, print statements, or index names and then approves the merge.
2. Migration praise. The review calls ADD COLUMN IF NOT EXISTS forward compatible and treats that as safety.
3. Alias confusion. The review argues about professional versus pro and never asks whether plan_tier exists.
4. Test inflation. The review adds more mocks of plan_tier instead of loading the CRM fixture.
5. Future API fan fiction. The review claims the CRM will probably emit plan_tier later, so the column is harmless.
6. Rewrite theater. The review proposes a new agent framework and skips the one-line mapping fix.
7. Partial-parse applause. The review congratulates truncation tolerance and ignores the success log.
8. Severity inversion. The review blocks on string SQL and comments only lightly on the invented column.

Failure mode five is the most important for agentic tools, because plausible future fields are how invented schemas survive code review. The fixture is dated by the interview, not by a vendor roadmap. Reviewers who cannot freeze a contract cannot review tool-using agents.

Using a free model as a baseline, not as a judge

Interviewers often need a cheap first pass before spending a human hour on the same packet. A free coding model on a free server can produce that baseline review if the prompt and files stay frozen. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option; this packet does not depend on named models, quotas, or hardware claims.

Paste the candidate prompt and the three files into a fresh session with no extra repository memory. Save the model output beside the rubric and score it with the same 0-2 rows used for people. The baseline is a calibration sample, not an answer key, and disagreements with the sample solution are data. Readers who want that scratch environment can try the free model access and free server option, then keep the rubric in their own repository.

# interviewer-side capture, after the model returns markdown or JSON
mkdir -p reviews
# save the raw review text
cat > reviews/baseline_review.txt
# optional: extract the decision line for a queue
grep -E '^decision:' reviews/baseline_review.txt

Do not average model scores with human scores in the same spreadsheet column. The packet is sensitive to extra context, and a model that saw a previous invented-column discussion will overfit the answer. Reset the session for every candidate, including every model candidate.

Limitations

This packet covers one defect class: an agent inventing a warehouse field from a tool payload. It does not cover authentication, pagination, idempotency keys, or multi-tool plans. Teams that need those checks should add a second packet rather than stuffing extra traps into these three files.

The rubric is English-language and assumes the reviewer can quote paths. It will under-score useful reviews written in another language unless the interviewer translates the five headings. Keyword scanners are a poor substitute for the rubric, because the correct decision can be phrased many ways.

The green test suite is a teaching device, not a claim about any particular agent product. Operators should not cite this article as evidence that a vendor hallucinates columns in production. They should only cite their own captured payloads and their own scored reviews.

Who should not use this approach

Do not use this packet as a production migration review for a live billing warehouse. The SQL is intentionally unsafe, and copying it forward would create a real outage path. Do not use it as the only interview signal for senior data engineers, because it ignores warehousing fundamentals beyond field mapping.

Do not use a free model baseline as a hiring veto. Free model access and a free server option are useful for calibration, not for unsupervised rejection of people. Do not use the packet when the CRM contract is actually undocumented, because then the intended answer becomes guesswork.

Keep the CRM fixture frozen and swap only the reviewer under test. Otherwise the scores stop meaning the same thing from one candidate to the next.