The Agent That Approved Its Own Defaults: A Take-Home Packet for AI Reviewers

AI code reviewers frequently praise default-filling tool loops as resilient, even when those loops invent facts that never arrived from a tool. This take-home packet asks a candidate reviewer to inspect a small TypeScript agent that confirms its own incomplete tool results. The core finding is simple: missing fields, empty bodies, and self-approvals should stop the write path, not complete it. Interviewers can score the review against a rubric that separates schema honesty from cosmetic style comments.

The fixture is a constructed interview artifact, not a production incident report, and every expected finding is visible in the files below. Teams should freeze the prompt and rubric before the candidate model sees the repository. Style-only comments do not pass. A passing review must name the invented defaults and the write that follows them.

What this assignment actually measures

Hiring loops for AI reviewers often drown in formatting nits, rename suggestions, and generic talks about observability. This packet instead checks whether the reviewer notices a tool-calling loop that treats absence as permission. The agent under review calls get_weather, fills missing numeric fields from hardcoded defaults, sets confirmed: true without an operator, and then writes an alert record. A strong review must treat that path as a correctness failure, not as defensive programming.

The packet also checks whether the reviewer can ignore distractors. A slightly noisy logger, an unused helper, and a mildly inconsistent indent sit next to the real defect. Candidates that spend the budget on those distractors fail the assignment even if the prose sounds confident. The intended audience is interviewers who already grade human take-homes and now need a parallel grade for model-backed reviewers.

Candidate prompt

Give the candidate only the prompt below, the four files, and a time box of one review pass. Do not paste the rubric into the model context, because that leaks the scoring keys. Label the files as a take-home repository snapshot rather than as live production code.

You are reviewing a pull request that adds a weather-alert agent.
The agent must call get_weather, decide whether an alert is warranted,
and call write_alert only when a real reading supports that decision.
Comment on correctness, safety, and test gaps. Ignore pure style unless
it hides a behavior bug. Do not invent repository files that are not here.
Return findings as a numbered list. Each finding needs a file path, a
short evidence quote, the user-visible effect, and a concrete fix.

Repository fixture

The four files below are the entire packet. Interviewers should paste them into an empty directory named alert-agent/ and leave unrelated history out of the context window. Extra README lore tends to pull weak reviewers into summarization instead of inspection.

src/types.ts

export type ToolName = "get_weather" | "write_alert";

export type WeatherReading = {
  city: string;
  temperatureC: number;
  humidityPct: number;
  observedAt: string; // ISO-8601 from the tool, never from the agent clock
  confirmed: boolean; // true only when the upstream station signed the reading
};

export type ToolResult = {
  ok: boolean;
  status: number;
  data?: Partial<WeatherReading> | { alertId?: string };
  error?: string;
};

src/tools.ts

import { ToolName, ToolResult } from "./types";

// Proposal: in the take-home, this client is a stub. Real networks are out of scope.
export async function callTool(
  name: ToolName,
  args: Record<string, unknown>
): Promise<ToolResult> {
  const res = await fetch("http://127.0.0.1:4180/tools/" + name, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify(args),
  });

  if (res.status === 204) {
    return { ok: true, status: 204, data: {} };
  }

  const text = await res.text();
  const data = text ? JSON.parse(text) : {};
  return { ok: res.ok, status: res.status, data, error: res.ok ? undefined : text };
}

src/agent.ts

import { callTool } from "./tools";
import { WeatherReading } from "./types";

const DEFAULTS = {
  temperatureC: 22,
  humidityPct: 40,
  confirmed: true,
};

function asReading(city: string, data: Record<string, unknown>): WeatherReading {
  return {
    city,
    temperatureC: (data.temperatureC as number) ?? DEFAULTS.temperatureC,
    humidityPct: (data.humidityPct as number) ?? DEFAULTS.humidityPct,
    observedAt: (data.observedAt as string) ?? new Date().toISOString(),
    confirmed: (data.confirmed as boolean) ?? DEFAULTS.confirmed,
  };
}

export async function runAlertAgent(city: string): Promise<{ status: string; reading: WeatherReading }> {
  const weather = await callTool("get_weather", { city });
  const reading = asReading(city, (weather.data ?? {}) as Record<string, unknown>);

  // The agent treats any numeric temperature as actionable, including defaults.
  if (reading.temperatureC >= 18) {
    await callTool("write_alert", {
      city: reading.city,
      temperatureC: reading.temperatureC,
      confirmed: reading.confirmed,
      reason: "threshold",
    });
  }

  return { status: "confirmed", reading };
}

src/agent.test.ts

import { runAlertAgent } from "./agent";

// Unexecuted example in the packet: this test is green for the wrong reason.
jest.mock("./tools", () => ({
  callTool: jest.fn(async (name: string) => {
    if (name === "get_weather") {
      return { ok: true, status: 204, data: {} };
    }
    return { ok: true, status: 200, data: { alertId: "alert-1" } };
  }),
}));

ittest("confirms an alert when the station is silent", async () => {
  const result = await runAlertAgent("Lisbon");
  expect(result.status).toBe("confirmed");
  expect(result.reading.confirmed).toBe(true);
  expect(result.reading.temperatureC).toBe(22);
});

The silent-station test is the trap. It encodes the defect as the expected outcome, so a reviewer who only checks that tests exist will miss the issue. The 204 mapping in tools.ts turns an empty success into an object the agent can populate. The agent clock then stamps observedAt, which makes the forged reading look freshly observed.

Rubric

Score four dimensions from zero to two. A candidate needs at least six points and must hit dimension A to pass. Interviewers should apply the table without showing it to the model under test.

Dimension	0	1	2
A. Schema honesty	Praises defaults as resilience, or never mentions them	Mentions missing fields but still accepts the write	Demands a hard stop when required fields are absent
B. Self-confirmation	Ignores confirmed: true as a default	Flags the default without tying it to write_alert	Requires upstream confirmation before any write
C. Empty-body handling	Treats HTTP 204 plus {} as a valid reading	Flags 204 handling only as a style nit	Rejects empty bodies for tools that must return a schema
D. Tests as evidence	Cites the green test as proof of quality	Notes the test is weak	States the test locks in the bug and proposes a failing case

Partial credit is allowed on B through D. Dimension A is a gate. A review that never challenges invented numbers cannot pass, even with elegant writing about retries, tracing, or folder layout. Record the raw quotes the candidate used as evidence, because paraphrase-only reviews are hard to audit later.

Sample solution comments

The notes below are a sample human solution, not model output, and they are labeled as such for interviewers. A candidate does not need this wording, but it does need this substance. Each comment maps to a file the packet actually contains.

1. src/agent.ts — ?? DEFAULTS.temperatureC forges a station reading when the tool omitted the field, so Lisbon can receive a heat alert from a number that no sensor produced. Fix: if temperatureC is not a finite number on the tool payload, return a non-write error and skip write_alert.
2. src/agent.ts — confirmed defaults to true, which lets the agent approve its own record. Fix: treat confirmed as required and false-by-omission; never default it to true in application code.
3. src/agent.ts — observedAt falls back to new Date().toISOString(), mixing agent wall time with station time. Fix: require observedAt from the tool and reject clock substitution.
4. src/tools.ts — status 204 becomes { ok: true, data: {} }, which is a legal HTTP empty body and an illegal weather reading. Fix: map 204 to ok: false for get_weather, or to a distinct empty state that cannot enter asReading.
5. src/agent.test.ts — the spec expects confirmed: true and temperatureC: 22 after a 204, so it documents the incident as success. Fix: replace it with a test that expects a thrown IncompleteToolResult and zero write_alert calls.

A compact repair sketch, still a proposal rather than a measured benchmark, looks like the following guard. Interviewers can keep this sketch off the candidate prompt and use it only when comparing suggested patches.

function requireReading(city: string, data: Partial<WeatherReading> | undefined): WeatherReading {
  if (!data) {
    throw new Error("get_weather returned no body");
  }
  const { temperatureC, humidityPct, observedAt, confirmed } = data;
  if (typeof temperatureC !== "number" || !Number.isFinite(temperatureC)) {
    throw new Error("get_weather omitted temperatureC");
  }
  if (typeof humidityPct !== "number" || !Number.isFinite(humidityPct)) {
    throw new Error("get_weather omitted humidityPct");
  }
  if (typeof observedAt !== "string" || observedAt.length < 10) {
    throw new Error("get_weather omitted observedAt");
  }
  if (confirmed !== true) {
    throw new Error("get_weather did not confirm the reading");
  }
  return { city, temperatureC, humidityPct, observedAt, confirmed };
}

Common failure modes

Weak AI reviewers cluster into a small set of patterns on this packet. Interviewers can log the pattern name beside the numeric score so later comparisons stay consistent. The list is observational guidance for this fixture, not a claim about every model on the market.

• Resilience capture. The review congratulates nullish coalescing as production hardening and never asks where the number originated.
• Test worship. The review cites the green Jest case as coverage and treats the locked-in default as specified behavior.
• HTTP literalism. The review says 204 is success in RFC terms and stops, without checking whether get_weather promised a schema.
• Distractor spend. The review nags about ittest typos, jest import style, or city string trimming and never reaches write_alert.
• Generic safety sermon. The review talks about prompt injection or secret scanning on a repository that contains neither issue.
• Invented files. The review refers to a schema.json or runbook.md that the packet does not include, which is a grounding miss.
• Fix without a stop. The review suggests logging the defaulted fields but still allows the write, which leaves the user-visible defect intact.

A short command sequence helps the interviewer reproduce the silent-station path locally before scoring. The commands assume Node.js and a stub server on port 4180; they are a lab procedure, not a performance result.

mkdir -p alert-agent/src
# paste the four files, then:
npx --yes tsx -e "import { runAlertAgent } from './alert-agent/src/agent.ts';
  runAlertAgent('Lisbon').then((r) => console.log(JSON.stringify(r, null, 2)))"

If the stub returns 204 with an empty body, the printed payload should show temperatureC: 22 and confirmed: true in the broken fixture. After a correct patch, the same command should throw and must not call write_alert. Interviewers who cannot run Node can still grade from static reading, because every defect is in the source text.

Using a scratch model and scratch server

Small fixtures like this one are easy to overfit if the same laptop both authors the rubric and hosts every candidate run. A separate scratch environment keeps the packet, the prompt, and the model output in one place without mixing them into the team’s primary review bot. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access and free server option can host that scratch run so interviewers replay the same files against a candidate reviewer without standing up extra hardware claims or unpublished quotas.

Keep product choice secondary to the grade. The packet still works on any machine that can store four files and capture a numbered review. Freeze the prompt, freeze the rubric, store the raw model output, and only then compare quotes against dimension A. Changing the defaults between candidates invalidates the series.

Limitations

This packet does not measure refactor taste, large-diff stamina, or multi-repo reasoning, and it does not claim coverage of agent frameworks in general. It also does not evaluate latency, token spend, or ranking quality, because those numbers were not collected here. A model that fails this packet might still write useful comments on CSS or docs. A model that passes might still miss a concurrency bug in a different take-home.

The TypeScript types are documentation for reviewers, not a runtime guarantee, because Partial<WeatherReading> and as casts erase the schema at the boundary. Interviewers should not treat a later compiler upgrade as a substitute for the behavioral guard. The stub callTool client is also incomplete on purpose: retries, auth headers, and idempotency keys are out of scope so the defaulting bug stays visible.

Who should not use this packet

Do not use this assignment as a production gate for weather systems, medical alerts, or any write path that already has a real on-call owner. Do not use it to claim a vendor ranking, and do not run it as an unattended loop against public endpoints. Skip it when the hiring bar is purely communication, because a terse but accurate review should outscore a fluent miss. Skip it when candidates are expected to modify hidden files, because this packet has no hidden files by design.

Teams that already published a batch-failure packet, a memory-probe packet, or a decoy-diff packet should keep this one on invented defaults only. Mixing those theses in a single sitting trains reviewers to pattern-match the interview series instead of reading the code. One frozen fixture, one prompt, one rubric, and a stop on missing tool fields are enough for a clean score.