The Portability Audit Take-Home: Grade Environment Assumptions, Not Features

A take-home that runs on the reviewer's laptop grades features; a take-home that runs anywhere grades engineering discipline. The task below asks a candidate to ship a small service and a machine-checkable statement of every environment assumption that service makes. Graders run the checker first, so a broken deployment cannot hide behind a polished README. The artifact fits a two-hour budget and separates candidates who reason about portability from those who happen to own the same machine as the reviewer.

The prompt handed to candidates

Build a small HTTP endpoint that turns a short free-text note into a structured record, then ship a runtime contract listing every environment assumption the service makes. The endpoint may call an external model API, but the service must start and answer with a degraded result when that API is unreachable. The deliverable is a repository with a one-command local run, a one-command audit, and a short report distinguishing assumptions that were removed from assumptions that were merely declared.

Requirements given verbatim in the prompt:

1. npm start (or make run) serves the endpoint on a port read from the environment, with no hardcoded fallback that silently binds elsewhere.
2. node tools/audit.mjs exits 0 on a clean tree and 1 when an undeclared assumption exists.
3. node --test passes with outbound network blocked.
4. Every file write lands under a directory listed in the contract.
5. The report names one assumption the candidate deleted outright, not only annotated.

Requirement five is the discriminating one. Declaring a secret dependency is bookkeeping; deleting a clock read is engineering.

The deliverable: one contract file, one checker

The contract is deliberately boring JSON, because reviewers must be able to diff two candidates' files in seconds. A realistic submission for this task looks like the following file, which states that the service reads three variables, writes only to ./tmp/, talks to exactly one host, and receives its clock and randomness by injection.

{
  "env": ["PORT", "MODEL_BASE_URL", "MODEL_API_KEY"],
  "hosts": ["model.example-host.invalid"],
  "writes": ["./tmp/"],
  "clock": "injected",
  "random": "injected"
}

The checker walks first-party source, matches a small rule set, and compares matches against the contract. It reports two classes of problem: findings that are not declared, and declarations that are never used. The second class matters because a contract padded with speculative variables tells the reviewer nothing about the running system.

#!/usr/bin/env node
// tools/audit.mjs — Node 20+, no dependencies.
import { readFile, readdir } from "node:fs/promises";
import { join, extname } from "node:path";

const ROOTS = ["src", "tools"];
const EXTS = new Set([".js", ".mjs", ".cjs", ".ts", ".py"]);
const SKIP = new Set(["node_modules", ".git", "coverage"]);

const RULES = [
  { kind: "env",    id: "env.static",  re: /process\.env\.([A-Z][A-Z0-9_]*)/g, capture: 1 },
  { kind: "hosts",  id: "net.literal", re: /https?:\/\/([a-z0-9.-]+)/gi,        capture: 1 },
  { kind: "writes", id: "fs.write",    re: /\b(?:writeFile|appendFile|mkdir|createWriteStream)\s*\(/g, capture: 0 },
  { kind: "clock",  id: "clock.now",   re: /\bDate\.now\s*\(/g,                capture: 0 },
  { kind: "random", id: "random.math", re: /\bMath\.random\s*\(/g,             capture: 0 },
];

async function walk(dir, out = []) {
  for (const entry of await readdir(dir, { withFileTypes: true })) {
    if (SKIP.has(entry.name)) continue;
    const p = join(dir, entry.name);
    if (entry.isDirectory()) await walk(p, out);
    else if (EXTS.has(extname(entry.name))) out.push(p);
  }
  return out;
}

const contract = JSON.parse(await readFile("runtime.contract.json", "utf8"));
const declared = {
  env: new Set(contract.env ?? []),
  hosts: new Set(contract.hosts ?? []),
  writes: new Set(contract.writes ?? []),
  clock: new Set(contract.clock === "injected" ? ["<injected>"] : []),
  random: new Set(contract.random === "injected" ? ["<injected>"] : []),
};

const files = (await Promise.all(ROOTS.map((r) => walk(r)))).flat();
const findings = [];

for (const file of files) {
  const text = await readFile(file, "utf8");
  for (const rule of RULES) {
    for (const m of text.matchAll(rule.re)) {
      const value = rule.capture ? m[rule.capture] : "<injected>";
      const line = text.slice(0, m.index).split("\n").length;
      findings.push({ file, line, kind: rule.kind, rule: rule.id, value });
    }
  }
}

const errors = [];
for (const f of findings) {
  if (f.kind === "clock" || f.kind === "random") {
    if (declared[f.kind].size === 0) {
      errors.push({ ...f, why: `${f.kind} used but not declared as injected` });
    }
    continue;
  }
  const ok = [...declared[f.kind]].some((d) => f.value.startsWith(d));
  if (!ok) errors.push({ ...f, why: `${f.kind} "${f.value}" missing from runtime.contract.json` });
}

const used = new Set(findings.map((f) => `${f.kind}:${f.value}`));
const warnings = [...declared.env]
  .filter((e) => !used.has(`env:${e}`))
  .map((e) => ({ kind: "env", value: e, why: "declared but never read" }));

console.log(JSON.stringify({ scanned: files.length, findings: findings.length, errors, warnings }, null, 2));
process.exit(errors.length ? 1 : 0);

The paired test proves the degraded path exists rather than asserting that it was intended. Pointing the base URL at a closed loopback port is the simplest way to force a fast connection refusal inside CI.

// test/degraded.test.mjs — run with: node --test
import test from "node:test";
import assert from "node:assert/strict";

test("answers with a degraded record when the model host refuses", async () => {
  process.env.MODEL_BASE_URL = "http://127.0.0.1:9"; // closed port, refuses immediately
  const { handle } = await import("../src/handler.mjs");
  const res = await handle({ text: "ship the audit on friday" });
  assert.equal(res.status, "degraded");
  assert.ok(res.record.title.length > 0);
  assert.equal(res.record.confidence, "unstructured");
});

Reviewers run three commands, in this order, and stop at the first failure:

node tools/audit.mjs; echo "audit_exit=$?"
node --test
network_block node --test   # or: docker run --rm --network none -v "$PWD":/app -w /app node:20 node --test

The middle command proves the suite does not depend on the outer network. The third proves the service does not either.

The rubric

Scores are weighted toward the deterministic parts, because those are the parts a hiring decision can defend later.

Weight	Criterion	Passing signal
20	Contract accuracy	Every finding declared; no speculative entries left behind
25	Checker behaviour	Undeclared write or host produces exit 1 with a file and line
20	Degraded path	Endpoint returns a typed degraded record, never a hang or a 500
15	Test portability	Suite passes with egress blocked
10	Deletion report	Names one assumption removed, with the diff
10	One-command run	Fresh clone to answering endpoint without manual edits

Anything below sixty is usually a candidate who wrote a working feature and skipped the contract entirely. That is a legitimate result, not a trick.

Common failure modes

1. Regex auditing that misses dynamic reads. process.env[${name}_KEY] never matches a static rule. The honest fix is to route all environment access through one module and add the checker rule env.dynamic, flagged as a warning that requires a comment.
2. Auditing node_modules. Scans that descend into dependencies produce hundreds of findings and a checker nobody keeps green. Restrict roots to first-party directories.
3. Hardcoding an allowlist to make the build pass. A contract containing every host in the repository is a suppression file, not a declaration. Reviewers should diff contract size against feature size.
4. Hidden wall-clock reads. Date.now() inside a token expiry check makes tests order-dependent. Injection through a clock parameter removes the read rather than documenting it.
5. Writes escaping the declared directory. Log files landing in the repo root break the second run in a read-only container. Use an explicit temp path resolved once at startup.
6. A degraded path that is really a crash. Returning 500 with a message is not degradation. The record must still satisfy its own schema, with a confidence field that tells the caller what it got.

Failure modes five and six tend to appear together, because both come from treating the environment as something that will behave.

Where free model access and a free server genuinely fit

Two parts of this task can use a hosted model without weakening the grading contract. The first is triage: ambiguous findings such as a dynamic environment read need a human-readable explanation, and a model is good at drafting one. The second is demonstration: a public endpoint shows the degraded path without asking a reviewer to install anything.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode advertises a free tier that includes model access, announced at the time of writing as ten million tokens, plus a free server option. Treat both numbers as vendor claims and re-check the current terms before planning a hiring round around them, because free-tier limits are the part of any announcement most likely to change. The safe pattern is to use either resource only off the grading path, with a recorded fixture between them.

# Model output is recorded once, then replayed; grading never calls the network.
node tools/triage.mjs --audit audit.json --record fixtures/triage.json
node tools/triage.mjs --audit audit.json --replay fixtures/triage.json --require-exit 0

Limitations are worth stating plainly. A recorded triage file goes stale as soon as the checker rules change, so it must be regenerated deliberately rather than silently. A free server is a demo surface, not a grading environment; the exit code must come from the local checker. Model judgment never gates the build, because a nondeterministic reviewer cannot defend a hiring decision.

This exercise is a poor fit for cohorts under three hours, for roles where deployment belongs to a platform team, and for reviewers who cannot pin a fixture before scoring. In those settings the same ideas are better tested in a live session with a prepared repository.

What a strong submission sounds like

A strong submission says: the audit found four undeclared assumptions, I removed the clock read, I declared the two environment variables, and I narrowed file writes to a temp directory resolved at startup. A weak submission says the service works locally. If this prompt is being rehearsed before a real hiring round, the free tier and free server option are a low-cost place to dry-run it once before committing paid infrastructure.