First Token Is a Vanity Metric

First token is a vanity metric, and I burned too many evenings treating a prompt box like a demo. The first fifteen minutes of an AI coding tool are logistics, not conversation, even when the UI insists otherwise. If a patch never lands on a failing test, that chat was theater rather than engineering work. I stopped scoring replies and started timing the first diff a test could actually reject.

Have you ever watched a model apologize while git status stayed clean and nothing on disk had moved? We grade fluency because fluency is visible, and we ignore the repository because the repository does not glow. The friction in those opening minutes is not a missing temperature slider or a prettier theme. It is the missing contract between a proposed change and a test that is already red.

I used to open the chat like it was a showroom floor, then paste a wish and admire the nested bullets. Fifteen minutes later I had a tutorial in a sidebar and a still-empty repository staring back at me. The product felt smart, my repo felt untouched, and I could not tell which one I was supposed to ship. That gap is the whole developer-experience bug, and it hides inside a friendly text box.

The teardown is almost boring once you say it out loud in a quiet room. Minutes zero through three disappear into accounts, keys, and a homepage that wants a card before an assertion. Minutes three through eight disappear into an empty folder, because inventing the problem feels easier than writing a test. Minutes eight through fifteen disappear into a confident essay that never has to survive a local test runner.

The model did not fail you in that window; the clock you used failed you, because it scored speech. So the one fix that mattered for me was rude, small, and slightly embarrassing to admit. I refuse to send a prompt until a local test is already failing on my machine. I also refuse to call the session a trial until some patch has been scored by that test.

Think of it like a kitchen timer sitting on a pot you actually put on the stove. Steam is not dinner, and a token stream is not a diff, no matter how warm the UI feels. I keep a throwaway clock in a Node script so the ritual cannot slide back into vibes. The script seeds a failing test, stamps times, and writes a JSON receipt I can read without a chat UI.

Here is the seed I run when I am about to evaluate a coding assistant, and it is deliberately ugly. Pretty samples hide the setup tax, which is exactly the tax I am trying to see. There is no framework museum and no dashboard, just a function that returns the wrong number on purpose.

// dx-clock.mjs — proposed local workflow, not a vendor benchmark
import { mkdirSync, writeFileSync, readFileSync } from "node:fs";
import { execSync } from "node:child_process";
import { hrtime, stdin as input, stdout as output } from "node:process";
import { fileURLToPath } from "node:url";
import { dirname, join } from "node:path";
import { createInterface } from "node:readline/promises";

const root = join(dirname(fileURLToPath(import.meta.url)), "dx-scratch");
const stamp = (label, t0, extra = {}) => ({
  label,
  ms: Number((hrtime.bigint() - t0) / 1000000n),
  at: new Date().toISOString(),
  ...extra,
});

function runTest() {
  try {
    const stdout = execSync("npm test --silent", {
      cwd: root,
      encoding: "utf8",
      stdio: ["ignore", "pipe", "pipe"],
    });
    return { ok: true, stdout };
  } catch (err) {
    return { ok: false, stdout: String(err.stdout || err.message) };
  }
}

mkdirSync(join(root, "test"), { recursive: true });
writeFileSync(
  join(root, "package.json"),
  JSON.stringify({
    name: "dx-scratch",
    type: "module",
    scripts: { test: "node --test test/sum.test.js" },
  }, null, 2),
);
writeFileSync(join(root, "sum.js"), "export function sum(a, b) {\n  return 0;\n}\n");
writeFileSync(
  join(root, "test/sum.test.js"),
  `import { test } from "node:test";
import assert from "node:assert/strict";
import { sum } from "../sum.js";

test("sum adds two numbers", () => {
  assert.equal(sum(2, 3), 5);
});\n`,
);

That file is the entire product surface I care about in minute one of a trial. If your assistant cannot stand in that tiny room, why are you handing it a real repository? The clock itself is a JSON timeline, because memory is a lousy instrument when a UI is charming you. I record a handful of marks and I refuse to invent an extra one for how smart it felt.

const t0 = hrtime.bigint();
const events = [stamp("seeded_failing_test", t0)];

const red = runTest();
events.push(stamp("confirmed_red", t0, { ok: red.ok }));
if (red.ok) {
  throw new Error("Clock bug: the seed was supposed to fail, and it did not.");
}

const server = process.env.SERVER_URL;
if (server) {
  const res = await fetch(server, { method: "GET" });
  events.push(stamp("server_reachable", t0, { http: res.status }));
}

const before = readFileSync(join(root, "sum.js"), "utf8");
const rl = createInterface({ input, output });
await rl.question("Patch dx-scratch/sum.js so the test can pass, then press Enter...");
rl.close();

const afterSrc = readFileSync(join(root, "sum.js"), "utf8");
events.push(stamp("patch_claimed", t0, { fileChanged: before !== afterSrc }));

const green = runTest();
events.push(stamp("test_after_patch", t0, { ok: green.ok }));

writeFileSync(
  join(root, "timeline.json"),
  JSON.stringify({
    passed: green.ok,
    fileChanged: before !== afterSrc,
    events,
    note: "Green with no file change is a clock bug, not a win.",
  }, null, 2),
);
console.log(readFileSync(join(root, "timeline.json"), "utf8"));

Run it like a stopwatch, not like a booth demo with a borrowed laptop and a smile. The optional SERVER_URL is only a reachability ping, not a secret API I am pretending a vendor documented for me.

node dx-clock.mjs
# later, if you want the network hop on the receipt:
SERVER_URL="https://example.com/health" node dx-clock.mjs
cat dx-scratch/timeline.json

What should you see before the fifteen-minute mark, if you are being honest with the clock? A red test that you caused on purpose, then a tiny edit or an honest miss, then a second run you did not narrate by hand. If timeline.json shows a long gap before confirmed_red, you spent the trial on account chrome. If server_reachable is fine and test_after_patch is still red, you learned something true about the loop instead of something flattering about the prose.

I started pointing that optional URL at a throwaway environment instead of pasting keys into shell history like a trail of breadcrumbs. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am writing about a workflow I actually want, not a bake-off, and MonkeyCode's free model access plus a free server option is why I can rerun this clock without turning the trial into a billing subplot. The local script still works if you never set the URL, which is the point of a clock that does not worship the model.

Does this mean the model is irrelevant, and we should all go back to sticky notes? Of course not, and that would be a cheap ending. It means the model is downstream of a contract, the way a plane is downstream of a runway. A free server is useful here because I can throw the session away after the receipt, the same way I throw away dx-scratch when the tea is gone.

There are limits, and they are not subtle once you stop defending the metaphor. This clock does not rank models, and it does not prove a tool will survive a brownfield monolith with three package managers. Free access can change shape without warning, and a remote server is the wrong place for secrets, customer dumps, or anything your threat model would call a crown jewel. If you need production latency numbers, this is not your harness, and I will not pretend the JSON file is one.

Who should not bother with this ritual, even though it fits in a coffee break? Anyone shopping for a permanent platform from a fifteen-minute vibe check should walk away. Anyone who will not write the failing test first should walk away, because then the clock has nothing honest to score. Anyone who needs a vendor SLA, a specific model card, or an offline air gap this script does not offer should use a different review. And anyone hoping a timestamp will replace code review should keep hoping somewhere else.

I still catch myself opening a chat because the empty prompt feels like a start, which is how vanity metrics recruit you. Then I look at git status, and git status has not agreed to anything. The analogy I keep is an airport departure board on a delay you can feel in your feet. A glowing status is not a plane, and a token stream is not a merged change you could defend tomorrow morning.

Start the clock on the diff, and let the first fifteen minutes be rude enough to tell you the truth. If you try the scratch clock against a free server session, keep the receipt and throw away the folder.