The harness matters more than the model: building Ziya Code

I'm 17, and I build an AI coding tool called Ziya Code. This post is not a launch pitch. It's the part almost nobody writes about: the code around the model that decides what to do when the first answer is wrong. That layer is where reliability actually comes from, and most of it is plumbing you can build yourself.

Here's the thesis I keep coming back to:

A better model gives you a better first draft. A better harness gives you a correct final answer. Those are two different problems, and the second one is mostly engineering.

Ziya Code runs on Ziya's own model, but this post is deliberately not about the model. It's about the loop I wrote around it. Everything below is the harness: a plugin layer I built on top of an agent runtime, plus the gateway and auth that sit in front of it.

The one idea: escalate, don't one-shot

Most coding agents do the same thing for every request: send the prompt, take whatever comes back. That's wasteful on easy tasks and hopeless on hard ones.

Instead, Ziya Code treats effort as a ladder. Cheap path first. Only climb when the cheap path fails.

// Simplified escalation controller.
async function solve(task) {
  // Rung 0: just try it. Most tasks stop here.
  let attempt = await run(task, { reasoning: "normal" })
  if (await passes(attempt, task)) return attempt

  // Rung 1: ground it. Pull the real docs for the APIs it's touching.
  const docs = await deepen(task.symbols)
  attempt = await run(task, { context: docs, reasoning: "normal" })
  if (await passes(attempt, task)) return attempt

  // Rung 2: reflexion + a reasoning bump. Make it critique its own failure
  // before it rewrites, and give it more room to think.
  const critique = await reflect(attempt, task.tests)
  attempt = await run(task, { context: [docs, critique], reasoning: "high" })
  if (await passes(attempt, task)) return attempt

  // Rung 3: self-repair. Actually execute, feed the real traceback back in.
  for (let i = 0; i < MAX_REPAIRS && !(await passes(attempt, task)); i++) {
    const failure = await execute(attempt)      // real sandbox run
    attempt = await run(task, { context: [docs, failure.trace] })
  }
  return attempt
}

Why a ladder and not "always do the expensive thing"? Cost and latency. Something like 80% of requests are solved at rung 0. If you pay the deep-reasoning tax and run tests on all of them, you burn money and make the fast case slow for no reason. The ladder means the expensive rungs only ever run on the tasks that actually earned them.

Rung 1: stop it inventing APIs

The single most annoying failure mode of a coding model is a confident, wrong API call. It writes client.completions.stream() when the method is client.messages.stream(), and it does it with total conviction.

So I built a grounding step. A corpus of official documentation, chunked and embedded, that gets retrieved by the symbols the model is about to use.

• ~34,000 chunks of official docs, embedded and stored.
• Retrieval is keyed off the identifiers in the working set, not just the raw prompt, so it pulls docs for the exact functions being written.
• The context gets injected before the model commits to an API surface.

The unglamorous truth: getting the ingester right took longer than the retrieval. I shipped three separate bugs in the pipeline (bad dedup, a chunk-boundary off-by-one that split code samples in half, and an embedding step that silently skipped ~5% of chunks). Retrieval quality is downstream of ingestion quality, and ingestion is where all the boring bugs live.

And here's the honest part: grounding helped hallucinated APIs, but on its own it was not the biggest accuracy win. Which brings me to the thing that surprised me most.

I measured it, and I was wrong about best-of-N

I didn't want to trust vibes, so I ran an A/B on BigCodeBench (a benchmark of realistic, library-heavy Python tasks). Same model, different harness configurations.

Configuration	Pass rate
Baseline (one-shot)	~31%
Deep reasoning + self-repair	~41%
Grounding only	barely moved
Best-of-N (sample 5, keep "best")	did not beat self-repair

Two results I did not expect:

1. Grounding alone barely moved the number. It fixed a category of error (wrong APIs) but that category wasn't the bottleneck on this benchmark. Fixing the loudest bug is not the same as fixing the most common one.

2. Best-of-N lost. Sampling five answers and keeping the best sounds like free accuracy. It wasn't, and the reason is the whole lesson of this post: to keep the best of five, you need something that can tell which of five is best. If your selector is weak, you're picking the most confident wrong answer out of five wrong answers. That's not better, it's just more expensive.

The takeaway I now design around:

The verifier is the bottleneck. Spend compute on verification, not on sampling.

Self-repair wins for exactly this reason. It doesn't guess which answer is good; it runs the code and reads the traceback. The test execution is the verifier. A cheap, real correctness signal beats five expensive guesses every time. Once I framed it that way, the whole ladder made more sense: every expensive rung is really just a way to get a better signal and act on it, not a way to roll the dice more times.

The gateway: OpenAI-compatible, but honest about limits

The tool talks to the model through a gateway I wrote. It's a drop-in /v1/chat/completions endpoint, so anything OpenAI-shaped can point at it, but it does two things a raw model endpoint doesn't: it logs every turn, and it meters carefully.

The metering detail I'm weirdly proud of: input and output tokens are capped separately, per day and per week.

Why bother splitting them? Because they are not the same resource. One agent session on my own account ballooned to about 5 million tokens while it was setting up a browser automation environment, reading files, dumping logs, retrying. Almost all of that was input: context the user never typed. Output, the part the model actually generates, is the scarce and expensive resource. If you cap them with one combined number, a single context-heavy session nukes someone's whole budget on tokens they didn't even write.

// Every /v1 turn passes through here before we call the model.
function enforceQuota(user, estIn) {
  const u   = usage(user)   // rolling counters, read from the DB
  const cap = plan(user)    // per-plan limits

  if (u.inDay   + estIn > cap.inDay)   reject("daily input cap")
  if (u.inWeek  + estIn > cap.inWeek)  reject("weekly input cap")
  if (u.outDay  >= cap.outDay)         reject("daily output cap")
  if (u.outWeek >= cap.outWeek)        reject("weekly output cap")
}

// After the response finishes, record the real split (not an estimate).
function record(user, promptTokens, completionTokens) {
  bump(user, { in: promptTokens, out: completionTokens }) // updates day + week
}

A small UX decision that falls out of this: the app shows usage as a proportion bar, never a raw token number. People don't think in tokens, and a bar ("you've used most of today's allowance") is more honest about what it means than a number nobody can calibrate.

Auth: a device flow so the CLI never holds a secret

The desktop tool needs to act as you against your account, but shipping any kind of secret inside a binary that users can unzip is a non-starter. So sign-in uses the OAuth 2.0 device authorization grant, the same flow a smart TV uses.

1. Tool asks the server for a device code.
2. Tool shows you a short user code and opens the approve page.
3. You approve in the browser (where you're already logged in).
4. Tool polls until the server says "approved," then gets a token.

The binary never contains a credential. The token is minted only after a human approves the exact code shown on screen. One bug I hit and fixed: if you weren't signed in when you opened the approve page, the redirect to log in dropped the pending code on the floor, so you'd log in and land nowhere. The fix was to carry the code through the auth redirect so you come back to the approval instead of a dead end. Little flow bugs like that are invisible until a real user walks into them.

What I'd tell myself six months ago

• The harness is the product. The model is a component. The loop around it is where the reliability, the cost control, and most of the actual code live.
• Measure before you believe. Best-of-N sounds obviously good and lost my A/B. I would have shipped the wrong thing on intuition.
• Verification is the whole game. If you can cheaply tell right from wrong, everything else (repair, selection, escalation) becomes easy. If you can't, no amount of sampling saves you.
• Boring pipelines break in boring ways. My worst grounding bugs were all in ingestion, not in anything clever.

Being 17 mostly means I have time to be wrong a lot and keep going. Half of what's above started as an idea I was sure about and had to walk back after looking at the numbers. That loop, guess, measure, correct, is the same loop I built into the tool, which I don't think is a coincidence.

If you want to poke at it, Ziya Code is here: https://ziya.codes

Happy to go deeper on any part of this in the comments, the escalation ladder and the verifier-bottleneck stuff especially.