DEV Community: InferHaven

How I built a tutor that refuses to write your code

InferHaven — Wed, 29 Jul 2026 18:54:58 +0000

Two weeks ago I launched CodeTrain, an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote, and grades it.

People probably assume the rule is a prompt. Something like "do not write code for the user," pasted at the top of a system message, and done. That was the first version. It lasted about an hour.

What follows is what the rule actually cost to build: a grading contract instead of a chat reply, two separate model calls that must never be merged, two execution paths so the free tier costs almost nothing to run, and a short list of things that broke in front of real users.

A refusal is not prompted, its contracted

Here is the failure mode nobody warns you about. Ask a model to teach rather than solve, and it agrees enthusiastically. Then the learner gets stuck, and the model helps. It writes "you could try something like this" and drops in four lines. Technically it did not actually solve the exercise. Practically the lesson is over..

The fix to this was to stop asking for prose and start requiring a decision. Every grading turn returns JSON with a fixed shape, or it is treated as a failure:

{
  "verdict": "advance" | "retry" | "comment",
  "feedback_md": "<concise Socratic markdown>",
  "checks": [{"label": "prints exactly Hello", "pass": true}],
  "learned": ["<short concept the learner demonstrated>"]
}

That single change did more for the product than any amount of instruction tuning in the prompt. A model writing prose has infinite room to be helpful in the wrong direction. A model that has to pick advance or retry, and list the criteria it checked with a boolean next to each, has to commit to a judgment about the learner's code.

Before	After
Agreeable by default, because agreeing reads as helpful	Has to commit: advance, retry, or comment
Slides into writing the fix when the learner struggles	Every criterion carries a boolean the learner can see
No stable signal for the UI to render	Failing criteria are required, so the gap is visible
Impossible to tell a pass from a polite non-answer	The UI renders check marks instead of paragraphs

The checks array is the part learners actually respond to. Each item is one concrete criterion with pass or fail, and the prompt requires failing criteria to be included rather than quietly dropped. You see exactly which two of five things your code did, which is a very different experience from a paragraph explaining that you are on the right track.

Keep the two calls apart

Lesson authoring and grading are separate calls, with separate prompts, and merging them is the single worst thing you can do to a tutor like this.

I know because I tried it. One call, full context, plan the next step and grade the last one at the same time. It saves a round trip and it is meaningfully cheaper. It also produces a tutor that could write the answer into the question. When the same call that just saw a struggling learner also composes the next step, the step it writes gets suspiciously specific. "Now add the except KeyError branch that returns an empty list" is just the solution with a task label on it.

Separated, the authoring call never sees the failure and cannot overfit to it. That constraint is now enforced in the codebase itself: control-plane bookkeeping stays in the router, the tutor engine stays synchronised with the agent's copy of the prompt, and a captured-prompt test fails the build if the two drift apart.

The rule the grader needed most

Not "never write code." That part was easy. The hard rule was: do not be pedantic. Early graders would refuse to advance correct code because the learner had not exercised every branch at run time. A correct if/else is correct even if only one side ran. The prompt now says so explicitly, and instead of gating, it politely suggests an input that would show the other branch.

Two execution paths, and why the free tier does none of the work

A tutor that grades code has to run code. The obvious design puts a container behind every Run button, and it is the reason a lot of similar products have no free tier, or one with a queue in front of it.

Python and JavaScript run entirely in the learner's browser. Python goes through Pyodide in a dedicated worker, JavaScript runs in its own worker, and both stay pooled so the second Run does not pay startup again. A prewarm kicks off when the lesson opens, which usually finishes before anyone types their first line. Runs are capped at 20 seconds, which is generous for a teaching step and short enough that a runaway loop does not hang the tab.

On the free tier, the Python and JavaScript a learner writes never reaches a server, which is the only reason the tier can exist without a card and without a queue.

free tier economics

That holds for the two languages most lessons use. bash, ruby, perl and php genuinely need a real interpreter and a filesystem, so those go to an isolated server sandbox with per-minute and per-day caps, on free accounts too. The browser runtimes cover the common case; the sandbox covers what a browser cannot honestly fake.

The measured cost tracks either way. Across a recent two-week window the managed model spend for every free-tier session on the platform came to about 25 cents total. Compute for running the learner's code was zero, because it happened on their machine, in a tab they already had open.

The languages that cannot run in a browser fall back to a real sandbox on our infrastructure: bash, ruby, perl and php, with a real shell and real files, no network, and per-user rate and daily caps. That sandbox is Alpine with BusyBox, which produced its own class of bug. The grader would demand a GNU coreutils flag that BusyBox does not implement, then fail a learner whose solution was correct. The prompt now names that constraint directly and instructs the grader to accept any working BusyBox-compatible approach.

Dockerfile lessons get static linting instead of a real build. Letting anonymous users build containers on my infrastructure is a speedrun to an incident writeup, and the lesson value was in the file, not the image.

Things that broke in front of real users

If you are going to skip ahead to anything in this article, here is the interesting stuff.

The model emitted JSON that was not JSON. Specifically, feedback containing a stray backslash, usually from a regex or a Windows path in the learner's code, which invalidated an otherwise perfect verdict. The parser now repairs unescaped backslashes before parsing, and falls back to a retry carrying the raw text if it still cannot be read. A tutor that hard-fails because a learner typed \d is not a tutor that can function well.

Older model output used plain strings for checks instead of objects with a label and a boolean. Rather than break those, the coercion layer treats a bare string as a passing check. It is not elegant. It is the difference between rendering something useful and rendering an error.

The tutor asked permission to continue. Early versions ended a correct submission with "ready to move on?" Every single time, learners answered the question instead of coding, and the lesson turned into a conversation. Advancing is now the confirmation, stated as a hard rule in the prompt: when the submission is correct, advance, do not ask.

Browser Python is not Python. No subprocess, no real files, no network. A learner following an authored step involving subprocess was hitting a wall the tutor had built for them. The grader is now told to never fault the learner for a runtime limit, because that step should not have been written in the first place, and to guide toward a runnable approach or simply advance. I will be improving this behaviour into the future as well.

The pattern in all four

Every one of these is a case where the model needed to be guided to perform as a more acceptable tutor instead of just a sophisticated chatbot. Each one had a simple yet effective fix and I plan to be always improving CodeTrains effectiveness as a tutor.

Repo mode and the patch at the end

The free tier teaches on examples and any public repository you point it at. Paid repo mode runs against a real checkout on your machine, which raises a question I got wrong at first: what happens to the code you wrote?

The first design applied each step as its own commit. It felt responsive and it was a mistake. Nobody wants seven commits titled "step 3" in their history, and a half-finished lesson left the working tree in a state the learner had to clean up.

Now the lesson holds everything and proposes one patch at the end, containing only the code you typed, against a branch you choose. You review it like any other diff. If you abandon the lesson, nothing touches your repository at all.

One rule that has not moved

The repo reader refuses secrets. The denylist that keeps .env files and key material out of lesson context is mirrored in two places, the local agent and the control plane, and changing one without the other is treated as a bug.

What I'm tuning next

The tutor is at its best on a focused concept in code it can read directly. Lesson scope is the dial still getting tuned in: a tight lesson on one module lands harder than one that ranges across a whole framework, so more of the authoring work is going into keeping lessons narrow by default.

Browser Python runs on Pyodide, whose package set is a subset of PyPI. The prompt-level guardrails already steer authored steps toward what the runtime can actually import, and widening that coverage, including routing more lesson types to the server sandbox where it makes sense, is the next piece of that work.

The grading loop is a model making a judgment, so the design choice was to make the judgment inspectable. Every criterion is listed with a pass or fail rather than one overall score, which means you can always see the basis for a verdict instead of blind faith in the model itself. That visibility is also our own best feedback channel: every reported step comes back with the checks attached, which is how the tuning above gets prioritised.

Try to break it

The free tier runs in your browser, ten lessons a month, no card and no trial clock: codetrain.ai. Point it at a public repository you did not write and ask for a lesson on the part you understand least. That is the fastest honest test of everything above.

If you talk it into writing your code outright, send me the screenshot. I will fix the loop, and probably frame the screenshot!

-- Ethan L.

Originally published on the InferHaven blog. I'm the founder, and I'm in the comments.

I built a production AI agent for our NOC ticket queue in one shift

InferHaven — Fri, 17 Jul 2026 14:00:00 +0000

Our NOC gets a steady stream of tickets every day, and a good chunk of them are routine: the same site issue reported twice, a ticket that just needs the right category to be organized clearly, a slow-connection report that needs proper network diagnostics. None of that requires real judgment. All of it eats time.

I'd already been using OpenCode to just mess around and test things for free while on shift, so I decided to wire up an autonomous agent to reduce some of the repetitive workload of managing our tickets.

What it actually does

Each separate task of the workflow basically lives in a skill file, loaded by the OpenCode agent in correct order with an AGENTS.md to orchestrate the entire workflow, pretty standard.
A webhook receives a new ticket, waits sixty seconds to see if more come in (batching saves a lot of redundant work), then hands the batch to OpenCode running in non-interactive mode. From there it checks for duplicates against other open tickets for the same site and issue, categorizes the ticket by type, and for anything flagged as a slow connection, kicks off a deeper network diagnostic and posts the result straight to the ticket. It pings the team in chat only for tickets that actually need a human, which is still most of them now as its main job here to start is simply take on 80% of the initial ticket analysis and investigation workload.

What actually took the day

The AI part was the easy part, weirdly. What ate the hours was the plumbing underneath it. Duplicate detection needed exact-match site names, and site naming in our system turned out messier than expected, learned that one the hard way. The network diagnostic tool uses different naming conventions than the ticket system for the same physical sites, so there's a whole lookup step just to translate between the two. The first version had a five-minute timeout on the diagnostic step; turns out some sites take seven-plus minutes to fully analyze especially if there is an issue causing network slowdowns on the way. I bumped the default timeout to ten and made configurable via the .env file so I'm not guessing next time. And there's a fallback notification for when the agent itself crashes or runs out of budget, because a ticket disappearing silently is worse than the ticket never being automated at all.

Where it landed

It's running as a systemd service now, processing real tickets. Duplicates get caught and routed without anyone touching them. The diagnostic runs automatically on the tickets that need it. The team only sees what actually needs attention, and the total ticket count is already trending down since it went live.

The part that surprised me most wasn't that an AI agent could do this. It's that the whole thing, idea to running in production, fit inside one shift alongside the other daily workloads I always have going. A year ago this would've been a sprint's worth of work for a team, and half of it would've gone to exactly the plumbing described above, not the interesting part.

If you're running a similar agent, curious what your batching and timeout setup looks like. Feels like the kind of dead simple thing everyone's probably independently reinventing right now.

[Boost]

InferHaven — Thu, 16 Jul 2026 12:49:51 +0000

InferHaven

Jul 16

I built an AI tutor that refuses to write your code

#showdev #ai #learning #coding

5 min read

I built an AI tutor that refuses to write your code

InferHaven — Thu, 16 Jul 2026 12:00:00 +0000

Somewhere on your team right now, a pull request is getting approved by someone who couldn't rewrite it from scratch. Not because they're lazy. The assistant wrote it, the tests passed, the diff looked plausible, and the sprint doesn't stop for philosophy. Do that for a year and you get a team that ships faster every quarter and understands a little less of its own codebase every quarter too.

I ran into this a few times while building the start of InferHaven. The whole point of this company is that you shouldn't have to hand your code to someone else to get the benefits of AI. But there's a second thing quietly leaving the building, and no firewall catches it: the skill. When the model types and the human tabs through, the knowledge of how your system actually works stops living in your engineers and starts living in a vendor's autocomplete.

So I built the opposite tool. Today I'm launching CodeTrain: a hands-on AI trainer that teaches developers on their own codebase, and refuses to write the code for them.

**The learner writes every line. CodeTrain plans the lesson, sets up each step, runs your code, and grades the result, but it will not type your solution. Not even a "tiny" one-liner. When you're stuck it shrinks the step or gives you a sharper hint. It does not give you the answer, because the answer was never the point.

What a lesson actually looks like

You ask CodeTrain something real. "Walk me through adding a health check to this Dockerfile." "I don't actually understand our retry logic, teach it to me." It breaks that into a short lesson, usually two to six steps, each one small enough to hold in your head.

Each step is one concrete thing to write in a syntax-highlighted editor with a Run button. You type it. The code executes immediately: Python and JavaScript run right in your browser, shell and a few other runtimes run in an isolated sandbox, web stuff renders in a live preview. Then the tutor reviews what you wrote against the step's criteria, shows you ✓ and ✗ per criterion, and either advances you or asks the question that makes you see what's off. Fail a step multiple times and it doesn't dump the solution on you. It cuts the step in half.

$ codetrain start --mode repo
Session a6502084e8254276a394d5be578e9229 started (repo mode).
  Plan:     team
  Modes:    sandbox, repo
  Sessions: 0/1000 this month
  Usage:    0% of your monthly allowance (100% left)
  Models:   haiku, sonnet

Your tutor is open in the browser:
    http://127.0.0.1:7341
(workspace: /tmp/codetrain-repo-v555lhi7/repo)

Tutoring is live in your browser — keep this window open. Ctrl-C to stop.

Before the lesson: pick your level, how much help you want, and whether the session may build toward a real edit. The goal here is the health check from the terminal above.

That's repo mode, the paid tier. The tutor reads your actual repository, builds lessons out of your actual code, and at the end of a guided lesson (can run exploration only if desired) it proposes one integrated patch that lands the code you wrote onto your real branch. You review it like any other diff. Free tier does not include repo mode and drops you into sandbox practice instead: same tutor, same you-type-it rule, generated exercises from natural language.

Here is that health-check lesson, start to finish. Step 2 has you draft the probe on a scratch Dockerfile where mistakes are free:

You write the HEALTHCHECK on scratch first. Run gives you the linter, Send gives you the tutor's verdict against the step's criteria.

Step 3 moves the line you wrote into the real Dockerfile, all 360-odd lines of it, in the one spot where Docker will actually honour it:

Same instruction, real file. Placement is the lesson here: Docker only honours the last HEALTHCHECK, so it goes right before ENTRYPOINT/CMD.

Pass the final step and the tutor offers the one targeted patch it guided you on: the diff of what you typed, applied to your working tree only after you approve it.

Every criterion checked, then the patch-back: your change as a reviewable diff. A .codetrain.bak backup is saved before anything is written.

Change shipped. A real HEALTHCHECK lives in the repo, and the recap lists what you practiced, not what was generated for you.

This is not another assistant

Don't get me wrong. I use coding assistants daily and they're great at their job. Their job is producing code. CodeTrain's job is producing engineers who understand code, and those jobs basically pull in opposite directions.

AI assistant (Cursor, Copilot, Claude Code)	CodeTrain
You describe, it writes	It plans tiny steps, you write them
Output: working code, fast	Output: a developer who gets it
Understanding is optional	Understanding is graded, per step
Great when shipping is the goal	Great when the goal is your team

Use both. Seriously. Let the assistant ship the sprint while CodeTrain makes sure the person accepting its diffs could have written them.

"Code got cheap. Understanding didn't. Train the part that's still expensive."

An InferHaven product

InferHaven exists because your code is yours and should stay on your hardware. CodeTrain extends the same instinct to where your code lives secondly: wetware, brains, good ol' humans. Keep the code in-house, keep the skill in-house with it.

It's built the way you'd expect from us. The agent runs on your machine. Sandbox lessons execute in your browser, not on a stranger's box. Repo mode works against your local checkout. And if you'd rather run lessons through your own model account, bring your own key: Anthropic, OpenRouter, Bedrock, Vertex, or even an Ollama endpoint you host yourself!

What it costs, honestly

The free tier is real: the in-browser trainer, ten sessions a month, streaks and progress tracking, no credit card. It's capped because every managed session costs us actual inference money.

Pro is $24/month and adds repo mode on your real projects, fair-use session limits, sync across devices, and optional BYOK. Team is $32 per seat and adds the part managers keep asking me about: a dashboard showing who's ramping, per-seat budgets, central billing, and onboarding journeys for new hires. Enterprise is the usual conversation about SSO and code that can't leave your network. If that's you, sales@codetrain.ai.

Go type something

CodeTrain is live at codetrain.ai. Sign in, ask it to teach you something you've been faking your way around, and see how it feels to be the one controlling the keyboard again. There's a one-minute demo on the landing page if you want to watch before you commit to the free tier's grand total of... zero dollars.

InferHaven keeps your code yours. CodeTrain keeps the skill yours. Same haven, different cargo. The beacon is active!

— Ethan L.

This is cross-posted from the InferHaven blog. I'm the founder and I'll be in the comments; if you try a lesson and it teaches you something wrong or hands you the answer on a silver platter, I want to hear about it more than I want the compliments.

Building haven bench in the open, and the flaky CI ghost it flushed out

InferHaven — Wed, 24 Jun 2026 17:19:43 +0000

A debugging story from building InferHaven in the open: how a benchmark feature flushed out a flaky-CI race that had nothing to do with it.

I shipped a tokens/sec benchmark for local models. The unit tests were green, and then CI turned red in a way that looked like my fault. It wasn't. Here's the whole hunt: a chown that raced the tide, set -e, and a zsh lock file that vanished mid-sweep.

I built a small thing this week and it caught a bigger thing. That's the whole post, really. But the shape of how it happened is worth writing down, because it's the kind of story that usually gets quietly squashed into a one-line commit message and never told. Building in the open means showing the part where the harbor light flickers, not just the part where the boat docks clean.

The small thing is haven bench, a command that tells you how fast a model actually runs on your hardware. The bigger thing was a flaky CI failure that wore my new feature as a disguise.

The small thing: a number you can trust

If you run local models, you live and die by tokens per second. It's the single number everyone in r/LocalLLaMA trades like baseball cards, and yet most people read it off a vibe: "feels fast on my 3090." I wanted InferHaven to just tell you, honestly, on your own box.

$ haven bench qwen2.5-coder:3b --runs 3

  InferHaven bench — qwen2.5-coder:3b-instruct-q4_K_M
  run 1          106.0 tok/s
  run 2          106.3 tok/s
  run 3          106.0 tok/s
  generation     106.1 tok/s  <- decode rate (avg of 3 runs)
  prompt eval   3507.8 tok/s  (40 prompt tokens)
  load            0.17 s     (run 1: weights -> VRAM)
  total           1.39 s     (run 1)
  method       num_predict=128, seed=0, temp=0, runs=3

The headline is generation, the decode rate. Under the hood Ollama hands back its timings in nanoseconds, and the math is just eval_count / (eval_duration / 1e9). The reason that's the honest number and not, say, total is subtle and important: eval_duration excludes model load and prompt ingestion. So it's the pure speed of the model writing tokens, and it holds steady whether the model was cold or already warm in VRAM.

The first run pays to haul the weights into VRAM (that load line). The second run doesn't. If you quote "total time" you're really benchmarking your disk and your luck. Decode rate is the speed of the engine itself, the thing that's actually true about the model on your card. That's the figure worth screenshotting.

One more bit of honesty baked into the output: prompt eval is gloriously noisy on a short prompt (3507 tok/s above, but it bounces between runs), because you're dividing a tiny token count by a tinier duration. So bench reports it, but quietly. The number it puts in green, the one it wants you to believe, is generation. A benchmark that oversells itself is just a vibe with extra steps.

I wrote it the slow way on purpose: the core tokens/sec math by hand, with a unit test seeded from a real run off my RTX 3060, so I could actually explain every line of it instead of cargo-culting a one-liner. Seven assertions, all green. Shellcheck clean. Ran it live against three models. Pushed the PR.

And then CI turned red.

The red light

Two smoke jobs run on every PR: a slim codespaces stack and the full full-stack one. Full-stack went green. Codespaces failed, and not in my test. It failed bringing the container up, before my code ever ran:

Running the postCreateCommand from devcontainer.json...
Error response from daemon: container 46a4… is not running
postCreateCommand from devcontainer.json failed with exit code 1.
##[error]Process completed with exit code 1.

This is the moment that decides whether you're a good engineer or a fast one. The tempting move is: it's my PR, it's the only thing that changed Poke at the test, re-run it, add a sleep, wrap something in a || true and move on. But if you don't get to the root of the issue it will most likely just surface again.

No fix without a root cause first. A red test you "fixed" by re-running is just a bug you've agreed to meet again later, usually in front of a stranger.

So I did the boring thing instead and actually looked.

Following the evidence, not the vibe

The accusation was "your PR broke the build." The evidence disagreed, layer by layer.

What it looked like -- My PR broke CI

-Only my branch changed
-Red appeared right after I pushed
-It's the new feature, obviously
-Just re-run it / patch the test

What the evidence said -- My PR was a bystander

-My diff touched zero boot-path files
-Full-stack booted the SAME image fine
-The failure was in container startup, before my code ran
-main had failed this exact way before, intermittently

Three facts did the work. First, my diff touched a CLI command, a library function, a test, and some docs. Nothing in the container's startup path. Second, the full-stack job built the very same workspace image and came up clean; if my scripts could kill a boot, both jobs would die, not one. Third, the failure happened during up, before the step that runs my new code ever executed.

That's not a guilty feature. That's a flaky boot that happened to be standing next to me when the cops showed up.

The good news is I'd wired the CI to dump the dying container's logs on failure, and the container's last words were the whole case:

workspace-1 | chown: cannot access '/home/haven/.zcompdump-46a4482c870c-5.9.lock':
              No such file or directory

There it is. 46a4482c870c is the container's own ID. .zcompdump-…-5.9.lock is a zsh completion lock file, created and deleted in milliseconds while the shell builds its completion cache. And the thing that tripped over it was my entrypoint's first-boot ownership sweep: a recursive chown -R over the home directory.

The bug: a chown that raced the tide

Here's the entire bug, and it's a beauty because there's almost nothing to it:

set -e
# ...
chown -R "${HAVEN_USER}:${HAVEN_USER}" "${HOME_DIR}"

chown -R walks the tree, builds a list of things to change, then changes them. If a transient file (say, a zsh completion lock) exists when the walk lists it but is gone by the time chown reaches it, chown exits non-zero. And set -e says: any command that fails, abort the script. So the entrypoint dies. So the container exits. So forty seconds later, when the devcontainer tries to run its setup step, the daemon shrugs and says that container is not running.

It's a time-of-check versus time-of-use race, and like all races it only loses sometimes, which is exactly why main was usually green and only occasionally, mysteriously, wasn't. My PR didn't cause it. My PR just rolled the dice enough times to hit it.

The fix is the same guard the other recursive chowns in that file already had. I'd just missed these two:

- chown -R "${HAVEN_USER}:${HAVEN_USER}" "${HOME_DIR}"
+ chown -R "${HAVEN_USER}:${HAVEN_USER}" "${HOME_DIR}" 2>/dev/null || true

|| true tells set -e to let this one slide, and a vanished lock file stops being a death sentence for the whole container. I proved the mechanism in isolation first (a deliberately failing chown -R under set -e halts the script; the guarded version sails right past it), then shipped it as its own small PR, separate from the benchmark, so the history reads honestly: here's a feature, and here's an unrelated bug the feature flushed out.

The instinct that || true is "hiding errors" is usually correct. But here the error is the bug. A recursive chown racing a temp file is a known, benign class of failure, and the surrounding chowns in this same script already tolerate it. The band-aid would've been re-running CI until it was green. Tolerating a known race at the exact line it occurs is the actual fix.

What I actually want you to take from this

The benchmark is nice. Go run haven bench on your own card and post the number. Honest decode rates are good for everyone, and the more of them in the wild the less anyone has to guess.

But the part I think is worth more than the feature is the shape of the hunt:

A green unit test is not a green build. My math was perfect. The bug was three layers away from my math, in startup code I didn't touch.
Make your failures talk. That "dump logs on failure" step cost me five minutes to write months ago and handed me the entire diagnosis in one line. Future-you is a stranger; leave them evidence.
Resist the re-run. The single most expensive habit in software is treating a flaky test as noise. Flaky almost always means real bug, intermittent trigger. This one had been quietly failing for weeks.
Fix the root, label it honestly. The race got its own PR with its own explanation. Nobody reading the history six months from now has to wonder why a chown grew a || true.

That's the whole reason I'm building this in the open. Not because the wins make good screenshots, but because the misses are where the actual craft lives, and most of the industry hides them.

Both PRs are merged. The light's back to steady.

$ git clone https://github.com/InferHaven/inferhaven-core
$ cd inferhaven-core
$ cp .env.example .env
$ docker compose up -d
$ ssh haven@localhost
$ haven bench    # tell me what your card does

Float your boat up to the dock, clone the repo, run the benchmark, and if you want the managed version when it's ready, the waitlist on the homepage is the way in. The lighthouse is on. And now it doesn't flicker.

— Ethan L.