DEV Community: InferHaven

Adding error monitoring nearly leaked my users' API keys

InferHaven — Thu, 13 Aug 2026 08:42:56 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

I found four places where my app fails without telling anyone. Fixing them meant sending those failures to Sentry, and the test I wrote to prove the fix worked failed for a reason I did not expect at all: the report would have carried the learner's whole conversation and a decrypted API key along with it.

The near miss turned out to be the more useful half, but first the bug.

Project Overview

CodeTrain is an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote and grades it. The control plane is FastAPI on Fly, Postgres on Neon, and it has had Sentry wired in since June.

"Wired in" is roughly just that. The entire integration was two calls: sentry_sdk.init() gated on a DSN, and one capture_exception inside a catch-all handler. No custom tags or spans, and no context beyond the environment name. It had never caught anything, which means there was no obvious issues right?

Bug Fix or Performance Improvement

None of the four is sloppiness. Each one is a piece of defensive code doing exactly what it was written to do, but with invisibility as an unintended side effect.

1. Every model provider failure. run_managed_turn is the single path every metered model call goes through. It catches provider exceptions and re-raises them as a clean 502:

except Exception as exc:
    raise HTTPException(
        status.HTTP_502_BAD_GATEWAY, f"model provider error: {str(exc)[:300]}"
    ) from exc

That is good API behaviour and a total blind spot. FastAPI routes HTTPException to its own handler, which is not my catch-all handler, so capture_exception never sees it. Every provider outage this product has had was a clean 502 for the learner and complete silence for me.

2, 3 and 4. Three places that swallow malformed model output. When a model returns something that is not the JSON the prompt asked for, the parser falls back to a plausible default:

except (ValueError, json.JSONDecodeError):
    raw_steps = []

No log line. The learner gets a lesson with no steps, or a "retry" verdict assembled from raw model prose, and nothing anywhere records that the model went off contract.

This is the same shape as the bug your test cannot see, which I wrote about last week: a failure that throws tells you where it is, and a failure that returns a plausible value does not. You cannot grep for something that was never written down.

Code

The provider site now reports before it converts. Three lines:

except Exception as exc:
    observability.capture_exception(
        exc, model_tier=model_tier, surface=surface,
        provider="managed" if used_managed or key_provider is None else key_provider,
        session_id=session_id or "none", repo=str(repo),
    )
    raise HTTPException(
        status.HTTP_502_BAD_GATEWAY, f"model provider error: {str(exc)[:300]}"
    ) from exc

The three parse sites were harder, because of a constraint specific to my codebase. The tutor engine exists twice: once in the control plane, and once in a CLI agent that ships as a tarball and has no Sentry wired in. Importing a control plane service into the shared engine would have broken that. So the engine declares a hook and defaults it to nothing:

def on_parse_failure(site: str, **meta) -> None:
    """Called when model output could not be parsed and a fallback was returned instead.

    A no-op by default, and deliberately so: the engine stays free of any control-plane
    dependency, and importing it bare must never need Sentry. `app.main.create_app`
    rebinds this to `services.observability.note_parse_failure`.
    """

and each fallback reports through it:

except (ValueError, json.JSONDecodeError) as exc:
    on_parse_failure("course", chars=len(text), lang=lang,
                     found_object=isinstance(exc, json.JSONDecodeError),
                     error=_parse_error(exc))
    raw_steps = []

Note what is in that payload and what is not. Length, whether a JSON object was located at all, and json's own positional complaint. Never the model's output. That constraint is the whole next section.

My Improvements

What actually changed:

	Before	After
Provider failures	Silent 502	Reported, tagged, rate limited
Malformed model output	Silent fallback	Reported with a positional reason
Frame locals	Prompt and key attached	Never transmitted
Sentry config	Inline at the init call	One shared dict, prod and tests
Tutor turns	Untraced	19 spans, tokens, cost

Plus 24 tests, six of them on the rate limiter alone, and a script that reproduces all four failures on demand.

Best Use of Sentry

I wrote the tests for this against a real Sentry client with a capturing transport, rather than mocking my own wrapper. A mock would only have proven that my code calls the function I told it to call. I wanted to see the actual bytes.

The test asserted that a captured provider failure contains no prompt and no key. It failed immediately.

Sentry attaches every stack frame's local variables to an exception event, and send_default_pii=False does not seem to cover them. That flag governs request and user data. Frame locals are a separate option, include_local_variables, and it does default to on.

Look at where that lands. The frames around a failed provider call hold messages, which is the learner's entire conversation, and org_creds, which is a customer's decrypted BYO API key.

Here is that event, sent with the SDK's default configuration:

key_override in plaintext. org_creds in plaintext, same key again. The full conversation. The system prompt.

The part that almost fooled me

My first version of that reproduction named its variables api_key and credentials. Sentry came back like this:

[Filtered], [Filtered]. If I had stopped there I would have written a very confident paragraph about how Sentry protects you automatically, and I would have been wrong.

Sentry's server-side scrubber matches on field name. api_key and credentials are on its list. org_creds and key_override are not, and neither is messages. Two things follow. The protection evaporates the moment you rename a variable, and it happens after transmission anyway, so it is redaction at rest rather than a control over what leaves your server.

Look again at that second screenshot, though. prompt is sitting there in plaintext, directly underneath two [Filtered] rows. Even in the case where the scrubber works, it never had any opinion about user content.

The fix

One option, in one place, shared by production and the test suite so they cannot drift:

def client_options(settings) -> dict[str, Any]:
    return {
        "environment": settings.app_env,
        "traces_sample_rate": settings.sentry_traces_sample_rate,
        "send_default_pii": False,
        "include_local_variables": False,
    }

Same failure, same frame, after:

The test that found this generates its secrets at runtime, so they cannot appear in the source context Sentry also attaches, and it fails if anyone flips the option back:

def test_frame_locals_never_reach_sentry(monkeypatch):
    prompt = "learner asked: " + uuid.uuid4().hex
    api_key = "sk-live-" + uuid.uuid4().hex
    ...
    assert prompt not in blob, "the learner's prompt reached Sentry via frame locals"
    assert api_key not in blob, "a decrypted BYO API key reached Sentry via frame locals"

I checked it is not a vacuous guard by flipping the option back and watching it go red.

Keeping the events useful

Scrubbing locals is only correct if what remains still tells you something. Tags carry the operational shape of the turn, and a parse failure carries enough to diagnose it without a byte of model output:

Expecting ',' delimiter (char 52) is json's own positional message. It says where the model's output broke without repeating any of it.

Not spending the month's quota on one bad afternoon

I am currently on the free plan, 5k errors a month, and Sentry's own per-key rate limiting is a paid feature. A provider outage fails every turn at once, so uncapped, one bad afternoon spends the month and then Sentry drops everything afterwards silently. That is the bug I just fixed from the perspective of my codebase.

So the ceiling lives in my code: a token bucket per failure kind, five events immediately because an incident should be visible on its first failure, then one an hour while it continues. A simulated 24 hour outage of 86,400 failures sends only 28 events.

The cap is only allowed to be quiet about volume, never about the fact of it. Every event that gets through reports what it held back:

201 failures, 6 events, and the last one says 195. The local log line still fires every time, because Fly's logs are free and an incident should stay fully reconstruct-able.

Agent tracing

The tutor is an agent loop, so I wrapped one turn in a manually created transaction. Sentry's AI instrumentation picked up the model call on its own:

Nineteen spans for one turn: the entitlement checks, the budget queries, the model call, the usage insert. gen_ai.chat, claude-haiku-4-5, 971 in and 248 out, cost to four decimal places, context utilisation at 1%.

Worth noticing what the Input panel says on that span: "No input for this span." The trace tells me the shape of the turn, which step, which model, how long, what it cost. It does not tell me what anybody typed, which is the same boundary everything else here is drawn on.

That is also why the transaction is created by hand rather than by raising the global sample rate. Raising it auto-instruments every request the app serves, which is unbounded volume for visibility I only want on model turns. Production runs at 0.0, so the transaction is created unsampled and never sent, and the traces above come from a dev environment. My privacy policy discloses Sentry for error monitoring, and I would rather keep the code inside that promise instead of widen it.

About the impact, honestly

I can tell you exactly how many learners have been hit by the parse bugs, because the fallback feedback is persisted and I can count it with a query like:

SELECT count(*) FILTER (WHERE r->>'fail_md' LIKE '%"verdict"%') AS parse_fallbacks
FROM study_sessions s
CROSS JOIN LATERAL jsonb_array_elements(coalesce(s.state::jsonb -> 'record', '[]'::jsonb)) AS r;

And the grand total of real customers affected = Zero. Across every completed step since late July. I then ran a real lesson through the instrumented build, four model calls including a wrong answer and a follow-up question, and got zero there too.

I could have left that out. It is a better story if the bug was hurting people. But the honest result is that instrumentation turned "I have no idea whether this happens" into a number, and the number is zero, and knowing that is worth something on its own. The parse sites were structurally unable to report, which is the bug, whether or not it has fired yet.

The leak is the one with teeth, and even there I want to be precise. My catch-all handler has had include_local_variables on since June, so the exposure was real for about seven weeks. I audited every event in the project before publishing this. Nothing sensitive had been captured. It never fired in a frame that held a real key, so nothing needs rotating.

What I would tell you to go check

If you run the Python SDK with send_default_pii=False and assume that covers you, open any exception event you already have and expand a frame. Everything in scope at the moment it threw is in there.

Then check it with your real variable names, not with something called api_key. That was the difference between a paragraph that was wrong and one that was right, and it took one extra test to find out which one I was writing.

The bug your test cannot see

InferHaven — Mon, 10 Aug 2026 07:15:16 +0000

Here is a function. It has one job: give me the three highest scores.

function topThree(scores) {
  scores.sort((a, b) => b - a)
  return scores.slice(0, 3)
}

It works. topThree([72, 98, 64, 91, 87, 55]) returns [98, 91, 87], which is correct. It throws nothing. It has a test, and the test is green.

Now watch what happens when real code calls it.

const names  = ["Ada", "Bo", "Cy", "Di", "Eve", "Fay"]
const scores = [  72,   98,   64,   91,   87,    55  ]

const pair  = (n, i) => n + " " + scores[i]
const board = () => names.map(pair).join("   ")

console.log("before:   ", board())
const top = topThree(scores)
console.log("after:    ", board())

before:    Ada 72   Bo 98   Cy 64   Di 91   Eve 87   Fay 55
after:     Ada 98   Bo 91   Cy 87   Di 72   Eve 64   Fay 55

Ada scored 72. Ada now has 98. Every name in that list is sitting next to somebody else's score, and the function that did it also returned exactly the right answer.

Why the test could not have caught this

The obvious reaction is that the test was lazy. Write a better one.

That reaction is wrong in a way I think is worth analyzing, because it is the difference between "we got sloppy" and "this class of bug needs a different kind of attention."

Array.prototype.sort sorts in place. It rearranges the array you handed it and then returns that same array. So the topThree function shown above has two effects: the value it returns, and the change it makes to its argument. The original test only ever looked at the first one.

A test of the return value cannot observe a side effect. Not because it was written badly, but because a return value is not where the side effect resides.

You could write an assertion that catches it. expect(scores).toEqual([72, 98, 64, 91, 87, 55]) after the call would go red immediately. But you only write that assertion if you already suspected the function touched its argument, and if you already suspected that, you would have fixed the function instead.

That is what makes this expensive. A bug that throws tells you where it is. A bug that fails a test tells you where it is. This one returns the right answer, passes review, and shows up three weeks later as a support ticket about the wrong name on a leaderboard.

The near miss that catches experienced people

The fix is to sort a copy instead of the original.

return [...scores].sort((a, b) => b - a).slice(0, 3)

Here is the version I find genuinely interesting, because it looks like the fix and is not:

return [...scores.sort((a, b) => b - a)].slice(0, 3)

Same characters, different order. The spread is on the outside, so .sort() runs first, on the caller's array, and by the time the copy happens the damage is already done. It returns the correct answer and mutates, exactly like the original.

I have seen people who have been writing JS for a decade write that one, and its not a knowledge gap. It is that the two versions look identical at a glance and only one of them is about ordering of operations.

toSorted, and why it is not the whole answer

toSorted is the modern answer and it does the right thing: it returns a new sorted array and leaves yours alone. It landed in ES2023, so Node below 20, Safari below 16 and Chrome below 110 throw on it. [...arr].sort() is the version with no version requirement.

This is roughly why lodash `orderBy` exists

If you have ever wondered what a utility library is buying you when the language already has sort, this is a large part of the answer. _.orderBy(scores, [], ['desc']) sorts a copy and hands that back. The array you passed in is never touched, and that guarantee is the product.

The naming does not help anyone. sort, reverse, splice, push and pop all change the array you call them on. map, filter, slice and concat all give you a new one. Nothing in those names really tells you which is which, so it comes down to memory, and memory is exactly what fails at the worst times.

Why I turned this into the demo on my landing page

I launched CodeTrain three weeks ago. It is an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote and grades it.

The thing I got wrong was assuming that if people heard about it, the ones who wanted it would sign up. So I did the whole distribution playbook. What I actually had was a funnel where people arrived, hit an email field, and had no way to find out whether the thing was any good before handing it over.

So the lesson above is now the demo, and it runs on the landing page with no account and no install.

It is a real lesson, not a video of one. Your code executes in a real Web Worker in your own tab. The checks are real assertions against what your function actually returned and what it did to its argument, including one case you cannot see, because a fix that only works on the numbers in front of you is not a fix. Break the slice and the check goes red. Delete the function and you get a real missing-entry error.

The one thing that is not live is the tutor's replies. Those are written in advance, the page says so before you start, and every reply is labeled as such. A preview that implies a live model when there is not one is a lie about the product, and I would rather lose the visitor than start there.

The bit I nearly shipped backwards

The first version of that demo had a problem I did not see until the fourth or fifth read.

Step one showed a passing check next to broken code, and the tutor said something like "the check did not catch it." I meant it as a statement about tests in general. What it actually reads as, on my own product page, is that CodeTrain wrote a check that misses bugs.

The fix for the demo lesson was to reword the claim. The check in step one belongs to the codebase in the example, not to CodeTrain, and it is now labelled that way on screen for clarity. What CodeTrain checks is step two, and there are three assertions there including the one about the argument. The original test asked one question. The tutor asks three, and one of them is the question that matters.

Try it

The lesson is at codetrain.ai/#try. Two steps, about a minute, no account.

Two steps is a real lesson at the short end, incidentally. The tutor builds between two and six depending on how big the topic is. The difference in the real thing is that it reads whatever you actually wrote, so the questions come from your code instead of from a list I authored ahead of time. Ten lessons a month on the free tier, no card, and you can point it at any public repository and get a lesson built from the code that is really in it.

Originally published on the InferHaven blog. I'm the founder, and I'm in the comments.

How I built a tutor that refuses to write your code

InferHaven — Wed, 29 Jul 2026 18:54:58 +0000

Two weeks ago I launched CodeTrain, an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote, and grades it.

People probably assume the rule is a prompt. Something like "do not write code for the user," pasted at the top of a system message, and done. That was the first version. It lasted about an hour.

What follows is what the rule actually cost to build: a grading contract instead of a chat reply, two separate model calls that must never be merged, two execution paths so the free tier costs almost nothing to run, and a short list of things that broke in front of real users.

A refusal is not prompted, its contracted

Here is the failure mode nobody warns you about. Ask a model to teach rather than solve, and it agrees enthusiastically. Then the learner gets stuck, and the model helps. It writes "you could try something like this" and drops in four lines. Technically it did not actually solve the exercise. Practically the lesson is over..

The fix to this was to stop asking for prose and start requiring a decision. Every grading turn returns JSON with a fixed shape, or it is treated as a failure:

{
  "verdict": "advance" | "retry" | "comment",
  "feedback_md": "<concise Socratic markdown>",
  "checks": [{"label": "prints exactly Hello", "pass": true}],
  "learned": ["<short concept the learner demonstrated>"]
}

That single change did more for the product than any amount of instruction tuning in the prompt. A model writing prose has infinite room to be helpful in the wrong direction. A model that has to pick advance or retry, and list the criteria it checked with a boolean next to each, has to commit to a judgment about the learner's code.

Before	After
Agreeable by default, because agreeing reads as helpful	Has to commit: advance, retry, or comment
Slides into writing the fix when the learner struggles	Every criterion carries a boolean the learner can see
No stable signal for the UI to render	Failing criteria are required, so the gap is visible
Impossible to tell a pass from a polite non-answer	The UI renders check marks instead of paragraphs

The checks array is the part learners actually respond to. Each item is one concrete criterion with pass or fail, and the prompt requires failing criteria to be included rather than quietly dropped. You see exactly which two of five things your code did, which is a very different experience from a paragraph explaining that you are on the right track.

Keep the two calls apart

Lesson authoring and grading are separate calls, with separate prompts, and merging them is the single worst thing you can do to a tutor like this.

I know because I tried it. One call, full context, plan the next step and grade the last one at the same time. It saves a round trip and it is meaningfully cheaper. It also produces a tutor that could write the answer into the question. When the same call that just saw a struggling learner also composes the next step, the step it writes gets suspiciously specific. "Now add the except KeyError branch that returns an empty list" is just the solution with a task label on it.

Separated, the authoring call never sees the failure and cannot overfit to it. That constraint is now enforced in the codebase itself: control-plane bookkeeping stays in the router, the tutor engine stays synchronised with the agent's copy of the prompt, and a captured-prompt test fails the build if the two drift apart.

The rule the grader needed most

Not "never write code." That part was easy. The hard rule was: do not be pedantic. Early graders would refuse to advance correct code because the learner had not exercised every branch at run time. A correct if/else is correct even if only one side ran. The prompt now says so explicitly, and instead of gating, it politely suggests an input that would show the other branch.

Two execution paths, and why the free tier does none of the work

A tutor that grades code has to run code. The obvious design puts a container behind every Run button, and it is the reason a lot of similar products have no free tier, or one with a queue in front of it.

Python and JavaScript run entirely in the learner's browser. Python goes through Pyodide in a dedicated worker, JavaScript runs in its own worker, and both stay pooled so the second Run does not pay startup again. A prewarm kicks off when the lesson opens, which usually finishes before anyone types their first line. Runs are capped at 20 seconds, which is generous for a teaching step and short enough that a runaway loop does not hang the tab.

On the free tier, the Python and JavaScript a learner writes never reaches a server, which is the only reason the tier can exist without a card and without a queue.

free tier economics

That holds for the two languages most lessons use. bash, ruby, perl and php genuinely need a real interpreter and a filesystem, so those go to an isolated server sandbox with per-minute and per-day caps, on free accounts too. The browser runtimes cover the common case; the sandbox covers what a browser cannot honestly fake.

The measured cost tracks either way. Across a recent two-week window the managed model spend for every free-tier session on the platform came to about 25 cents total. Compute for running the learner's code was zero, because it happened on their machine, in a tab they already had open.

The languages that cannot run in a browser fall back to a real sandbox on our infrastructure: bash, ruby, perl and php, with a real shell and real files, no network, and per-user rate and daily caps. That sandbox is Alpine with BusyBox, which produced its own class of bug. The grader would demand a GNU coreutils flag that BusyBox does not implement, then fail a learner whose solution was correct. The prompt now names that constraint directly and instructs the grader to accept any working BusyBox-compatible approach.

Dockerfile lessons get static linting instead of a real build. Letting anonymous users build containers on my infrastructure is a speedrun to an incident writeup, and the lesson value was in the file, not the image.

Things that broke in front of real users

If you are going to skip ahead to anything in this article, here is the interesting stuff.

The model emitted JSON that was not JSON. Specifically, feedback containing a stray backslash, usually from a regex or a Windows path in the learner's code, which invalidated an otherwise perfect verdict. The parser now repairs unescaped backslashes before parsing, and falls back to a retry carrying the raw text if it still cannot be read. A tutor that hard-fails because a learner typed \d is not a tutor that can function well.

Older model output used plain strings for checks instead of objects with a label and a boolean. Rather than break those, the coercion layer treats a bare string as a passing check. It is not elegant. It is the difference between rendering something useful and rendering an error.

The tutor asked permission to continue. Early versions ended a correct submission with "ready to move on?" Every single time, learners answered the question instead of coding, and the lesson turned into a conversation. Advancing is now the confirmation, stated as a hard rule in the prompt: when the submission is correct, advance, do not ask.

Browser Python is not Python. No subprocess, no real files, no network. A learner following an authored step involving subprocess was hitting a wall the tutor had built for them. The grader is now told to never fault the learner for a runtime limit, because that step should not have been written in the first place, and to guide toward a runnable approach or simply advance. I will be improving this behaviour into the future as well.

The pattern in all four

Every one of these is a case where the model needed to be guided to perform as a more acceptable tutor instead of just a sophisticated chatbot. Each one had a simple yet effective fix and I plan to be always improving CodeTrains effectiveness as a tutor.

Repo mode and the patch at the end

The free tier teaches on examples and any public repository you point it at. Paid repo mode runs against a real checkout on your machine, which raises a question I got wrong at first: what happens to the code you wrote?

The first design applied each step as its own commit. It felt responsive and it was a mistake. Nobody wants seven commits titled "step 3" in their history, and a half-finished lesson left the working tree in a state the learner had to clean up.

Now the lesson holds everything and proposes one patch at the end, containing only the code you typed, against a branch you choose. You review it like any other diff. If you abandon the lesson, nothing touches your repository at all.

One rule that has not moved

The repo reader refuses secrets. The denylist that keeps .env files and key material out of lesson context is mirrored in two places, the local agent and the control plane, and changing one without the other is treated as a bug.

What I'm tuning next

The tutor is at its best on a focused concept in code it can read directly. Lesson scope is the dial still getting tuned in: a tight lesson on one module lands harder than one that ranges across a whole framework, so more of the authoring work is going into keeping lessons narrow by default.

Browser Python runs on Pyodide, whose package set is a subset of PyPI. The prompt-level guardrails already steer authored steps toward what the runtime can actually import, and widening that coverage, including routing more lesson types to the server sandbox where it makes sense, is the next piece of that work.

The grading loop is a model making a judgment, so the design choice was to make the judgment inspectable. Every criterion is listed with a pass or fail rather than one overall score, which means you can always see the basis for a verdict instead of blind faith in the model itself. That visibility is also our own best feedback channel: every reported step comes back with the checks attached, which is how the tuning above gets prioritised.

Try to break it

The free tier runs in your browser, ten lessons a month, no card and no trial clock: codetrain.ai. Point it at a public repository you did not write and ask for a lesson on the part you understand least. That is the fastest honest test of everything above.

If you talk it into writing your code outright, send me the screenshot. I will fix the loop, and probably frame the screenshot!

-- Ethan L.

Originally published on the InferHaven blog. I'm the founder, and I'm in the comments.

I built a production AI agent for our NOC ticket queue in one shift

InferHaven — Fri, 17 Jul 2026 14:00:00 +0000

Our NOC gets a steady stream of tickets every day, and a good chunk of them are routine: the same site issue reported twice, a ticket that just needs the right category to be organized clearly, a slow-connection report that needs proper network diagnostics. None of that requires real judgment. All of it eats time.

I'd already been using OpenCode to just mess around and test things for free while on shift, so I decided to wire up an autonomous agent to reduce some of the repetitive workload of managing our tickets.

What it actually does

Each separate task of the workflow basically lives in a skill file, loaded by the OpenCode agent in correct order with an AGENTS.md to orchestrate the entire workflow, pretty standard.
A webhook receives a new ticket, waits sixty seconds to see if more come in (batching saves a lot of redundant work), then hands the batch to OpenCode running in non-interactive mode. From there it checks for duplicates against other open tickets for the same site and issue, categorizes the ticket by type, and for anything flagged as a slow connection, kicks off a deeper network diagnostic and posts the result straight to the ticket. It pings the team in chat only for tickets that actually need a human, which is still most of them now as its main job here to start is simply take on 80% of the initial ticket analysis and investigation workload.

What actually took the day

The AI part was the easy part, weirdly. What ate the hours was the plumbing underneath it. Duplicate detection needed exact-match site names, and site naming in our system turned out messier than expected, learned that one the hard way. The network diagnostic tool uses different naming conventions than the ticket system for the same physical sites, so there's a whole lookup step just to translate between the two. The first version had a five-minute timeout on the diagnostic step; turns out some sites take seven-plus minutes to fully analyze especially if there is an issue causing network slowdowns on the way. I bumped the default timeout to ten and made configurable via the .env file so I'm not guessing next time. And there's a fallback notification for when the agent itself crashes or runs out of budget, because a ticket disappearing silently is worse than the ticket never being automated at all.

Where it landed

It's running as a systemd service now, processing real tickets. Duplicates get caught and routed without anyone touching them. The diagnostic runs automatically on the tickets that need it. The team only sees what actually needs attention, and the total ticket count is already trending down since it went live.

The part that surprised me most wasn't that an AI agent could do this. It's that the whole thing, idea to running in production, fit inside one shift alongside the other daily workloads I always have going. A year ago this would've been a sprint's worth of work for a team, and half of it would've gone to exactly the plumbing described above, not the interesting part.

If you're running a similar agent, curious what your batching and timeout setup looks like. Feels like the kind of dead simple thing everyone's probably independently reinventing right now.

[Boost]

InferHaven — Thu, 16 Jul 2026 12:49:51 +0000

InferHaven

Jul 16

I built an AI tutor that refuses to write your code

#showdev #ai #learning #coding

5 min read

I built an AI tutor that refuses to write your code

InferHaven — Thu, 16 Jul 2026 12:00:00 +0000

Somewhere on your team right now, a pull request is getting approved by someone who couldn't rewrite it from scratch. Not because they're lazy. The assistant wrote it, the tests passed, the diff looked plausible, and the sprint doesn't stop for philosophy. Do that for a year and you get a team that ships faster every quarter and understands a little less of its own codebase every quarter too.

I ran into this a few times while building the start of InferHaven. The whole point of this company is that you shouldn't have to hand your code to someone else to get the benefits of AI. But there's a second thing quietly leaving the building, and no firewall catches it: the skill. When the model types and the human tabs through, the knowledge of how your system actually works stops living in your engineers and starts living in a vendor's autocomplete.

So I built the opposite tool. Today I'm launching CodeTrain: a hands-on AI trainer that teaches developers on their own codebase, and refuses to write the code for them.

**The learner writes every line. CodeTrain plans the lesson, sets up each step, runs your code, and grades the result, but it will not type your solution. Not even a "tiny" one-liner. When you're stuck it shrinks the step or gives you a sharper hint. It does not give you the answer, because the answer was never the point.

What a lesson actually looks like

You ask CodeTrain something real. "Walk me through adding a health check to this Dockerfile." "I don't actually understand our retry logic, teach it to me." It breaks that into a short lesson, usually two to six steps, each one small enough to hold in your head.

Each step is one concrete thing to write in a syntax-highlighted editor with a Run button. You type it. The code executes immediately: Python and JavaScript run right in your browser, shell and a few other runtimes run in an isolated sandbox, web stuff renders in a live preview. Then the tutor reviews what you wrote against the step's criteria, shows you ✓ and ✗ per criterion, and either advances you or asks the question that makes you see what's off. Fail a step multiple times and it doesn't dump the solution on you. It cuts the step in half.

$ codetrain start --mode repo
Session a6502084e8254276a394d5be578e9229 started (repo mode).
  Plan:     team
  Modes:    sandbox, repo
  Sessions: 0/1000 this month
  Usage:    0% of your monthly allowance (100% left)
  Models:   haiku, sonnet

Your tutor is open in the browser:
    http://127.0.0.1:7341
(workspace: /tmp/codetrain-repo-v555lhi7/repo)

Tutoring is live in your browser — keep this window open. Ctrl-C to stop.

Before the lesson: pick your level, how much help you want, and whether the session may build toward a real edit. The goal here is the health check from the terminal above.

That's repo mode, the paid tier. The tutor reads your actual repository, builds lessons out of your actual code, and at the end of a guided lesson (can run exploration only if desired) it proposes one integrated patch that lands the code you wrote onto your real branch. You review it like any other diff. Free tier does not include repo mode and drops you into sandbox practice instead: same tutor, same you-type-it rule, generated exercises from natural language.

Here is that health-check lesson, start to finish. Step 2 has you draft the probe on a scratch Dockerfile where mistakes are free:

You write the HEALTHCHECK on scratch first. Run gives you the linter, Send gives you the tutor's verdict against the step's criteria.

Step 3 moves the line you wrote into the real Dockerfile, all 360-odd lines of it, in the one spot where Docker will actually honour it:

Same instruction, real file. Placement is the lesson here: Docker only honours the last HEALTHCHECK, so it goes right before ENTRYPOINT/CMD.

Pass the final step and the tutor offers the one targeted patch it guided you on: the diff of what you typed, applied to your working tree only after you approve it.

Every criterion checked, then the patch-back: your change as a reviewable diff. A .codetrain.bak backup is saved before anything is written.

Change shipped. A real HEALTHCHECK lives in the repo, and the recap lists what you practiced, not what was generated for you.

This is not another assistant

Don't get me wrong. I use coding assistants daily and they're great at their job. Their job is producing code. CodeTrain's job is producing engineers who understand code, and those jobs basically pull in opposite directions.

AI assistant (Cursor, Copilot, Claude Code)	CodeTrain
You describe, it writes	It plans tiny steps, you write them
Output: working code, fast	Output: a developer who gets it
Understanding is optional	Understanding is graded, per step
Great when shipping is the goal	Great when the goal is your team

Use both. Seriously. Let the assistant ship the sprint while CodeTrain makes sure the person accepting its diffs could have written them.

"Code got cheap. Understanding didn't. Train the part that's still expensive."

An InferHaven product

InferHaven exists because your code is yours and should stay on your hardware. CodeTrain extends the same instinct to where your code lives secondly: wetware, brains, good ol' humans. Keep the code in-house, keep the skill in-house with it.

It's built the way you'd expect from us. The agent runs on your machine. Sandbox lessons execute in your browser, not on a stranger's box. Repo mode works against your local checkout. And if you'd rather run lessons through your own model account, bring your own key: Anthropic, OpenRouter, Bedrock, Vertex, or even an Ollama endpoint you host yourself!

What it costs, honestly

The free tier is real: the in-browser trainer, ten sessions a month, streaks and progress tracking, no credit card. It's capped because every managed session costs us actual inference money.

Pro is $24/month and adds repo mode on your real projects, fair-use session limits, sync across devices, and optional BYOK. Team is $32 per seat and adds the part managers keep asking me about: a dashboard showing who's ramping, per-seat budgets, central billing, and onboarding journeys for new hires. Enterprise is the usual conversation about SSO and code that can't leave your network. If that's you, sales@codetrain.ai.

Go type something

CodeTrain is live at codetrain.ai. Sign in, ask it to teach you something you've been faking your way around, and see how it feels to be the one controlling the keyboard again. There's a one-minute demo on the landing page if you want to watch before you commit to the free tier's grand total of... zero dollars.

InferHaven keeps your code yours. CodeTrain keeps the skill yours. Same haven, different cargo. The beacon is active!

— Ethan L.

This is cross-posted from the InferHaven blog. I'm the founder and I'll be in the comments; if you try a lesson and it teaches you something wrong or hands you the answer on a silver platter, I want to hear about it more than I want the compliments.

Building haven bench in the open, and the flaky CI ghost it flushed out

InferHaven — Wed, 24 Jun 2026 17:19:43 +0000

A debugging story from building InferHaven in the open: how a benchmark feature flushed out a flaky-CI race that had nothing to do with it.

I shipped a tokens/sec benchmark for local models. The unit tests were green, and then CI turned red in a way that looked like my fault. It wasn't. Here's the whole hunt: a chown that raced the tide, set -e, and a zsh lock file that vanished mid-sweep.

I built a small thing this week and it caught a bigger thing. That's the whole post, really. But the shape of how it happened is worth writing down, because it's the kind of story that usually gets quietly squashed into a one-line commit message and never told. Building in the open means showing the part where the harbor light flickers, not just the part where the boat docks clean.

The small thing is haven bench, a command that tells you how fast a model actually runs on your hardware. The bigger thing was a flaky CI failure that wore my new feature as a disguise.

The small thing: a number you can trust

If you run local models, you live and die by tokens per second. It's the single number everyone in r/LocalLLaMA trades like baseball cards, and yet most people read it off a vibe: "feels fast on my 3090." I wanted InferHaven to just tell you, honestly, on your own box.

$ haven bench qwen2.5-coder:3b --runs 3

  InferHaven bench — qwen2.5-coder:3b-instruct-q4_K_M
  run 1          106.0 tok/s
  run 2          106.3 tok/s
  run 3          106.0 tok/s
  generation     106.1 tok/s  <- decode rate (avg of 3 runs)
  prompt eval   3507.8 tok/s  (40 prompt tokens)
  load            0.17 s     (run 1: weights -> VRAM)
  total           1.39 s     (run 1)
  method       num_predict=128, seed=0, temp=0, runs=3

The headline is generation, the decode rate. Under the hood Ollama hands back its timings in nanoseconds, and the math is just eval_count / (eval_duration / 1e9). The reason that's the honest number and not, say, total is subtle and important: eval_duration excludes model load and prompt ingestion. So it's the pure speed of the model writing tokens, and it holds steady whether the model was cold or already warm in VRAM.

The first run pays to haul the weights into VRAM (that load line). The second run doesn't. If you quote "total time" you're really benchmarking your disk and your luck. Decode rate is the speed of the engine itself, the thing that's actually true about the model on your card. That's the figure worth screenshotting.

One more bit of honesty baked into the output: prompt eval is gloriously noisy on a short prompt (3507 tok/s above, but it bounces between runs), because you're dividing a tiny token count by a tinier duration. So bench reports it, but quietly. The number it puts in green, the one it wants you to believe, is generation. A benchmark that oversells itself is just a vibe with extra steps.

I wrote it the slow way on purpose: the core tokens/sec math by hand, with a unit test seeded from a real run off my RTX 3060, so I could actually explain every line of it instead of cargo-culting a one-liner. Seven assertions, all green. Shellcheck clean. Ran it live against three models. Pushed the PR.

And then CI turned red.

The red light

Two smoke jobs run on every PR: a slim codespaces stack and the full full-stack one. Full-stack went green. Codespaces failed, and not in my test. It failed bringing the container up, before my code ever ran:

Running the postCreateCommand from devcontainer.json...
Error response from daemon: container 46a4… is not running
postCreateCommand from devcontainer.json failed with exit code 1.
##[error]Process completed with exit code 1.

This is the moment that decides whether you're a good engineer or a fast one. The tempting move is: it's my PR, it's the only thing that changed Poke at the test, re-run it, add a sleep, wrap something in a || true and move on. But if you don't get to the root of the issue it will most likely just surface again.

No fix without a root cause first. A red test you "fixed" by re-running is just a bug you've agreed to meet again later, usually in front of a stranger.

So I did the boring thing instead and actually looked.

Following the evidence, not the vibe

The accusation was "your PR broke the build." The evidence disagreed, layer by layer.

What it looked like -- My PR broke CI

-Only my branch changed
-Red appeared right after I pushed
-It's the new feature, obviously
-Just re-run it / patch the test

What the evidence said -- My PR was a bystander

-My diff touched zero boot-path files
-Full-stack booted the SAME image fine
-The failure was in container startup, before my code ran
-main had failed this exact way before, intermittently

Three facts did the work. First, my diff touched a CLI command, a library function, a test, and some docs. Nothing in the container's startup path. Second, the full-stack job built the very same workspace image and came up clean; if my scripts could kill a boot, both jobs would die, not one. Third, the failure happened during up, before the step that runs my new code ever executed.

That's not a guilty feature. That's a flaky boot that happened to be standing next to me when the cops showed up.

The good news is I'd wired the CI to dump the dying container's logs on failure, and the container's last words were the whole case:

workspace-1 | chown: cannot access '/home/haven/.zcompdump-46a4482c870c-5.9.lock':
              No such file or directory

There it is. 46a4482c870c is the container's own ID. .zcompdump-…-5.9.lock is a zsh completion lock file, created and deleted in milliseconds while the shell builds its completion cache. And the thing that tripped over it was my entrypoint's first-boot ownership sweep: a recursive chown -R over the home directory.

The bug: a chown that raced the tide

Here's the entire bug, and it's a beauty because there's almost nothing to it:

set -e
# ...
chown -R "${HAVEN_USER}:${HAVEN_USER}" "${HOME_DIR}"

chown -R walks the tree, builds a list of things to change, then changes them. If a transient file (say, a zsh completion lock) exists when the walk lists it but is gone by the time chown reaches it, chown exits non-zero. And set -e says: any command that fails, abort the script. So the entrypoint dies. So the container exits. So forty seconds later, when the devcontainer tries to run its setup step, the daemon shrugs and says that container is not running.

It's a time-of-check versus time-of-use race, and like all races it only loses sometimes, which is exactly why main was usually green and only occasionally, mysteriously, wasn't. My PR didn't cause it. My PR just rolled the dice enough times to hit it.

The fix is the same guard the other recursive chowns in that file already had. I'd just missed these two:

- chown -R "${HAVEN_USER}:${HAVEN_USER}" "${HOME_DIR}"
+ chown -R "${HAVEN_USER}:${HAVEN_USER}" "${HOME_DIR}" 2>/dev/null || true

|| true tells set -e to let this one slide, and a vanished lock file stops being a death sentence for the whole container. I proved the mechanism in isolation first (a deliberately failing chown -R under set -e halts the script; the guarded version sails right past it), then shipped it as its own small PR, separate from the benchmark, so the history reads honestly: here's a feature, and here's an unrelated bug the feature flushed out.

The instinct that || true is "hiding errors" is usually correct. But here the error is the bug. A recursive chown racing a temp file is a known, benign class of failure, and the surrounding chowns in this same script already tolerate it. The band-aid would've been re-running CI until it was green. Tolerating a known race at the exact line it occurs is the actual fix.

What I actually want you to take from this

The benchmark is nice. Go run haven bench on your own card and post the number. Honest decode rates are good for everyone, and the more of them in the wild the less anyone has to guess.

But the part I think is worth more than the feature is the shape of the hunt:

A green unit test is not a green build. My math was perfect. The bug was three layers away from my math, in startup code I didn't touch.
Make your failures talk. That "dump logs on failure" step cost me five minutes to write months ago and handed me the entire diagnosis in one line. Future-you is a stranger; leave them evidence.
Resist the re-run. The single most expensive habit in software is treating a flaky test as noise. Flaky almost always means real bug, intermittent trigger. This one had been quietly failing for weeks.
Fix the root, label it honestly. The race got its own PR with its own explanation. Nobody reading the history six months from now has to wonder why a chown grew a || true.

That's the whole reason I'm building this in the open. Not because the wins make good screenshots, but because the misses are where the actual craft lives, and most of the industry hides them.

Both PRs are merged. The light's back to steady.

$ git clone https://github.com/InferHaven/inferhaven-core
$ cd inferhaven-core
$ cp .env.example .env
$ docker compose up -d
$ ssh haven@localhost
$ haven bench    # tell me what your card does

Float your boat up to the dock, clone the repo, run the benchmark, and if you want the managed version when it's ready, the waitlist on the homepage is the way in. The lighthouse is on. And now it doesn't flicker.

— Ethan L.

DEV Community: InferHaven

Adding error monitoring nearly leaked my users' API keys

Project Overview

Bug Fix or Performance Improvement

Code

My Improvements

Best Use of Sentry

The part that almost fooled me

The fix

Keeping the events useful

Not spending the month's quota on one bad afternoon

Agent tracing

About the impact, honestly

What I would tell you to go check

The bug your test cannot see

Why the test could not have caught this

The near miss that catches experienced people

This is roughly why lodash orderBy exists

Why I turned this into the demo on my landing page

The bit I nearly shipped backwards

Try it

How I built a tutor that refuses to write your code

A refusal is not prompted, its contracted

Keep the two calls apart

Two execution paths, and why the free tier does none of the work

Things that broke in front of real users

Repo mode and the patch at the end

What I'm tuning next

Try to break it

I built a production AI agent for our NOC ticket queue in one shift

[Boost]

I built an AI tutor that refuses to write your code

I built an AI tutor that refuses to write your code

What a lesson actually looks like

This is not another assistant

An InferHaven product

What it costs, honestly

Go type something

Building haven bench in the open, and the flaky CI ghost it flushed out

The small thing: a number you can trust

The red light

Following the evidence, not the vibe

The bug: a chown that raced the tide

What I actually want you to take from this

This is roughly why lodash `orderBy` exists