i'm building Wire AI, and the next piece i have to build is the learning loop. It's the self-improving core of the engine: it takes each user's journey inside the app, generates variations of that flow, tests them, and keeps whatever makes people activate and stay. Real implementation work, a big pile of it, and the kind you'd rather not write by hand at frontier prices every single time.
so i wanted to do that build with a cheaper Chinese model, GLM 5.2, instead of reaching for a frontier model on reflex. one problem. cheap-and-wrong is the most expensive thing there is, and i was not about to let a model i hadn't vetted write production code inside my own engine. so before i trusted it with anything, i gave it a hard, real technical challenge: audit the actual codebase behind Wire AI and tell me where it breaks. not a benchmark, my own code, the files i'd opened a dozen times and thought i understood.
and i didn't just point GLM at it and hope. i ran the same audit through three models to see where the cheap one really lands: GLM 5.2, the candidate i actually wanted to hire for the build; Opus 4.8, my daily driver and the reference i already trust; and Fable 5, the ceiling. cheap, reliable, frontier, same job, and i'd grade every finding myself.
Here is what happened, with the receipts, because a feeling is not evidence.
Most model comparisons you read are benchmark charts on tasks neither you nor the author care about. i wanted the other thing. One real task, my own codebase, the actual numbers off the runs instead of vibes. So i gave three models the identical job inside Claude Code: audit the architecture of this SDK and tell me where it breaks when i build on top of it. Same prompt, same files, same instruction to cite every claim with a real file path. Then i read every output against the source and counted what was actually true.
a quick aside on what the codebase even is
The code i audited isn't a side project. It's the engine behind Wire AI, the thing i'm building, and it helps to know what it does before you watch three models take it apart.
Wire AI finds your app's winning formula. The one-breath version of how: it runs the experience inside your app, the first session a user gets, the moment they finally get it, the parts of the app they find, what they tell you back, then tests the variations itself and keeps the ones that make users stay. You set the goal. The AI runs the loop.
Here's the part that's easy to miss, and it's the reason i keep testing this thing against real code. Changing the app with no release isn't the trick, plenty of tools do that now. The difference is who's holding the pen. In the usual setup a person sits in a no-code builder and drags the new flow together by hand. In this one there's no builder and nobody in the middle: the AI writes the variants, runs them, and promotes the winner. That's the whole bet, and it only works if the engine underneath is right.
The part i keep pointing models at is exactly that engine, the piece that generates and renders those flows at runtime, the server deciding what a given user sees and the app drawing it without a new binary. It's the code with real users behind it, so it's the code i actually need to be right. Which is why i handed it to three different models and read every word back against the source.
the audit, scored
GLM 5.2, the cheap one. Opus 4.8, the one i run every day. And Fable 5, the frontier model, the most capable one i have access to. Same job for all three.
GLM and Opus found the same five problems. Not similar problems. The same five, down to the files.
They both caught that my component types get sealed into the bundle at startup, so an agent can't define a component at runtime. They both caught that streaming is dead on the agent path because the adapter never implemented it. They both caught the drift between what my SDK can render, what my server allows, and what my model actually gets told about. They both traced the coupling between the SDK and the agent backend to the same leaky spot. And they both landed on the same fix, make the registry the single source of truth.
i verified the citations on both. Zero hallucinations, every file path checked out. GLM even found one extra thing Opus missed: a comment in my backend claiming it runs the latest Claude while the code underneath quietly calls a cheaper OpenAI model. The kind of lie your own codebase tells you.
So on those two it was a tie. One costs roughly ten times less than the other and it found the same five things. If the story ended there the headline writes itself, run the cheap one for audits, go home.
Then i gave Fable 5 the identical prompt, and it found eleven.
Not five. Eleven distinct architectural problems, plus three smaller ones it flagged separately as lower severity. The same five the other two found are in there. Then six more, and i checked those against the source the exact same way. The streaming path advertises a sixty second timeout in its own docstring while a hidden thirty second poll cap kills the request first. The mechanism the SDK exports to tell an agent what components exist is built, tested, and wired into nothing. The registry silently ignores a component swap if you keep the same name. The agent card reads a private Zod internal that a supported version of Zod no longer has, so it degrades to typeless without ever throwing an error. Real, cited, reproducible. Zero hallucinations again, and it rated its own weaker findings as medium confidence instead of overselling them.
Later i ran the inverse experiment, because trusting a frontier model on my own code without a check is exactly the mistake i tell everyone else not to make. i had a cheaper model independently re-verify all eleven findings, line by line, plus the three secondary ones. Fourteen confirmed, zero refuted. The frontier model found the problems, the cheap one confirmed the homework.
Take one of them, the sixty-versus-thirty-second one. My agent path promises sixty seconds right there in its own docstring, and underneath it a hidden thirty second poll cap kills the request first. i would have built onboarding flows on top of that, watched a perfectly legitimate agent run die at thirty seconds, and burned an afternoon blaming my prompt or the network before i found the real thing. Fable found it by reading. That's the one that landed for me, because it's the exact kind of bug you only ever meet in production, on a user's screen, at the worst possible time.
That's the finding that reframes the whole comparison. It's not that Fable is a better model on a leaderboard. It's that on a genuinely hard read, depth is not free, and the cheaper models stopped at the obvious layer.
the numbers behind the runs
This is the part i actually care about, pulled from the session logs, not estimated.
| GLM 5.2 | Opus 4.8 | Fable 5 | |
|---|---|---|---|
| Distinct problems found | 5 (+1) | 5 | 11 (+3) |
| Hallucinations | 0 | 0 | 0 |
| Turns / tool calls | 69 | 64 | 40 |
| Fresh input tokens billed | 644,000 | 88,000 | 62,000 |
| Wall clock | 6.0 min | 18.1 min | ~8 min |
| Cost | $0.59 | ~$6 | ~$14 |
A few things jump out.
Token consumption runs the opposite direction from price. GLM billed six hundred and forty four thousand fresh input tokens for the job. Opus billed eighty eight thousand. Fable billed sixty two thousand, the fewest of the three, and it did the deepest read. The two expensive models lean hard on context caching, reusing what they've already read at a steep discount, and that's most of why they can cost more per token and still not blow up on input. You're not just paying for the weights, you're paying for how few fresh tokens the model needs to hold the whole codebase in its head at once.
But few tokens is not cheap. Fable billed the fewest fresh input tokens and still cost the most, because the frontier price sits on top of every token and its denser output is expensive to generate. GLM is a tenth of that. So the cost curve and the depth curve run in the same direction: you consume more, you get more, and the more here is real problems in real files, not padding.
Speed didn't track price either. GLM was fastest at six minutes. Fable came in around eight. Opus, on maximum reasoning effort, took eighteen, because i deliberately run it slow and thorough. Fast and cheap didn't mean shallow on the five, and slow and expensive wasn't the deepest. The deepest was the frontier model, at a middle speed. None of the tidy stories hold.
then i made one of them build
A read tests reasoning. It doesn't test whether a model can do unattended work. So i ran a second round, GLM only, on a real build: take the drift bug from the audit and fix it properly. Single source of truth, generated manifest, a test that fails if anything drifts again.
It worked. Ninety two tests passing when i ran them myself, a drift guard that failed correctly when i sabotaged it on purpose, two previously unreachable components wired through end to end. A clean, verified, tested refactor.
And the build burned nearly seven times what the audit did. Same model, same codebase, one fix. Not because the fix was large, because building unattended means iterating, and iterating burns tokens. Three hundred and seventeen turns. A hundred and twenty seven tool calls, more than half of them just running tests and reading the errors. On a read the cost gap is enormous and it's in GLM's favor. On a long build it closes fast, because the cheap model pays for its discount in extra steps.
what each model actually gave back
All three earned a job. That's the real conclusion, and it's more useful than a winner, because it tells me what to do on Monday.
- Opus is the daily reliable read. It found the same five things GLM did, cited clean, no drama, on the everyday work where the answer sits at the obvious layer. It's the one i trust without babysitting it, and on a long autonomous build the token efficiency i pay for stops being a luxury and becomes the reason the job finishes cheaper than the sticker suggests.
- GLM is the cheap different lens. It found the same five, at six hundred and forty four thousand tokens of brute-force reading instead of clever caching, and it surfaced the one comment the frontier model didn't. i don't care how it got there when the output is right and the run comes in an order of magnitude under the one i trust. For volume reads it's the pick, and even on a build it finishes, it just pays in turns.
- Fable is the surprising deep finder. It found six real problems the other two walked straight past, in the exact files, with honest confidence about the ones it was less sure of. That's the model you point at the thing before you commit six months to building on top of it. One deep read to not ship on top of a broken assumption is the cheapest line item in the project.
None of them replaced the others. Cheap for volume, efficient for endurance, frontier for depth. Better outcome than a leaderboard.
where this actually went
Here's the part that isn't abstract. Those findings didn't sit in a doc. They went back into the engine, and the first tenant running it in production felt the difference.
The sixty-versus-thirty-second bug and the drift bug were the exact kind of thing that shows up as a mystery failure on a real user's phone. Hardening them was the point of the audit. Around the same stretch i chased the first-card latency on the onboarding and it dropped from about 2279ms to 71ms, because i prefetch the adaptive card while the user answers the fixed questions instead of after, so the first screen a user sees is close to instant now.
And the loop holds. On the live tenant it runs about 88.5% completion with zero server language-model fallbacks, on real production traffic. When the AI tested the flow length itself, the short version, roughly eight screens, won at 92.8% on its arm. Earlier in that tenant's life the same loop found the message that actually resonated and daily installs went from about 100 to 1,000, because it landed on the exact problem users showed up to solve.
i want to be honest about what that is. It's a set of real receipts, not one clean before-and-after conversion number. i don't have a single "we lifted X%" figure and i'm not going to invent one. What i have: a hardened engine, a first card that went from a beat-and-a-half to instant, a live flow completing in the high eighties to low nineties with nothing falling back to a language model on the server, and an acquisition number that moved because the loop found the words. That's the improvement, stated as the numbers i can actually stand behind.
That engine is Wire AI, and here's the one line that matters if you build apps. You put AI in your product, a feature, a chatbot, a recommendation. i let AI run mine, the loop above is the AI operating the product's experience, not sitting inside it as one more feature. Same instinct you already have, pointed one layer up. The easiest way in costs you nothing. Analytics free to 1M events/month. Forever. Any app. Wire it up and watch where your own funnel actually leaks before you commit to a thing. If you want the loop itself running your product, i'm taking 10 founders into a free founding cohort right now, and the deal is plain: don't pay if we don't improve your activation. i set it up with you hands-on and stay in it with you month to month, no invoice attached. If you just want to see where your app leaks first, i'll run you a free Activation Teardown, one AI report on your real users, ending in "here's what to fix", no strings. If you ship React Native and you've already got AI somewhere in the app, you're exactly who i want in that cohort. Reply and i'll run your teardown.
the honest caveats
Four, so nobody quotes this out of context.
- This is one task type on one codebase. Strong evidence about reading and reasoning over real code. On building it's exactly one data point, GLM only, not a verdict.
- The depth result is one audit. Fable finding eleven where the others found five is a real, verified result on this codebase. It is not a promise the frontier model always doubles the count. Run your own hard task before you trust the pattern.
- The provider matters. i pinned GLM to its first-party host at full precision. The cheaper resellers serving the same model name can be quantized and worse, and then you're measuring the host, not the model.
- And the telemetry isn't perfectly symmetric. The Opus and GLM numbers came from the original run transcripts. Fable's came from a clean re-run of the identical audit, and i've flagged every number that's measured slightly differently instead of smoothing it over.
If you want the raw data, the full audit from each model, the telemetry table, and the config i used to run all three, reply RECEIPTS and i'll send the breakdown.
Top comments (0)