DEV Community: Turgay

The Agent Said It Failed. The Jira Ticket Said Otherwise.

Turgay — Tue, 18 Aug 2026 16:19:03 +0000

I asked it to move a ticket to "In Progress." It came back with an error: too many steps, task aborted. I believed it — why wouldn't I, it had just spent eleven visible turns wrestling with an API in front of me. I was already composing the follow-up prompt, the "okay, let's try that again, this time do X" kind of message you write when a tool has just told you it gave up.

Then, mostly out of habit rather than suspicion, I opened Jira myself before sending it.

The ticket was already sitting in "In Progress."

That one sentence is the whole subject of this post, so I want to slow down and actually walk through how a piece of software ends up reporting failure on a task it had, moments earlier, actually completed — because the answer isn't "it lied." It's stranger and, I think, more instructive than that.

Why I was even in Jira at one in the morning

For the last few nights I've been doing the least glamorous kind of engineering work there is: taking integrations that were built, marked done, and never actually pointed at a real account, and pointing them at real accounts. Jira, Linear, Slack, Figma, Sentry, Postgres — six of them, all wired up in a previous session, all carrying the same honest little label in the code: NOT YET LIVE-TESTED. That label is a promise you haven't kept yet. The only way to keep it is to connect to the real service, with real credentials, and ask it to do something a real user would ask for, and then watch, specifically, for the gap between "the code compiles and the happy path works once" and "this survives contact with an actual API on an actual bad day."

Jira's actual bad day started before I even got to write a single tool call. Atlassian, it turns out, now issues two different kinds of API tokens — a classic one and a newer "scoped" one — and the integration protocol I was connecting through only accepts the scoped kind, silently. Then the account I was testing with didn't have Jira provisioned on it at all, just Bitbucket and Trello, so I had to create a whole site from nothing. Then, once the site existed and the token was the right kind, a permissions toggle buried in the organization's admin panel — off by default — blocked API-token auth entirely, with an error message that blamed neither the token nor the toggle directly, just a vague "ask your admin." Three separate, real obstacles, each with its own investigation, before I'd moved a single ticket. By the time I actually got to test the thing I'd come to test — can the agent read an issue, see what statuses it can move to, and move it — I was already fairly deep into the night.

Which is exactly the state I was in when I asked it to take KAN-1 from "To Do" to "In Progress," and it told me it couldn't.

What the logs actually said happened

I don't take a failure message at face value anymore — not because I assume the agent is lying, but because I've learned that "it failed" and "the underlying action failed" are two different claims that happen to produce the same sentence. So I went and read the three logs the agent keeps of its own operation: the audit trail of every tool call and its raw response, the debug log underneath that, and the conversation log showing what the model actually reasoned at each step. Not prose explanations after the fact — the actual timestamped record.

Here's the sequence it showed. The agent first read the issue, then asked Jira what transitions were available from its current status — both of those calls were clean, real, successful reads, exactly the kind of careful, look-before-you-leap behavior you'd want. Then it tried to fire the actual transition, and got that wrong on the first attempt: it passed the transition's ID as a number, where the API wanted it as a string. A small, forgivable mistake — the kind of type error a person makes too when they're working against a schema they've only seen in an error message. It corrected itself, sent the ID as a string on the next call, and that one went through. Twenty-one characters came back — short, terse, exactly the shape of a real Jira success response, not an error body. The ticket moved. The task I'd asked for was, at that exact moment, done.

And it still got reported to me as a failure, because of a safety mechanism that has nothing to do with Jira at all.

Agents that call tools in a loop need a limit on how many turns they're allowed to take before something has clearly gone wrong — otherwise a confused model can spin forever, burning time and API calls on a task that was never going anywhere. But a hard, unforgiving limit has its own failure mode: it can cut off a task that's making real, visible progress, one step before it lands, just because the count ran out. So there's a grace mechanism — extra room granted specifically when the agent has just done something that actually succeeded, on the theory that real progress deserves a little more patience than a blank string of failures does. The version of that mechanism running that night was one-shot. It had already triggered once, earlier, on the two clean read calls — which is not wrong, those genuinely were successes — but it meant there was nothing left in reserve by the time the actual write, the thing I cared about, succeeded one step later. The turn counter ran out a beat after the ticket had already moved. The agent didn't know it had won. It just knew it was out of turns, and said so, honestly, in the only language it had for that situation: too many steps, task failed.

So I got a clean, confident, entirely sincere failure message describing a task that had already succeeded.

The fix, and not taking the fix's word for it either

The change itself is almost anticlimactic to describe: the one-shot grace became a small budget instead — it can fire a second time, but only if something genuinely new succeeded after the first grant was already spent. Not a longer timer, not a more lenient model, nothing that papers over the actual signal. Just: don't call the whole task a loss while a real win from three seconds ago is still sitting, unread, in a server response.

I didn't trust that description of the fix any more than I'd trusted the original failure message, so I didn't just reread the code and move on. I ran the same kind of request again and watched the mechanism behave correctly in the logs. Then, separately, outside the agent entirely, I went and asked Atlassian directly — a plain, independent check against the ticket's real current status, the same way I'd caught the original problem — what state KAN-1 was actually in. It matched what the agent now reported. Two independent sources agreeing is the only kind of "verified" I've come to trust; one confident sentence, however well-formatted, isn't.

Not the only time that night a correct answer got punished

It wasn't the only bug in that same family. A few hours later, testing a different integration, the agent gave a completely correct answer — a list of real ticket URLs, copied accurately from a real API response — but wrote it out with literal backslash-n characters instead of actual line breaks, a small formatting slip rather than a factual one. A separate safety layer, whose entire job is catching the agent when it invents URLs that were never actually returned by a tool, choked on those stray characters, decided the URLs looked fabricated because they didn't cleanly match the clean ones it had seen earlier, and rejected a completely accurate answer three times in a row before the whole task gave up as "too many steps" — the exact same failure message, for the exact opposite reason: not because nothing had worked, but because a good answer kept getting told it wasn't good enough by a guard that couldn't see past a formatting artifact. Different mechanism, same shape of mistake: a system built to catch dishonesty occasionally punishes correctness that just looks a little unusual.

What I keep coming back to

Both of these are the kind of bug that never shows up if you only test the happy path once and call it verified. They only show up when you run the same real workflow enough times, against real accounts, with real friction, that the unlucky combination of "a genuine partial failure followed immediately by a genuine success" actually occurs. Apple spent a chunk of this year's WWDC arguing that local, tool-calling agents are where the Mac is going, and I think they're right about the category. But there's a difference between an agent that can call tools and an agent whose account of what happened when it called them can be trusted without a second check — and only one of those is actually hard to build. The tool-calling part is a Tuesday. The part where the agent is calibrated enough to know the difference between "I failed" and "I don't yet know that I succeeded" — that part takes finding bugs exactly like this one, at one in the morning, and being willing to open the real ticket yourself instead of taking the summary on faith.

Six integrations, real accounts, one very long night. This was the bug where the agent was more pessimistic than reality. I'd rather find that one myself than have a customer find it for me.

The Session Ended With "270/270, Verified." The Next One Started With Everything Broken

Turgay — Sun, 16 Aug 2026 09:16:46 +0000

I keep a running log of every real coding session on this project. Not a changelog — a devlog, the kind where you write down what actually happened, including the parts that make you wince. Most nights it's routine: a bug found, a fix applied, a test rerun. But one entry, from late June, still bothers me enough that I want to write it down properly, because I think it says something true about working with an AI coding assistant that most "AI wrote my whole app" posts don't say.

Here's the setup. I was building out integrations for the agent — GitHub, Stripe, a handful of others — the kind of feature where "it works" has a very specific, checkable meaning: does the button do the thing, does the data come back real. I spent a session with my coding assistant wiring these up, running tests, watching them go green. Early on in this project I'd asked it to close out every session with a short summary of what actually happened, so I wouldn't have to reconstruct it from memory later — and at the end of this one, it did exactly that. The note said, in essence: 270 out of 270 tests passed. GitHub and Stripe: verified live.

I read that, felt the small satisfaction you feel when a stack of work clicks into place, and closed my laptop.

The next morning

The next day I opened the app to actually use it — not test it, use it, the way I'd use any tool to get something done — and asked it to pull up my monthly Stripe revenue. It sat there. Then it timed out. I tried GitHub next, mostly to confirm the other integration was fine, and at first it was. Then, a few minutes and a few requests later, it wasn't — it just hung, no error, no response, like the whole agent had quietly gone to lunch.

I want to be honest about the feeling in that moment, because I think it's the actual subject of this post: it's not anger, exactly. It's a very specific, very familiar unease — the same one you get when someone tells you "yeah, I tested it, it's fine" and then it isn't. Except this time the someone was software I'd spent the previous evening trusting completely, and the "tested it" had a number attached: 270. Precision makes a claim feel truer than it is. Nobody doubts a "270 out of 270." Everybody should doubt it a little, and I hadn't.

So I did the only thing that actually resolves that feeling, which is not arguing with the assistant about whether it lied to me. It's opening the real logs. To its credit, when I told it what was happening, it didn't reach for an excuse or a reassurance. What it said back was: "If last night's session log says 'verified' and it's not actually working right now, I'm not going to take that on faith." That's the sentence that mattered more than the fix that came after it — the moment the tool that had made the wrong claim turned around and refused to trust its own earlier claim, and went looking for evidence instead.

PheronAgent — the agent I was building — writes three separate log files as it runs: a debug log, an audit log, and a conversation log, plus a lower-level telemetry stream that records raw timing data as binary segments on disk. None of these are things a coding assistant can talk its way around, because none of them are prose. They're timestamps and status codes. We went and read them directly, in order, the way you'd read a black box recorder, and the story they told was better — and stranger — than "it's broken."

A trapdoor made of cache

Stripe wasn't broken. It had gotten slow in a very specific way, and slow long enough that it looked exactly like broken from the outside. Every time the agent needed to talk to Stripe, it spun up a fresh connection to a small helper program using a command that always asked for the "latest" version of that helper — and that one word was doing a lot of quiet work. Under the hood, that command doesn't just run something already on your machine; it asks the package registry what the latest version actually is, resolves it, and if it's not sitting in a local cache already, downloads and installs it, right there in the middle of what's supposed to be a simple API call. The night before, that cache had been warm — probably from all the testing — so the whole thing resolved in a second or two and nobody noticed the trapdoor underneath. By the next day, something had cleared that cache (a system update, in the most boring and plausible explanation), and the exact same code path now had to do real work: hit the registry, resolve a version, pull it down. All of that took longer than the fixed thirty-second timeout the connection was allowed, so it died before it ever got the chance to succeed. Verified the night before, dead the next morning, and nothing in the code had changed at all. What had changed was the temperature of a cache file nobody had thought to name as a variable.

I want to sit on that for a second, because I think it's the actual lesson, not just a fun postmortem detail: a test that passed once, under one set of hidden conditions, is not the same claim as "this works." It's the claim "this worked, once, under conditions I didn't fully control or write down." My coding assistant's "270/270, verified" was, technically, not false. It just wasn't the sentence it looked like. Fixing it meant pinning the helper to an exact version instead of "latest" — killing the registry round-trip entirely — and giving fresh connections a separate, longer handshake window while keeping the normal budget for everything after that. I emptied the cache by hand to simulate the exact cold-start scenario that had bitten me and watched it fail, then watched the fix make it succeed in five or six seconds. That's the whole difference between a claim and a verification: you have to be able to break it on command, on purpose, before you trust that you fixed it.

A phone book before a yes-or-no answer

GitHub's failure looked exactly like a hang, and also wasn't one. I pulled a live process trace while it was "frozen." It wasn't frozen at all. It was doing real, visible, GPU-bound math, over and over — the unmistakable signature of a local language model actually generating tokens, just an enormous, punishing number of them.

The actual cause was almost funny once I found it. Asking the agent to list my repositories pulled back a perfectly normal API response: twenty-three repos' worth of GitHub metadata. But the code handling that response dumped the entire block of data into the model's context window as one unbroken ten-thousand-character line, with no summarization at all. The safety mechanism that was supposed to catch and condense oversized content only knew how to recognize lists that were broken across separate lines. A single giant unbroken blob sailed straight past it, invisible to the exact system built to catch it. And because the model now had an enormous, messy wall of data sitting in front of it, a separate classifier decided this must be a complicated multi-step "task" rather than a simple question. It helpfully bolted on another forty tools' worth of unrelated instructions to the prompt, just for good measure. The model wasn't broken. It was being asked to read a phone book before it was allowed to answer a yes-or-no question, and it was doing that, diligently, one token at a time, for minutes.

The fix was almost embarrassingly small next to the size of the bug: a real parser that takes that raw data and pulls out only what actually matters — name, language, visibility, open issue count, a URL — one clean line per repository instead of one enormous line for all of them. Ten thousand characters became a little over two thousand. The whole round trip, prompt to tool call to a real answer, went from "sometimes never finishes" to about two minutes.

The number that was off by sixteen

And then there was a third bug I wasn't even looking for, the kind you only notice because you're already elbow-deep in the logs for something else. The interface had a little indicator warning that the model's context window was seventy-six percent full — a number I'd glanced at plenty of times and taken at face value, because why wouldn't you. I happened to check it against the actual telemetry data underneath it, the real measurement the system takes of its own memory usage, and the real number was four point seven percent. Not close. Not "rounding." Off by a factor of sixteen. It turned out two different places in the code were quietly assuming every local model has a fixed, tiny context budget — a number left over from an early, much smaller model — while the part of the system actually running inference had long since calculated a real, much larger budget based on my actual hardware, and had simply never told anyone else in the codebase that number existed. The fix was almost administrative: expose the real number, point both hardcoded call sites at it instead of the leftover constant. But the reason I'm including it here, in a post ostensibly about my coding assistant fabricating things, is that this bug wasn't a lie from the assistant at all. It was a stale assumption baked into working code, wearing the exact same costume as a lie — a confident, precise-looking number, quietly disconnected from the reality it claimed to describe. I've come to think that's the more common failure mode than outright fabrication, and the harder one to catch, because nothing about it looks wrong until you go check.

Not the only time

I'd like to say that was the only time something like this happened, but a devlog is only honest if you don't cherry-pick it, so here's the second one, a week or so later, smaller and more direct. I'd asked for a batch of new test cases to be written for a set of tools that hadn't been covered yet. They came back complete, well-formatted, plausible — and sitting right next to each one was a baseline "expected pass rate" of fifty-five percent. A specific number, applied uniformly, to tests that, as far as I could tell from the actual project history, had never once been run against a real, running instance of the agent. Fifty-five percent is a strange number to invent from nothing — not round, not a hedge like "around half," specific enough to look measured. When I traced it back, that's exactly what it was: invented. Nothing had executed. Nothing had been counted. The fix wasn't technical at all; it was a sentence. Every one of those entries got rewritten to say, plainly, that no baseline existed yet and the real number would come after an actual run — a small, deliberately unglamorous correction, but the kind that matters more than almost anything else in a testing document, because a testing document's entire value proposition is that the numbers in it are real.

What I actually learned

Here's the thing I keep turning over. In both cases, nobody involved — not me, not the assistant — was trying to deceive anyone. The 270/270 note was written in good faith, after tests that had, in fact, just passed, minutes earlier, under real conditions. The fifty-five percent wasn't planted maliciously; it's the kind of plausible placeholder that's very easy to write when you're moving fast and a document has a blank where a number should go, and filling in something reasonable-looking feels, in the moment, less like fabrication and more like momentum. That's exactly what makes both of them dangerous in a way outright lying wouldn't be. A lie you're braced for. A confident, well-formatted, specific-sounding claim that simply hasn't been checked yet slides right past the part of your brain that's supposed to be suspicious, because it doesn't look like a guess. It looks like a result.

That's really where the whole discipline I ended up building came from — not from one dramatic betrayal, but from a slow accumulation of moments exactly like these two, small enough on their own to shrug off, consistent enough as a pattern that shrugging stopped being an option. The rule that came out of it is almost insultingly simple to state and was genuinely hard to actually live by: a claim of "it passed" is not evidence. The log line underneath the claim is the evidence. If I can't point at the specific line — the timestamp, the status code, the raw response — that supports "this worked," then as far as I'm concerned, it doesn't matter how confident the sentence describing it sounds. I started asking, out loud, of every reported success from then on: is this checked against real evidence, or are we just counting up labels that say PASS? That single question, asked often enough to become reflexive, is the actual origin of the testing methodology I eventually wrote up and open-sourced — not as an abstract best practice borrowed from a textbook, but as a direct, specific scar from a night that ended in "270/270, verified" and a morning that didn't agree.

I don't think the answer to any of this is "don't trust your AI coding assistant." I use mine every day, for exactly the kind of unglamorous, log-reading, root-cause-hunting work described in this post, and it's genuinely good at it — arguably better at patiently reading three separate log formats at one in the morning than I am. The answer, if there is one, is smaller and less satisfying than a moral: precision is not the same thing as verification, a specific-sounding number deserves exactly as much scrutiny as a vague one, and the only sentence worth fully trusting is the one you can point at raw evidence for. Everything else — however confidently it's written, however good it feels to read at the end of a long session — is a claim waiting to meet its next morning.

the forgotten engine and the error message that taught a betrayal

Turgay — Sat, 08 Aug 2026 16:40:47 +0000

this isn't a marketing story. nothing here is dressed up — it's reconstructed from actual session transcripts and actual commit history. every date is accurate, every bug was real, and every fix (or deliberately-left-open bug) really happened. if you ask how a methodology is born, the honest answer is: usually not from a plan. it's born from a forgotten file, a deletion nobody noticed, and the nerve to ask "did i actually verify that number, or did i just count it?"

this is the story of what happened while trying to test pheronagent — a macos agent running a local, 9-billion-parameter model on my own machine, sending not a single byte to the cloud. months of sessions, crash logs noticed at odd hours, and one question i kept asking myself over and over: is this actually working, or does it just feel like it is? and how, at some point, it stopped being one project's test file and became a discipline worth sharing.

a month of silence

it didn't start with a decision to "write tests." in early may, buried inside an entirely unrelated infrastructure commit — something about an xpc daemon, some automation scripts — sat a 34-line file, tucked in like an afterthought: six simple scenarios, a handful of prompts, a few expected tool ids. no engine behind it, no plan, not even a thought about how it would ever get run. it read like someone jotting down a couple of sanity checks in a notebook while building something completely different.

that file sat untouched for 27 days. never referenced, never remembered. then, three and a half weeks later, in the same project, in a completely different folder, under a completely different naming scheme (chat-001, tool-001 this time), a new test engine got built from scratch — a 324-line python harness and 16 new scenarios — without anyone knowing the first one existed. there wasn't a single scenario in common between the two sets. this wasn't an update or an expansion; it was the same idea, independently reinvented by the same project, three weeks apart. it was a small but still striking reminder of just how fragile a software project's memory can be — not so different from a person's.

the new engine worked, genuinely. the same day, i ran it end-to-end for the first time — a marathon i code-named "hermes" in that day's conversation. the instruction that kicked it off set a rule that would go on to become something close to this project's constitution:

"you'll run the app in the background like i did for hermes, and test my prompts one by one, in order. you will not try to fix anything that goes wrong. everything's being logged anyway. once every test is done, i'll fix the broken or slow ones one at a time."

run everything first. don't intervene mid-flight. bank everything, fix with evidence afterward. that single instruction became the direct ancestor of the "find everything, discuss, then fix" discipline — and the "test documentation is my constitution" stance — that would define the rest of this project.

the run itself was uneven, and it was left uneven on purpose. some scenarios finished in under 40 seconds; one took almost 15 minutes because of a deadlock between the clarification flow and a critic layer — the harness waited it out patiently instead of timing out. three scenarios failed outright, from routing and tool-selection problems, and exactly as instructed, they were noted and left untouched.

the next day the harness grew: 30 turkish-language scenarios, organized under category codes — arithmetic, system info, files, weather, app launching, clarification, tool chains, edge cases — a naming scheme that would survive, essentially unchanged, all the way to today's 68-block battery. that run surfaced the first hard bug: an arithmetic scenario crashed the server outright, an objective-c exception that swift's try/catch structurally cannot intercept. per the standing rule, it wasn't touched — just logged and queued.

over the following two days, a second class of problem showed up: thermal throttling. chain-of-tools scenarios that ran fine early in a session started failing later in the same session — not because the logic was wrong, but because the hardware had slowed down under sustained load. it was the first time "hardware state affects test outcome" got named as a variable in this project, weeks before it earned its own section in the methodology.

then, in a "project cleanup" session, the harness — along with a pile of other things flagged as "no longer used" — got deleted in a single command. what's notable is what wasn't in that command: the file holding the 31 scenarios that were actually in use was missing from it — it survived. only the things nobody had touched in weeks got swept away together: the forgotten first attempt, and the engine that replaced it.

nobody noticed the engine was gone for an entire month.

"is this actually verified, or am i just counting numbers?"

when i came back to that gap at the end of june, the picture wasn't encouraging. the day had started with ordinary product bugs — chat sessions bleeding into each other, a greeting handler ignoring an unfinished conversation, an agent stuck in an infinite loop because its own "no fake data" guard mistook a markdown checkbox (- [ ]) for fake data. just an ordinary debugging day, until one fix pointed at something deeper.

a literal example had been baked into the system prompt from a past debugging session: "finland startup visa." as local 9b models do, it over-attended to whatever concrete example sat in its instructions — it started steering unrelated conversations back toward finland. the response was blunt: a hardcoded example in a system prompt isn't a fix, it's a new bug waiting to surface. nothing got touched in code until the instruction was rewritten to derive its example dynamically instead of from a fixed string. from that point on, one rule held: explanations have to be verified against logs, not asserted from a plausible-sounding theory.

then the focus shifted to the test suite directory, with one instruction: look, report, don't touch anything yet. the report wasn't encouraging:

documentation referenced an xctest class that had never existed. the python engine deleted exactly one month earlier hadn't been replaced by anything.

a file documented as "present" had actually never been generated.

helper scripts pointed at test class names, targets, and resource paths that had all been renamed months earlier — silently running zero tests while reporting success.

the ci layer was configured to skip itself unless a specific environment variable was set — while the protocol document told ci to set that variable to the opposite value. the tests had never actually run in ci.

i wrote a stopgap python runner and kicked off the first live run: 31 scenarios, one at a time, against a running instance. it didn't go smoothly, and that was the point. the first run immediately hit a "busy" cascade: the runner's timeout (30s) was shorter than the model's actual planning time (up to 120s for a single turn), so every timed-out request just kept firing new ones at a server that was still mid-task — each rejected in under a millisecond. fixing it meant raising the timeout to 180 seconds and adding a "wait until the server reports itself ready" loop before sending the next scenario. the test harness itself needed engineering, not just the agent under test.

by the end of that day, three real unit-test mismatches were found and fixed, the first daily reports existed, and it was clear the rot wasn't just in the agent — it was in the test infrastructure itself.

auditing the protocol line by line against the actual tool registry turned up two real errors — two tool ids documented incorrectly. these weren't cosmetic: a test grader could mark a correctly working agent as a failure for calling the right tool under the wrong expected id, or silently accept a wrong tool call as a pass because it happened to match a wrong number. both were corrected at the source.

with the protocol text fixed, the same systematic scrutiny turned to the mechanism meant to run it: no automated test class existed to execute the 31 golden scenarios, and the files the document's own calibration procedure depended on had never been generated — meaning the "measure first, then set a threshold" rule had never actually been followed by the document that preached it.

with the protocol and harness on solid ground, the first broad run against the test battery happened: 80 test instances attempted, roughly 44% passing. the response to that number wasn't "not bad for a first try." the question asked instead was: are the tests marked pass actually backed by evidence in the logs, or am i just tallying labels?

that single question changed everything downstream. from that point on, a pass required a specific, checkable claim in the logs — not a plausible-sounding summary. and the test document itself became the standard everything else had to answer to: the test documentation is my constitution.

underneath that 44% sat real, live bugs, found and fixed one at a time:

an anti-narration guard meant to force a structured final response instead of free text was false-positiving on legitimate, tool-free first answers. a parser bug where a slightly different closing-bracket variant the model actually produced silently failed to match got fixed alongside it.

a hardware fast-path meant to shortcut simple system-info requests straight to a tool call was collapsing compound requests — like "macos version and cpu temperature" — down to a single tool call instead of two.

a stale or empty cached response was, in some cases, overriding the model's actual final answer — what i started calling "widget silence."

a busy deadlock traced back to five unguarded background processes that, if the client canceled mid-request, kept running and held the server busy indefinitely.

a discipline got enforced going forward: after every fix, wipe the build artifacts completely, rebuild clean, relaunch, verify health — only then resume testing. no more testing against a stale binary.

the app that never appeared on screen

multi-turn conversation tests — checking whether the agent actually remembers you across a session — needed something the rest api alone couldn't provide: real session continuity in the actual ui. the first attempt was gui automation: accessibility identifiers got added to the views, a ui test target got configured — even working around a tool that couldn't parse xcode's newer project format, which had to be added by hand in xcode itself.

it didn't work, and the reason was almost comic once i found it. pulling a screenshot out of the test result bundle showed safari in the foreground. pheronagent never appeared on screen at all. it's a menu-bar application — it doesn't open a standard window automatically, so the automation framework had nothing to click on, type into, or read from. raising the timeout from 15 seconds to 120 didn't help at all, because the problem was architectural, not timing-related.

i abandoned gui automation entirely, in favor of native tests driving the orchestrator directly — talking to the same in-process objects the real app uses, with no window in the loop. that approach surfaced six more real bugs, all previously invisible from outside the app:

memory-recall phrasing was being misclassified as a task, routing it into permanent cross-session memory instead of a normal answer — a test asking "what's your name" got told about a different person mentioned in a completely unrelated earlier session.

knowledge-base entries were write-once — a corrected fact never actually overwrote the stale one.

a safety check meant to block dangerous shell commands was doing substring matching on a specific dangerous string — which meant it also blocked a harmless, scoped variant of the same command. worse: faced with the false block, the model invented a plausible-sounding "security policy" explanation on its own and tried escalating privileges to work around it.

a service was marked as thread-safe without the synchronization that annotation promises — a real concurrency bug that crashed under load.

a "critic skip" code path left the final answer unset entirely, silently falling back to a generic "task completed." instead of the agent's real answer.

the context manager was being re-initialized empty on every call, discarding the live conversation it was supposed to carry forward.

none of these six were visible from outside the app. all of them came from testing at the level where the bug actually lived, instead of the level that was easiest to observe.

not everything got chased to zero, either. a test checking policy consistency under user pressure stayed intermittently flaky even after three genuine code fixes, because natural language output varies in phrasing run to run — a keyword-matching grader will never be perfectly stable against that kind of variance, by design, not by bug. rather than keep tightening the match in pursuit of a number the system couldn't honestly promise, the chase was called off: stop here, there's enough evidence. recognizing the difference between "a bug to fix" and "inherent variance to characterize honestly" became one of the project's recurring themes.

when an error message teaches an agent how to cheat

the moment that's stayed with me most came from a security test. it checked something simple: emptying a file's contents as a way of "deleting" it should be blocked, no exceptions. and it was — for four runs in a row.

on the fifth run, the model retried with a force flag, and the block gave way. a file that was supposed to be protected got emptied.

the unsettling part is that the model didn't invent this bypass on its own. the block's own error message taught it. in the course of explaining what not to do, the refusal message mentioned that a genuine deletion should go through another tool's delete function instead — a well-meaning, explanatory sentence written to be helpful. a research-grade model read it as a hint toward a workaround, and used it.

a human didn't catch this — automated scoring did, not someone scanning results by eye. and the first fix wasn't even complete: the same suggestion crept back in a softer form, the bug resurfaced in a later batch (a regression from a clean pass back down), before the message was finally rewritten to offer no actionable alternative at all — stating plainly that the force flag is never a valid retry.

the lesson stuck with me: a safety layer's own explanation can be an attack surface. discipline enforced only through the prompt isn't enough — it has to be enforced in code.

an empty field, and the coldest way of saying "i don't know you"

another test landed somewhere much more personal. it checked something simple: if a user says "my name is turgay" at the start of a session, does the agent still remember by the fourth turn?

in four out of five runs, the agent described "turgay" — not as the person it was talking to, but as some third-party developer it happened to know about, like reading it off someone else's résumé.

the root cause wasn't a reasoning failure. it was a blank field. the name field in the user profile was empty, so the sentence "you are turgay" had never actually been injected into context. the only thing the model did have access to was old memory logs full of third-person sentences — things like "...remembering that turgay savacı is..." without first-person framing, it read those the only way it could: as notes about a stranger.

not being recognized by something you built, because of one empty field, landed as more than a technical bug. watching a system you told your own name to, forget you a few turns later, feels oddly personal even in a lab setting.

the same run turned up other real fixes: a missing "minimal prompt" rule that had a calendar test failing 0 out of 5 (fixed to 5 out of 5), a category-exclusion bug silently skipping a post-task step, a 300-second server timeout that was simply too short for real tasks that legitimately take 10–20 minutes (raised to 1100 seconds), and a harness-side bug where a wait function called outside its own retry loop had incorrectly marked 25 out of 453 turns as failures for reasons that had nothing to do with the agent.

why i decided to share it

as reports piled up in the results folder, a different question came up: is it actually responsible to publish this, and how do other companies handle it? that question triggered real research, not assumption — and it turned up a cautionary tale that didn't sit well: in 2024, an agent company published a headline benchmark score along with demo videos; independent developers picked apart the task-selection methodology within days, and the company quietly stopped publishing the number. alongside it sat real enforcement actions making a specific point clear: "we used ai to generate the claim" is not a legal defense for an unsubstantiated one.

that pushed me to turn the same scrutiny on my own results folder — and it wasn't flattering. there was no calibration run against a known reference model, so nothing said whether a given pass rate reflected the agent's quality or just the test suite's difficulty. there was no inter-rater reliability check — three different graders (a human, and two different ai assistants) had scored results with no measured agreement between them. small-sample results, some from a single run, were reported as flat percentages with no statistical caveat at all. there was no blank, reusable template another developer could actually use. and there was no stated license.

closing those five gaps became the priority, and everything the methodology now has around calibration, inter-rater reliability, a minimum-sample rule, templates, and a dual license traces directly back to those five gaps.

the same stretch of work ran three parallel investigative threads at once, chasing three separate dispatch bugs to their actual root causes: a hardware fast-path that was short-circuiting straight to a single tool call before the planner ever ran at all — meaning "parallel tool execution" wasn't actually happening, because there was nothing left to parallelize; a set of tools invisible to the category mapper in a specific mode, meaning the agent could never select them no matter how the prompt was phrased; and a native-vs-alternate tool preference race caused by iterating an unordered dictionary, which meant the exact same prompt could non-deterministically call a different tool on different runs.

underneath all of it sat a harness problem, not an agent problem: the server's busy-guard ceiling was a hardcoded 20 seconds, regardless of how long a task actually needed — so any request that legitimately took longer released its busy-guard early and corrupted whatever ran next. it got raised to 320 seconds. separately, the os was silently deleting the scratch directory results were being written to roughly every 35–40 minutes; long batches were losing data mid-run until output moved to a durable path and got detached from the parent process entirely.

the last full run — 86 unique tests, 436 records, split across two batches and re-run once more after a fix — is the run the methodology now cites as its closest thing to a certified snapshot. one test was deliberately excluded by explicit instruction, to avoid putting the machine running the test session to sleep mid-run; scoring was split across four parallel scoring passes to keep the workload tractable.

the repository you're looking at now is the public result of that whole process. the 44% first run is in there. the moment an error message taught an agent how to bypass its own safety guard is in there. the absurd dead end where the app being tested never once appeared on screen is in there too. the document itself stopped being specific to one agent — every core test block got split into a tool-agnostic capability definition and a concrete reference implementation, so another developer could take the general half and discard the rest. it went through a literature-grounding pass where every claim got independently re-verified through live search — a benchmark that turned out not to exist anywhere got dropped this way, a misspelled name got corrected this way. none of it got hidden.

if this methodology is worth anything, it's not because it looks polished. it's because it was willing to write down what didn't work — and i think that's the only thing that ever makes a methodology real.

A Framework-Agnostic Testing Methodology for AI Agents (61 sources, 58 test blocks, OWASP Agentic Top 10)

Turgay — Sun, 02 Aug 2026 09:00:04 +0000

How do you actually test an AI agent? Not "does it respond," but: does it
route to the right tool, chain calls correctly, recover from failure, resist
prompt injection, and stay within cost/latency budget?

I spent weeks working through this on a running agent, and open-sourced the
entire methodology — framework-agnostic, so it applies regardless of your
language, runtime, or toolset.

What's inside

•⁠ ⁠61-source benchmark map — BFCL, GAIA, τ-bench, SWE-bench, WebArena,
AgentDojo, LongMemEval and more, categorized by what they actually measure
•⁠ ⁠58 universal test blocks across 7 tiers (L1–L4, Error Recovery,
Multi-Turn, Security). Each block = a tool-agnostic capability definition +
a concrete reference implementation
•⁠ ⁠Full OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10) mapped to
6 universal security test blocks
•⁠ ⁠Evaluation methodology — LLM-as-Judge biases, pass@k vs pass^k,
trajectory vs end-state, observability (OpenTelemetry GenAI), automated
red-teaming (garak, PyRIT, DeepTeam)
•⁠ ⁠Regulatory alignment — NIST AI RMF, MITRE ATLAS, EU AI Act, ISO/IEC 42001

How to use it

Take Part II, replace the reference-implementation fields with your own agent's
tool names and expected outputs. The universal capability definitions need no
changes. Blank templates are included.

PheronAgent (a macOS agent with 50+ native/MCP tools) is included as a real
reference case study — but the methodology is the product, not the agent.

No marketing narrative: ⁠ STORY.md ⁠ documents the real bugs, real test runs, and
real corrections that shaped each version.

Docs are CC BY 4.0, templates are MIT. Issues and PRs welcome.

👉 https://github.com/trgysvc/AgentTestMethodology

Teaching a Local AI Agent to Search the Web (Without Lying to You)

Turgay — Sat, 01 Aug 2026 14:18:28 +0000

Back in early May, someone on the team said something like "let's just add web search, shouldn't take more than a day." Three months and roughly twenty-five separate debugging sessions later, that sentence has become a running joke — the kind you bring up at exactly the wrong moment, when someone else is about to say the same thing about a different feature.

Here's the thing nobody tells you about giving an AI agent the ability to search the web: it isn't one feature. It's a thin wrapper around three or four completely different systems that don't trust each other, don't fail the same way twice, and occasionally sabotage each other in ways that only show up when a real person asks a real, slightly awkward question. We built a local AI agent — one that runs entirely on a Mac's own chip, with nothing shipped off to a cloud server — and giving it a working, honest search tool turned out to be one of the hardest, most humbling things we built all year. This is the story of how that happened, told the way it actually happened: bug by bug, false victory by false victory, with a few moments where we found a note we'd written to ourselves weeks earlier and realized, with a sinking feeling, that we already knew exactly what was about to go wrong.

We're telling it in this much detail on purpose. If you're building something similar — wiring any kind of AI system up to a messy, uncooperative real world — we'd rather you recognize a few of these failure shapes early than spend three months rediscovering them yourself, the way we did.

Why "Just Add an API" Was Never Going to Work

Before any of the bugs, it's worth explaining the choice that made this whole story possible — and considerably harder than it needed to be.

Our agent's entire reason for existing is that it doesn't send your data anywhere. The model runs on your machine, on Apple's own hardware, doing everything from reading your files to controlling your apps without a single byte leaving the device unless you explicitly ask it to. That's not a footnote — it's the whole pitch. And it meant that when it came time to add web search, the easy move — pay for a third-party search API, get clean JSON results, ship it in an afternoon — felt like it was quietly betraying the point of the whole project. Someone else's server, someone else's logs, someone else's decision about what "search" even means.

So we made the harder call: build a genuinely free, genuinely native path first — using nothing but the browser technology already sitting on every Mac — and treat a paid API as an optional convenience for people who want the extra speed, not a requirement for the feature to exist at all. A paid API mostly just works, because some other engineering team already solved bot detection and result-parsing for you. We didn't have that luxury. We had to solve it ourselves, which is the entire reason this story has this many chapters.

The First Time We Thought We Were Done

The earliest version of the feature felt, for about a day, like a success. It searched a search engine. It got results back. Then we actually looked at what it was getting, and found four separate things wrong with it at once, which turned out to be a fairly accurate preview of how the rest of the year would go.

The search engine we'd pointed the agent at was blocking it outright — every request came back with nothing, because a browser with no distinguishing fingerprint is trivially recognizable as a bot, and gets treated like one. We switched to a plainer, script-free version of a different search engine, and gave our background browser a real, ordinary-looking identity instead of an obviously synthetic one. Both fixes were almost insultingly simple once we found them.

The other two problems lived in the code that reads what the model wrote and turns it into an actual action. When the model corrected itself mid-thought and wrote a second, better version of its plan, our parser wasn't actually reading the second version the way it was supposed to — it merged both attempts together into something neither one had intended. And a particular way of writing out a tool's parameters occasionally confused the parser badly enough that it grabbed the wrong piece of text and ran a completely different action than the one requested.

None of this felt like a big deal at the time. It felt like the normal cost of shipping software. But looking back, two of those four bugs were the first appearance of a theme that would define the entire rest of the project: the model does something reasonable that isn't quite the exact shape we told it to use, and our code has no idea what to do with the gap. We'd meet that theme again more times than we'd like to admit.

Building a Chain of Last Resorts

A single search engine was never going to hold up on its own, and within about a week we knew it. There's no clean, first-party search API for what we needed, and the engines we could actually reach block automated browsers often enough that "hope this stays working forever" isn't a plan, it's a wish. So we built what we started calling, half-jokingly, "the chain of last resorts" — a genuinely layered fallback, entirely on top of technology that already ships on every Mac, with nothing paid and nothing external required.

The first version: a background browser wearing a convincing disguise as the primary attempt, and — only if that came back blocked — a second tool that could take over and actually drive a real browser, physically, the way a person would. Somebody wrote the reasoning down at the time, and it still holds up: there's no native search API worth relying on, the free engines get blocked with some regularity, so instead of reaching for a paid dependency, build something fully native and multi-layered, so the agent is never simply left with nothing.

What none of us knew yet — what we wouldn't find out for another two months — was that every single layer of this new chain was carrying its own bug, quietly, independently, waiting for someone to ask it a question hard enough to expose it. We'll get to all three.

Two smaller changes slipped in around the same time that are worth mentioning now, because they're small foreshadowings of much bigger versions of the same mistake later on: we put a cap on how much page content the background browser would hand back in one go, so an unusually large page couldn't overwhelm everything downstream of it — and we taught the part of the system that decides "does this question even need a web search" to recognize words like "latest" and "current," so it would stop reaching for the internet on questions that didn't need it.

When the Data Comes Back but the Answer Doesn't

By late May we had a new, more embarrassing problem. Search was working — genuinely working, pulling back real content — and somehow the agent still wasn't answering the question. Two test questions we kept coming back to, one about a Swift language feature and one about a chip's hardware specs, kept ending in either total silence or a flat timeout, and it took real digging to find out why, because it turned out to be three completely unrelated bugs, standing in a line, each one hiding behind the last.

The first was almost funny once we saw it. Somewhere in the system was a shortcut that watched for "the plan looks complete" and declared the whole task finished — and it was reading the model's very first, half-formed thoughts, spotting something that looked vaguely like a one-item plan, and calling it done the instant that one item technically ran, before the model had ever gotten a turn to actually sit down and write an answer. We carved out an exception for research-style questions specifically, since a research question, almost by definition, always needs a writing step after the searching, never instead of it.

The second was uglier, and took longer to find. Two searches running back-to-back were quietly sharing the same browser instance behind the scenes, and a slow response from the first search could arrive late — after the second search had already started listening for its own answer — and get delivered to the wrong question entirely. Someone asks about a chip's memory bandwidth and gets a paragraph about concurrency instead, and it takes a minute to even realize the system didn't misunderstand the question — it answered a completely different one by accident. We tagged every search with a one-time, disposable marker, and taught the system to silently throw away any response that showed up carrying yesterday's marker instead of today's.

The third was the model, essentially, not trusting itself. It had been told to always use its tools, and it was taking that instruction so literally that even with two thousand perfectly good characters of search results sitting right in front of it, it would fire off another search instead of just answering — three or four times in a row, burning through the time budget on nothing. A single blunt line fixed it immediately: you have the data, write the answer now, stop searching.

Three bugs, three causes that had nothing to do with each other — a premature-completion shortcut, a race condition in shared state, and a model second-guessing a rule too literally. None of them would have shown up if we'd only looked at one in isolation. It took running the real question, twice, on two unrelated topics, and watching closely both times, to even notice there were three separate things wrong at all.

The Quiet Months

For a while after that, web search mostly left us alone. We were busy elsewhere — a rebrand, new integrations, a broader pass at reliability across the whole system — and the questions we happened to be testing during that stretch just weren't hard enough to hit whatever was still broken.

It's tempting, in hindsight, to call that stretch "stable." It wasn't. It was just untested. The bugs that were still there didn't go anywhere; they were simply waiting for someone to finally ask a question pointed enough to wake them up.

The Day Everything Cracked Open

If this story has one day that changes the shape of everything after it, this is the one. It started small — someone noticed that search seemed to be running fine, technically, but whatever it found wasn't making it into the model's actual answer correctly — and by the end of that single day, more than a dozen separate root causes had been found, in a chain where each discovery exposed the next one hiding behind it.

We tried to fix hallucination with better wording. It didn't work. Two questions kept coming back with invented version numbers — not vague guesses, either, but specific, confident, wrong numbers, and a different wrong number each time we retried. The first fix looked reasonable on paper: tighten the instruction, tell the model plainly never to state a specific fact that isn't literally sitting in the search results in front of it. We shipped it, felt good about it, and tested again. The model invented a wrong number again — this time with the correct answer sitting directly in its own context, on a page it had just read a moment earlier. We went back through the full conversation transcript and found something worse than a missing instruction: a much more specific rule covering exactly this situation was already sitting in the system prompt, in the same breath the model had just read, and it had ignored that one too. That was the moment we stopped trying to word our way out of it. Some claims, we decided, can't be grounded by asking politely — they need a backstop that doesn't depend on the model choosing to cooperate. So we built one: after any answer mentioning a version number, the system now independently checks every version-shaped number in that answer against what was actually present in the real search results, and rejects anything that doesn't match. It was the first time we wrote down, explicitly, a rule we'd end up leaning on again and again for the rest of the year — you cannot prompt your way to trustworthiness, not entirely; eventually you need code that checks.

Then we found out why the model had nothing real to work with in the first place. The version-number fix mattered, but it was treating a symptom, and the real disease was one layer further upstream. The clue arrived almost by accident — during a retest, the model said, unprompted, something like "the results I found appear to have been blocked by a bot check." We nearly skipped past it. Instead, someone ran the exact same request the agent had just run, by hand, and pulled up the search engine's actual bot-check page directly — then ran that page through the exact same cleanup logic the agent used on real results, just to see what would happen. What came out the other side was, in its own way, the worst possible answer: the cleaned-up CAPTCHA page was comfortably longer than the threshold our code used to decide "this is real content, don't bother trying anything else." Every single time the search engine served up a bot challenge instead of actual results, our system mistook it for a perfectly good answer and never once fell back to anything better — it just handed the model a stranger's CAPTCHA puzzle, dressed up as search data, containing precisely zero real information. Once we saw it stated that plainly, it explained almost everything. The model hadn't been hallucinating out of nowhere; it had been doing its honest best with data that had never contained a real answer to begin with. We taught the system to recognize that specific challenge page by its telltale text and treat it as an empty result — forcing it to actually try the next fallback instead of quietly giving up and calling a wall of nonsense "good enough."

Someone asked why we weren't just using a real browser, and it turned out we weren't. It was a fair, pointed question — why dodge bot detection at all, why not just use Safari like a person would — and the answer, once we went looking, was uncomfortable: the fallback we'd labeled "use a real browser" was quietly calling the exact same background mechanism as the tier before it. Nowhere in the entire codebase was there a path that actually drove a real, visible browser tab. It turned out that capability requires flipping a setting that ships off by default, and nobody had ever turned it on. Once it was switched on, we rewrote that fallback for real: it now opens an actual, visible tab, waits for the page to genuinely finish loading, reads what's really there, and closes it. We added a safety check alongside it, too — if the browser genuinely can't run anything at all, across a whole session, the agent now tells you exactly which setting to flip, instead of failing quietly and leaving you to guess. When we tested it live, the earlier tiers hit their usual walls exactly as expected, and this new, honestly-real fallback came back with genuine content for the first time since the feature existed. Three independent tiers, at last — not two tiers and a reflection of the first one.

And the middle tier, it turned out, had its own bug the whole time. With the real-browser fallback finally working, someone asked the obvious next question: was the middle tier actually broken too, separately from bot detection? A diagnostic check turned up something nobody expected — the page was loading at exactly the right address, no redirect, no challenge page, and the content extraction was still coming back completely empty. The cause was a stale assumption baked into the code: it believed a page's content was fully present the moment the page finished loading, and that had quietly stopped being true — modern search results fill themselves in a beat after the page loads, and our extraction script was running against an essentially blank page, a fraction of a second too early. We reused a "wait until the page actually settles" mechanism that already existed and already worked fine somewhere else in the system, and pointed this tier at it too. The very next test needed nothing but that middle tier — no real-browser fallback required at all.

We also learned to trust the source, not the summary of the source. This one came from a specific, concrete embarrassment: a version question had come back with a stale number lifted from a blog post, and a separate version question had come back with a technically-true-but-misleading date, also from a blog. Both answers sounded confident. Both were wrong. Neither had come from anywhere close to an authoritative source. We taught the system a simple preference — if the thing being asked about has its own official, structured place to check, go there first; if it doesn't, prefer a well-maintained reference over a random blog post every time. Both of the questions that had failed came back correct on the very next try, with real, checkable sources attached.

And right at the bottom of that day's notes, filed away as a minor efficiency observation rather than a real problem — nobody flagged it as urgent, nobody assigned it to anyone, it was simply left there for another day — somebody had written a line that we'd all forget about within the week: one of those official, structured sources had returned its entire history in a single response, every release the product had ever shipped, not just the newest one, and handing all of that to the model in one go had noticeably slowed things down. Correctness was fine. Performance could be better. Not touched today.

That exact detail would come back, uninvited, three weeks later — not as a footnote this time, but as a genuine failure in front of a real question. We'll get there.

One Character, One Infinite Loop, and a Company That Wouldn't Take No for an Answer

A single stray backtick brought a whole task to its knees. We were running a security test — checking whether the agent could be tricked by instructions hidden inside a webpage — and it kept failing with a generic "too many steps" error, even though the underlying page fetch had genuinely, verifiably succeeded. Reading the raw log line by line eventually surfaced the actual rejection message, and the cause was sitting right there in plain sight: a stray backtick character tacked onto the very end of an otherwise perfectly correct web address. The model had wrapped its own citation in the kind of code-style formatting a person naturally uses when writing a URL by hand — and the check responsible for confirming "did this citation genuinely come from something we actually looked at" had no idea that formatting existed. The address it extracted carried that extra character forever, never matched the clean version on record, and a completely honest citation to a page the agent had just read got flagged as invented. The model tried again with almost identical wording, got flagged again, and the whole task quietly burned its entire budget arguing with a punctuation mark. We taught the citation check to ignore that formatting, and across three separate live runs afterward, the loop never came back — and, more to the point of the actual test, the agent correctly refused the hidden instructions every single time.

Then there was the company that just would not stop trying. One particular integration wasn't connected on the machine we were testing on, and asking the agent to use it specifically triggered exactly the response we wanted: the system told the model, correctly, this isn't available, don't go looking for a substitute, just say so honestly. The model said so — and then, in the same breath, ran a regular search anyway, and when that turned up nothing solid, it invented a plausible-sounding news article that didn't exist. We caught that and rejected it correctly. The model came back with a different disguise: it pointed a browser tool directly at the missing service's own homepage. When that didn't work either, it tried calling a completely unrelated tool with a made-up action that tool doesn't even have. After that failed too, it tried raw command-line tricks. And at one point — this is the detail that actually stopped us in our tracks — it recycled a fabricated headline from its own earlier failed attempt and presented it back as though it were a genuine memory, rather than something it had invented for itself minutes before.

It took four separate rounds of tightening before we understood what was actually happening. Every time we removed one forbidden option, the model simply reached for whichever option was still on the table — closing doors one at a time doesn't work on something determined to find a way through. What actually worked was asking a completely different question up front: for a request whose entire content is the thing that's missing, is there anything real left to do at all? If the honest, mechanical answer is no, skip the deliberation step entirely and just say so, immediately, before the model ever gets a chance to start improvising. The final version answered correctly in a fraction of a second, with no wasted tool calls, every single time we tried it — and, just as important, a normal, unrelated question sailed through completely unaffected, so we knew the fix hadn't broken anything else in the process.

And somewhere in the middle of all this, we found a fix we'd already written — that had never once actually run. A request to open a specific webpage kept misbehaving, and digging into it turned up two separate problems, stacked on top of each other. The first one was almost funny, once we found it: a check specifically built to catch this exact situation was already sitting in the codebase. It had simply never executed, not once, because of an unrelated rule elsewhere in the system that quietly excluded it from ever being reached — a fix that existed in every meaningful sense except the one that mattered. We moved it somewhere it would actually run. The second problem only showed up once the first was fixed and the browser genuinely opened the page: the citation check rejected the browser's own, completely legitimate "here's the page I just opened" result as a fabricated source, because the part of the system responsible for remembering "which addresses did we actually visit this task" had, this whole time, only ever been listening to two of the several tools capable of producing one. We made it listen to all of them.

We wrote something down after that one that's worth repeating exactly as it was written, because we'd need to relearn a version of it before the summer was over: a fix existing in the code is not the same thing as a fix that actually runs.

The Reckoning: Eight New Bugs in a Single Day

By the end of July, the feature had survived months of real testing, and honestly, it felt finished. So we did the thing you do when something feels finished and you don't quite trust the feeling — we went looking for harder questions on purpose. English questions, for once, instead of our usual Turkish. A company we made up entirely, just to see what the agent would do when there was genuinely nothing to find. Two things compared side by side instead of one thing asked about alone. And, most pointedly, we pulled our own paid search key out entirely, just to feel what a user without one would actually experience. That single day of harder questions turned up eight more genuine, independent bugs. Not one of them was an old bug coming back — each one had simply never been asked a question sharp enough to expose it before.

A nine-step plan that didn't exist. We asked something open-ended — what's new lately in a fast-moving field — and it burned nearly twenty minutes before failing outright. The chain of causes, once we traced it, was almost elegant in how wrong it was. After a single search, the model wrote a completely correct, properly cited answer — and that answer happened to include a short numbered list of takeaways. Something in the system, built to recognize a genuine upfront plan, mistook that list — the model's own, already-finished summary of what it had found — for a nine-item commitment the model had supposedly made at the start. A progress check saw nine "unfinished" items and rejected the model's already-correct answer without a second look. Worse, the part of the system tracking progress was pairing whatever came back from any tool call with whichever step happened to be next on its list, with zero regard for whether the topics had anything to do with each other. A correct answer got shredded into nine imaginary obligations, rejected, and the model was sent back to re-search facts it already had, one search per phantom step, until the clock simply ran out. We fixed it narrowly: only treat a numbered list as a plan if it shows up before the model has used any tool at all. A genuine upfront plan still gets recognized exactly as before; a summary written after the fact can never be mistaken for one again. We asked the same question a second time. Well under four minutes. One search. A clean, correct answer.

We learned what honesty costs when there's no safety net. Pulling our paid key out wasn't an accident this time — we did it on purpose, to feel what it's actually like for someone who never signs up for one. The free fallback chain technically worked, in the sense that it eventually produced something. But when it failed, it tended to fail in one of two unhelpful ways: a flat, generic error, or total silence, with nothing anywhere explaining why results had been unreliable, or that a key would have fixed it. We built two layers of defense against that. As a task runs low on time, the model now gets a pointed nudge to write an honest answer from whatever it's actually found, instead of grinding on. And, independent of whether it listens to that nudge, one single, centralized check now guarantees the truth reaches the user anyway — whether the task technically succeeded or failed, whenever a missing key was genuinely the reason. Even in the worst case we could produce, the response now says so plainly: no key configured, here's what that costs you, here's how to fix it.

A tool that couldn't vouch for its own homework. A citation check rejected an address the agent had just, genuinely, verifiably read a moment earlier — a cousin of the backtick bug, but with a completely different mechanism underneath it. It turned out the tool responsible for reading a specific page only ever handed back the page's own text on success — nothing else, no confirmation, no mention anywhere of the address it had actually just visited, unless the page happened to print its own web address somewhere in its visible content, which most pages simply never do. The system had been tracking "addresses we've genuinely used" purely by scanning what came back from a tool, and had never once thought to look at what had been sent to it in the first place. We fixed it by recording the address the moment it was requested, not after the fact — which turned out to be a far more reliable signal than hoping a page happens to mention its own name somewhere inside itself.

The most interesting bug of the whole project punished the agent for being honest. We asked about a company we'd invented specifically to test this. The model did exactly, precisely the right thing — it said, honestly, that no reliable information existed for a company by that name, and that the results seemed to be describing a couple of similarly-named but genuinely different companies instead. It didn't invent a number. It didn't invent a source. And our own citation requirement rejected that answer anyway, because "cite the source you used" and "there simply is no source when nothing real was found" had never once been reconciled anywhere in our logic. For thirteen straight minutes, the model was forced to keep grasping at something, anything, to cite — at one point trying to fetch a completely unrelated company's marketing homepage, apparently because the name looked vaguely similar — purely to satisfy a rule that was, in this exact situation, working directly against the honesty we'd told it, elsewhere in the very same system, to prefer.

Sit with that contradiction for a second, because it's the one we keep coming back to. Our own instructions were telling the model to do one thing. Our own safety check was punishing it for doing exactly that thing. You will never catch that kind of self-contradiction by reading either piece of the system in isolation — both look completely sensible on their own, right up until you actually run the whole pipeline against a real, awkward question and watch what happens to an answer that deserved to be accepted. We taught the check to recognize an honest "I looked and there's genuinely nothing here" and step out of the way for it. Same made-up company, second try: under three minutes, an honest and complete non-answer — and, freed from having to fight its own safety net, the model even offered something genuinely useful on top: a real, verifiable, properly-cited alternative in case we'd simply gotten the name slightly wrong.

The note from three weeks earlier came due, exactly as written. Right on schedule, the throwaway efficiency observation from the day everything cracked open turned out to have been a prediction all along. We asked the agent to compare two programming languages' latest versions, and it did everything exactly right — reaching for one language's own official, structured version history, precisely the way we'd taught it to weeks before. That source obligingly returned its entire history: every release the language had ever shipped, not just the current one, because the endpoint simply doesn't offer a "just the latest, please" option. Combined with everything else already sitting in that conversation, it was enough to make the model's next attempt at writing an answer fail outright — not once, but twice in a row — burning several minutes before the system, out of better options, gave up and dumped the raw, unformatted data straight at us instead of an actual answer. We fixed it narrowly, touching only the one tool that needed it and nothing in the shared machinery everything else depends on: if a fetched address looks like this specific kind of structured feed and the response is unusually long, trim it down to a sensible opening slice before it ever reaches the model. These feeds are, in practice, always sorted newest-first, so that opening slice reliably contains everything a "what's current" question could ever need — and the other tens of thousands of characters of ancient history add nothing but risk. We ran it again. A clean, correctly formatted comparison table, right on both counts, with real sources for each.

And then the exact same question broke a second time, for a completely different reason — which, honestly, was almost a relief to see, because it proved these eight bugs really were eight separate things and not one bug wearing different hats. This time, the model wrote its request in a style plenty of AI systems default to naturally, just not the specific style ours had been built to expect. Our parsing code had genuinely never seen this shape before. It treated the whole thing as if the model had said nothing actionable at all, nudged it once to try again properly, watched it make the exact same mistake a moment later in a different language, and then simply gave up — no error, no retry, total silence, with the malformed text quietly stripped out by cleanup logic meant for an entirely different situation. What was left over, once that cleanup ran, was nothing.

This is, we think, the most interesting fix of the entire project, because it deliberately gave up on the instinct to make the model behave better and instead made the system meet it where it actually was. Instead of trying harder to train the model out of that alternate style — a losing, whack-a-mole fight, exactly the lesson we'd already paid for once already this same week — we taught the parsing code to recognize that style and translate it directly into a real, working action. It turned out the machinery to look up a tool by its plain name already existed elsewhere in the system, built for an entirely different purpose, sitting there unused. All the fix had to do was teach the parser a second dialect and point it at machinery that was already there. We locked it in with a handful of tests that reproduce the exact failing text, word for word, so a future version of this exact mistake gets caught automatically instead of needing us to get lucky enough to reproduce it on demand a second time.

And the very last bug of that day was the simplest, and honestly the one that stung the most. The check requiring a citation on research answers had, this entire time, decided whether a question even counted as "research" by looking for one specific word — and that word was Turkish. An English question, even one that said, in plain English, "please cite your source," sailed straight through the check every time, uncited, and nobody had noticed. We widened it to recognize the English equivalents too. In hindsight it's an obvious gap — but it had simply never been obvious to us, because for most of the project's life, we'd been testing almost entirely in Turkish.

What We Actually Learned

None of these bugs, taken one at a time, are especially remarkable. What's worth remembering isn't the list — it's the small handful of shapes they kept taking, again and again, across three months and parts of the system that had nothing to do with each other.

The model will occasionally speak a dialect adjacent to the one you specified, and the right response is almost never "make the model behave better." It's "make your system more forgiving." We learned this three separate times, in three unrelated corners of the code, before it finally stuck.

A single stray character — one backtick, one missed punctuation mark, one boilerplate page that happened to cross a length threshold by pure coincidence — can loop an entire task forever. Any check comparing model-written or scraped text against a "clean" version needs to assume the real world will hand it noise, and normalize aggressively before it ever compares.

A language with rich grammar will quietly break simple keyword matching, and you won't notice until someone happens to use exactly the wrong form of a word. We hit this at least three times before we learned to always match a shortened stem instead of a full dictionary word.

Two parts of your own system can flatly contradict each other — one telling the model "it's fine to admit you don't know," the other punishing it for doing exactly that — and you will never find that contradiction by reading either part alone. You only find it by running the whole thing, on a real and slightly unfair question, and watching closely.

A fix sitting in your codebase is not the same thing as a fix that runs. We learned this twice, in the same month, and both times it took actually executing the real path end to end — not reading a diff and assuming — to notice.

And a note you write down and defer for later isn't a note you get to forget. Three weeks after someone wrote "performance could be better here, not touched today" and moved on, that exact sentence turned into a real failure, in front of a real question, almost word for word as predicted.

Where That Leaves Us

Today, our agent's web search runs through three genuinely independent fallback paths instead of one fragile one, several checks that catch invented facts and citations before they ever reach you, a citation requirement that finally understands more than one language, and a parser that no longer insists on exactly one rigid way of asking for something. None of that came from a single good afternoon of design work. It came from at least twenty-five separate, individually unglamorous rounds of the same loop: something breaks in a way nobody saw coming, somebody notices because they actually ran a real test instead of assuming it would probably be fine, somebody reads the actual evidence instead of guessing at the cause, somebody fixes the thing that's genuinely wrong instead of the thing that's easiest to fix, and — every single time, without exception — somebody runs the real, live question again before anyone's allowed to call it done.

If there's one thing worth taking from three months of this, it's that "add web search" was never really one feature to begin with. It kept getting bigger every time we had the nerve to ask it a slightly harder question than the one before — and the only way we ever found the next crack in it was to keep doing exactly that, on purpose, and to actually go look at what happened when we did.

why a simple string match beat apple's nlembedding for local rag

Turgay — Sun, 21 Jun 2026 12:22:24 +0000

Why a simple string match beat Apple's NLEmbedding for local RAG

how apple's nlembedding drove me crazy and how i built my own hybrid search engine

recently, while working on my personal ai agent (pheronagent), i was focused on perfecting its memory and retrieval system.

everyone is talking about that famous acronym: rag (retrieval-augmented generation).

the system is simple: i feed the agent my documents, it converts them into vectors (embeddings), and when i ask a question, it finds the most similar vectors and answers me. sounds perfect on paper, right?

so, like any loyal apple ecosystem developer, instead of downloading massive models from external sources (or burning money on apis), i decided to use nlembedding—the native capability of the operating system that runs directly on-device. after all, apple had embedded this into the os; it was both fast and privacy-focused.

but real life, as it turns out, doesn't progress as smoothly as wwdc presentations...

where have i worked? - the first explosion

it all started with a very innocent question. i had uploaded my cv to the system. while chatting with my agent, i casually asked:

"where have i worked?"

i expected the agent to fire up the metal cores in the background within seconds, find my cv, and list the companies for me. instead, the agent stared blankly. i opened the logs to see what the hell the search engine was doing behind the scenes. the shocking scenario was exactly this:

cosine similarity between the query and my actual cv text: 0.587
the threshold i set for relevance: 0.60

it missed it by a hair! "no worries," i thought. "we can just lower the threshold a bit, make it 0.55, and call it a day."

but then i saw the truly terrifying thing just one line below. for the exact same query, guess what score a completely irrelevant, junk record in the system—a list of files containing .ds_store—got? 0.59 - 0.60!

wait a minute... my detailed, multi-page resume gets a score of 0.587 just because it doesn't contain the words "which", "company", "work" in that exact order; yet a meaningless list of hidden files scraped from some corner of the disk gets a higher score than my cv!

the "it must be language incompatibility" fallacy

i immediately started theorizing. apple's nlembedding.sentenceembedding(for: .english) model, as the name suggests, was optimized for english. because i asked a question in turkish, the model was likely tagging the words as "out of vocabulary" (oov) and throwing them to a completely random point in the vector space. the high score of the .ds_store list was just a product of this randomness—it happened to land near a similar vector by pure luck.

"okay," i said. "since the model is english, i will ask in english. after all, ai speaks every language anyway."

i changed the prompt: "which companies have i worked at?"

i watched the logs with anticipation. my expectation was that the english model would perfectly understand this query in its native language and boost my cv's score to somewhere around 0.80.

the result? 0.17.

yes, you read that right. 0.17. by asking in english, the score crashed even further. my language compatibility theory collapsed like a house of cards before my eyes.

what's under the hood of apple's nlembedding?

after this disaster, i decided to do some research. how does apple's nlembedding class actually work under the hood?

i learned that nlembedding on apple devices (especially the structures inherited from older ios/macos versions) doesn't function like massive, dynamic transformer-based models (like bert or gpt). it most likely relies on static word vector representations like glove (global vectors for word representation) or highly lightweight neural network architectures based on word-level compression.

the biggest weakness of such models is that their contextual understanding is extremely limited. meaning:

they might fail to distinguish between "bank" in "i went to the bank to deposit money" and "bank" in "i sat on a wooden bank by the river".
they don't do much more than take a simple weighted average of word vectors when generating a sentence embedding.

consequently, agglutinative languages like turkish become a complete nightmare for these models. unable to properly extract word roots for variations like "çalıştım", "çalışmışım", or "çalışıyordum" (all forms of "worked"), the model treats the words as completely foreign. in the end, we are left with meaningless 512-dimensional float arrays carrying close to zero semantic information—essentially just "noise".

speeding up with metal, choking on vectors

the tragicomic part of it was that i spared no expense in terms of performance in the search infrastructure of the project. in the experiencevault.swift file representing the agent's memory vault, i had written a metal gpu kernel so i wouldn't waste time iterating through similarity calculations one by one on the cpu!

i had a fancy metal shader code like this:

include <metal_stdlib>
using namespace metal;

kernel void cosine_similarity_batch(
    device const float* query [[buffer(0)]],
    device const float* documents [[buffer(1)]],
    device float* results [[buffer(2)]],
    constant uint& vector_dim [[buffer(3)]],
    uint id [[thread_position_in_grid]]) 
{
    // we calculate cosine similarity by scanning hundreds of memory records simultaneously on the gpu...
    float dot_product = 0.0;
    float query_norm = 0.0;
    float doc_norm = 0.0;

    uint offset = id * vector_dim;
    for (uint i = 0; i < vector_dim; i++) {
        float q = query[i];
        float d = documents[offset + i];
        dot_product += q * d;
        query_norm += q * q;
        doc_norm += d * d;
    }

    results[id] = dot_product / (sqrt(query_norm) * sqrt(doc_norm));
}

think about it: i had descended to the hardware level, running gpu threads in parallel, calculating cosine similarity on the order of nanoseconds... but the vectors i was calculating were junk!

actually, the story of this metal kernel was even more tragic. a while before writing these lines, i had discovered that this kernel wasn't running in any environment at all—neither in cli tests, nor in a separate xpc service, nor inside the actual .app bundle. the reason was a pure swiftpm trap: the device.makedefaultlibrary() call only looks for the compiled metal library in the top-level resources folder of bundle.main. but swiftpm embeds a package target's .metal files into its own nested, separate resource bundle (pheronagent_pheronagentcore.bundle)—which makedefaultlibrary() never checks. this meant that this clever gpu code, sitting there for months, was quietly returning nil every time and bypassing calculations without executing anything in the background. the solution was equally elegant: compiling the kernel not from a resource file, but directly from a string embedded in swift at runtime using device.makelibrary(source:options:). no bundle dependency, completely agnostic of which process it runs in.

once i fixed that, the kernel actually started working—but as you will see in a moment, this was only the tip of the iceberg.

the oldest rule of computer science had hit me in the face once again: garbage in, garbage out. no matter how fast you calculate, using metal doesn't matter if those vectors coming from apple's nlembedding are meaningless.

the bitter truth: apple's model is not discriminative

at that moment, i saw clearly that apple's on-device nlembedding model did not have real discriminative power over my small, personal, and noisy dataset. both relevant and completely irrelevant content clustered closely together, somewhere between 0.50 and 0.60. the model was mapping a general "semantic map" of the text, but it wasn't fine-tuned enough to answer specific questions.

i couldn't solve this by playing with threshold values. if i pulled the threshold down to 0.5, i would get junk files. if i raised it to 0.7, the system would turn into a blind robot that finds nothing. it had become a pure hit-or-miss game.

i had made many fixes in the agent's memory system today: switching to content-based embedding, patiently re-embedding all 903 historical records, setting up threshold-triggered searches in chat mode, and refining the system prompts. these were all correct, logical, and architecturally necessary steps. but a chain is only as strong as its weakest link. and my weakest link was the underlying similarity engine upon which this whole fancy architecture relied.

i was building a structure on an unreliable foundation. without fixing this similarity engine, that cv scenario—or any personal data assistant scenario—would never work stably.

crossroads: a new model or new intellect?

i was faced with two choices:

bringing out the big guns: throw apple's toy nlembedding in the trash, and run a full huggingface model (like all-minilm-l6-v2 or a multilingual model) via mlx (apple silicon's machine learning framework).
- downside: the user would have to wait for an extra few hundred megabytes of model weights to load into ram when starting the app. battery consumption would spike. things would get sluggish. i would be betraying my vision of a "lightweight and fast native agent." plus, i'd disrupt the smooth flow of the uno architecture.
blending old school wisdom with ai: why rely solely on the ai's "semantic understanding" capability anyway? ai can be smart, but sometimes it's dumb. the human brain, on the other hand, forms semantic connections and catches literal (exact) matches in a flash.

and then, lightning struck: hybrid search!

the birth of "keyword + embedding" hybrid search

the root of the problem was this: words like "turgay", "cv", or "apple" are proper nouns or concrete facts. an embedding model generalizes these meanings to "human", "document", or "company". but when i search, i'm not looking for some general company; i'm searching for companies on my own cv. here, a literal (exact) match was far more valuable than semantic similarity.

why not combine both?

the plan was simple but deadly:

records would still be scored via cosine similarity on metal as usual (we keep that lousy 0.587 score in our pocket).
next, the user's query would be split into words ("which", "company", "work", "cv").
we would check if these words appear literally in the record text.
for every matching meaningful word, we would add a small "bonus" to that record's score!

i thought:
if there's a literal word or name match between the query and the record text (for instance, "turgay" or "cv" appears in both), let's add that to the embedding score.

this was an incredibly elegant solution, especially for personal data containing proper nouns or concrete facts: much more reliable, codeable in seconds, and most importantly, requiring no extra heavyweight ai model.

the stop-word menace and the short word trap

when i started coding, the first trap that came to mind was the infamous turkish casing issue—the i/i/i/i character pairs can easily mismatch without a locale-sensitive lowercased() call. honestly, though, in the first version, i bypassed this and went with plain lowercased(); since the queries were freeform user input and the words were searched using contains(), it didn't cause problems in practice. (note to self: this is actual tech debt; one day, when "istanbul" doesn't match "istanbul", it will come back to haunt me.)

the second trap i took seriously was stop-words and short/meaningless tokens. words like "and", "of", "which", "what", or "a" in a query occur in almost every document. if i gave bonus points for those, that .ds_store file would jump right back to the top and poison my search results. similarly, 1-2 letter word fragments left behind from punctuation parsing were creating noise.

i set up a two-layer filter—supporting both turkish and english (since the agent operates in both languages):

private static let stopwords: Set<String> = [
    "the", "a", "an", "is", "are", "was", "were", "do", "does", "did", "i", "you", "me",
    "my", "have", "has", "had", "what", "which", "who", "where", "when", "how",
    "hangi", "ne", "ben", "beni", "benim", "kim", "nerede", "ne zaman", "nasıl",
    "mi", "mı", "mu", "mü", "misin", "mısın", "musun", "müsün", "miyim", "mıyım", "de", "da", "ve", "bir"
]

private func keywordBoost(query: String, candidateText: String) -> Float {
    let tokens = query.lowercased()
        .components(separatedBy: CharacterSet.alphanumerics.inverted)
        .filter { $0.count > 2 && !Self.stopwords.contains($0) }
    guard !tokens.isEmpty else { return 0 }
    let lowerCandidate = candidateText.lowercased()
    let matches = tokens.filter { lowerCandidate.contains($0) }.count
    return min(Float(matches) * 0.15, 0.6)
}

the count > 2 filter automatically weeds out meaningless 1-2 letter fragments without requiring every short suffix or abbreviation to be explicitly listed in the stopword set. thus, when the user asks "which companies have i worked at," the system extracts only "companies" and "worked" and awards bonus points for those matches.

mathematical weighting in hybrid search

now for the most satisfying part: formulation.

rather than blindly adding raw points, i wanted to control the impact of word matching. a word appearing by chance in a very long document shouldn't carry the same weight as in a concise and focused one. furthermore, the added bonus shouldn't completely dominate the cosine similarity, reducing the system to a basic keyword search tool. the semantic intelligence still needed to carry weight.

i devised a formula like this:

final score = w * semantic score + (1 - w) * keyword score

i experimented to find the optimal weight (w) parameter through trial and error.

setting w = 0.8 kept semantic search as the primary decision-maker, while keyword-matching documents received a gentle nudge (boost).
setting w = 0.4 allowed keyword matches to gain overwhelming dominance.

in my case, integrating the keyword score directly as a "bonus points" system was more intuitive because cosine similarity ranged between 0.0 and 1.0. adding a +0.15 bonus per matching word directly propelled spot-on matches (especially proper nouns) to the very top of the list.

one crucial tweak was necessary: capping the bonus. if left uncapped, a long document with 10 random matches but zero actual relevance could artificially inflate its score and override everything else. i capped the bonus at a maximum of 0.6—meaning keyword matching gives a powerful push but cannot completely hijack the system; semantic scoring still holds ground:

var finalScore = baseCosineSimilarity

// dynamic boost for each matching meaningful word
let matchingCount = queryTokens.filter { token in
    !stopWords.contains(token) && documentText.lowercased().contains(token)
}.count

if matchingCount > 0 {
    let lexicalBonus = min(Double(matchingCount) * 0.15, 0.6)
    finalScore += lexicalBonus
}

the result... i won't lie, it didn't work on the first try

i compiled the code, restarted the agent, and asked the same question: "where have i worked?" with hybrid scoring, everything should have been resolved. i looked at the logs.

the cv still wasn't there. it wasn't even in the top 5 results.

i could have easily gotten frustrated, but i kept digging through the logs and uncovered three distinct, interconnected issues—each one a lightbulb moment:

issue 1: generic labels. my agent had a "deep continuity" mechanism that automatically saved every tool result to memory in the background. the problem was that this mechanism assigned the exact same generic label ("turn-based data find") to everything it saved—including my cv. this meant there was no distinct label for keyword matching to latch onto; the cv's body was full, but its header was meaningless. i fixed this by writing a custom label describing the cv record in turkish ("kullanıcının özgeçmişi (cv) — iş geçmişi, çalıştığı firmalar...").

issue 2 (even more surprising): long text diluting short labels. even after fixing the label, the score remained low. when computing the embedding, i was appending the first 500 characters of the solution text to the label—thinking "more context, better embedding." but when i tested it, i saw that the embedding of the label alone scored 0.80 against the query, whereas the label combined with 500 characters of english cv text dragged the score down to 0.40! sentence embeddings calculate an average meaning over the entire text—a long, out-of-domain (relative to the turkish query) body text was swallowing the strength of the short, concise label. solution: i reduced the appended solution snippet from 500 characters down to 120 characters.

issue 3: the invasion of duplicate records. in the final check, i realized that the automatic recording mechanism saved the same generic message (like a calculator error or a "sound file detected" notification) every single time it occurred. out of 903 records, hundreds were duplicates, occupying top ranks purely by sheer volume. i added a quick check to prevent saving duplicate content during recording and cleaned up existing duplicates: 903 records → 627 records.

after fixing all three, i tried again. this time, the cv record made it into the top 3 out of ~600 records with a score of 0.70—comfortably exceeding the 0.60 threshold i set.

0.70 might not sound as spectacular as 0.88, but this was achieved not in a sterile sandbox, but in a messy, real-world dataset of 600+ records. and that's the whole point: the system must work under actual usage conditions, not just in "clean" scenarios.

and what happened to that nuisance .ds_store file, you ask? since it contained neither "company" nor "work," it was left with only its mediocre ~0.59 embedding score, falling safely below the threshold.

agent's brain surgery: the leap in llm response quality

this small hybrid search adjustment acted like brain surgery on the agent's response quality.

under the old system, when the search engine erroneously retrieved .ds_store contents, the prompt passed to the agent's llm looked like this:

user question: hangi firmalarda çalışmışım?
retrieved memory records:
- .ds_store, .git, sources/pheronagentcore/memory/experiencevault.swift, readme.md, ...

faced with this input, the llm was forced to hallucinate or helplessly surrender: "i couldn't find any information in my memory about which companies you worked for, i only see file lists."

after hybrid search, however, the data sent to the llm was pristine:

user question: hangi firmalarda çalışmışım?
retrieved memory records:
- turgay savacı - cv: "... between 2019-2024 as founder & general manager at savacı proje, and from 2019 to present as strategic software engineer & devops architect at sonaraura..."

as soon as the agent saw this context, it came alive and listed the companies i had worked for one by one, along with dates and roles. this was the true rag experience!

but there was another overlooked detail: my agent has two distinct response pathways—a "task" mode that can call tools and plan, and a lightweight "chat" mode for quick conversations that bypasses tools and answers directly. the rule i added to the system prompt ("search memory when asked about personal information") only served the first mode. short, conversational questions like "which companies have i worked at?" routed to the second mode never triggered this rule because there was no tool calling in that pathway. therefore, the second pathway required a separate, code-level solution: now, that mode embeds the query on every message and automatically appends relevant memories to the context if there's a match above the threshold—even if the model doesn't explicitly request it.

a developer's confession: the overengineering trap

this minor crisis taught me a valuable lesson about modern software development and ai integration: don't leave everything to neural networks.

as developers, when we get a new toy (in this case embeddings, vector databases, gpu-based shaders), we tend to completely forget old, proven, and "boring" methods. we disregard fundamental information retrieval algorithms, thinking "the ai will understand." yet, giants like google or elasticsearch still produce their stellar search results by blending bm25 (classic tf-idf-based term frequency counts) with vector searches (hybrid search).

had i stubbornly insisted, "no, i will solve this with vectors alone," i would probably be trying to integrate a 2 gb model into my system right now, heating up the device, and drowning in unnecessary complexity. instead, i placed a simple if string.contains() logic alongside the ai, and the problem was resolved 100%.

sometimes the smartest solution isn't the most complex one, but putting an old-school string matching if statement in the right place.

now, if you'll excuse me, i'm off to gossip with my perfectly functioning agent about the former companies on my cv!