DEV Community: Thorin Tabor

A hallucinated module, a backfiring RAG pipeline and the MCP server that fixed it

Thorin Tabor — Wed, 07 Oct 2026 18:34:52 +0000

It’s a surprisingly humid day for San Diego. We’re running a workshop, teaching GenePattern to a dozen medical students, and we’re about to demo something in public for the first time: GenePattern Copilot, our first generative AI offering. Sweat drips. What could go wrong.

One of the students types an innocent question into the chat window: What GenePattern modules are available? The answer comes back, and most of it is exactly what you’d expect: Gene Set Enrichment Analysis, Hierarchical Clustering, PCA. The usual suspects. Then he says something nice about the platform. He’s glad to see we offer scPerturb, a toolkit for single-cell perturbation analysis.

I cringe. It’s sitting right there on the screen in the list the assistant just generated. And I know it’s a hallucination.

We don’t have scPerturb. It’s a good idea for a module, one we should probably build someday, but it does not exist on GenePattern, and the assistant just told a room full of medical students that it does.

I don’t correct him in front of the group. I let it slide and make a mental note to fix it, which ends up taking longer than I expect.

None of that makes sense without knowing about GenePattern. It was a platform for reproducible bioinformatics research, taking tools built by labs worldwide and wrapping them into something a non-technical scientist could run without learning to code. The mission was accessibility.

Fast forward to early 2024. Generative AI was blowing up, and it looked like an obvious fit, even for a lab that spent most of its history doing traditional Bayesian AI rather than anything resembling deep learning.

A colleague got there first. He put together an early prototype of GenePattern Copilot, then left the group. I inherited the project without much ceremony. One day I didn’t have a generative AI project, and the next I had a mandate to turn a rough prototype into something we could put in front of real users.

The credit for the original idea belongs to him. Everything downstream of that, for better or worse, is mine.

Inheriting a moving target

I inherited a LangChain project written in Python, capable of switching between models through AWS Bedrock. At the time that meant Llama, GPT, Claude and Mistral, each with its own quirks, and each compared against the others manually.

I quickly found manual, informal comparisons between models to be unwieldy, and it only got worse once I started varying what data went into the RAG pipeline.

The assistant was never meant to be just a wrapper around an inference API. Its main value-add was the retrieval layer: a ChromaDB database, OpenAI embeddings and a RAG corpus built from our documentation, a decade’s worth of help forums and a handful of Jupyter notebooks. In theory, this gave the model everything it needed to answer questions about the platform and the underlying bioinformatics.

At this point, the prototype’s eval process was someone reading outputs and deciding if they looked right. The first real work I did on this project, beyond some basic code cleanup, was building a proper eval suite. I needed a concrete way to measure how well the assistant performed.

The RAG that made things worse

Once the eval infrastructure existed, I ran the most obvious comparison: the assistant with full RAG retrieval turned on against the same model given a raw query and no retrieval.

I expected the RAG version to win comfortably. That’s the whole point of building a RAG pipeline.

Surprisingly, the assistant with RAG performed worse, and not by a narrow margin.

The raw model, with no access to our documentation or forum data, outperformed our fully retrieval-augmented assistant on almost every substantial question I threw at it.

That result sent me digging through the LangChain graph trying to figure out what was happening at each step, but I quickly ran into a wall: I had basic logs, but they weren’t structured in a way that made it easy to trace a single request end to end.

So I improved that logging by hand and was eventually able to track down the issue. Later we would move to a real observability platform, Logfire. But in the moment, it was mostly a lot of scrolling through raw output trying to find the thread.

Getting the chunking right

The cause of the problem was in how we chunked the corpus, particularly the help forum data.

We used a naive, token-based chunking approach, the kind every RAG quick-start walks you through, and I discovered that it routinely split a forum thread’s question from its answer. When the vector search ran, it pulled in the question fragment, along with loosely related noise, and that got stuffed into the context window without actually picking up the answer.

The signal-to-noise ratio was bad, and that degraded the model’s reasoning.

The fix that worked was, before ingestion, preprocessing the corpus using an LLM-based fact extraction strategy. Instead of chunking documents by token count, I had a model read through each source and break it down into atomic statements of fact. Each statement was written to stand on its own, with ambiguous pronouns replaced by the nouns they referred to. Something like “GenePattern supports Gene Set Enrichment Analysis” became its own row in the vector database rather than a fragment of a longer paragraph that only made sense in context. That change alone measurably improved answer quality.

From this I learned that there is no one-size-fits-all chunking strategy. What works depends on the shape of the data you feed it. Help forum threads, module documentation and notebooks all have a different internal structure. Treating them identically at the chunking stage was the problem.

The module list solution

Which brings me back to that workshop. By the time we ran it, the chunking fix was in place, and the assistant’s answers had gotten better across the board. Still, module hallucinations were a persistent problem. And it wasn’t a RAG issue: even with no retrieval involved, calls to the model routinely invented plausible-sounding GenePattern modules that never existed.

I tried a few approaches for fixing the hallucination issue before discovering one that worked. The solution was dumping a complete list of every real module name into the system prompt, along with an explicit instruction not to reference anything outside that list.

There were only a few hundred modules at the time, fitting comfortably inside the context window.

But while this was a working solution, it was also a band-aid. The list was accurate the day I wrote it and stale the moment we shipped a new module or deprecated an old one.

Maintaining a hardcoded list by hand, forever, was not a good longterm plan. What I wanted was something that could generate that list dynamically so it stayed current without anyone having to remember to update a prompt.

Enter MCP

That thought turned into the next phase of the project. Rather than hardcoding the list of modules, I built a tool that the assistant could call to retrieve the current list from the server.

That tool ended up being one small piece of something bigger: a GenePattern Model Context Protocol server that wrapped the platform’s REST API, allowing the assistant to take actions on a user’s behalf.

By the time the MCP server was done, a user could ask about their running jobs, check the provenance of the data behind a result, launch new analyses, check statuses, retrieve results, summarize them or hand them off to a downstream visualization step—all through the same conversational interface.

Still, MCP was new at the time, and LangChain’s support for it was early. Almost immediately I ran into sync and async issues trying to get LangChain’s MCP integration to cooperate with the rest of the pipeline. I spent more hours than I’d like to admit fighting basic plumbing problems that had nothing to do with the actual AI work I was supposed to be doing.

Finally, after chasing down yet another LangChain bug, I decided to migrate the project to Pydantic AI, which was working well for me at the time in a separate project, the GenePattern Module Toolkit.

Pydantic AI gave me better output validation, noticeably better developer ergonomics and none of the sync/async headaches that kept surfacing in LangChain’s MCP support. I never looked back.

Trimming the toolbox

The first, naive version of the MCP server was, predictably, too ambitious. I wrapped nearly every endpoint in the GenePattern REST API on the theory that more capability was strictly better. It wasn’t.

In MCP, every tool comes with a description that gets injected into the model’s context whether or not that tool is relevant to the current conversation. With that many tools available at once, some models started to get noticeably confused about which one to reach for. At the time, MCP did not offer progressive disclosure, just a wall of tool descriptions competing for attention.

This needed fixed. So rather than wrapping all available REST endpoints, I began to think of MCP tool sets in terms of what user stories we wanted to enable, trimming unnecessary tools from the list. This made an immediate difference in how reliably the assistant picked the right tool.

From there I started iterating on the tool descriptions themselves, which quickly turned into its own small eval suite. Evals tested whether a specific tool should or shouldn’t be used, checked against whether a given model picked the right one on the first try or wandered into a wrong tool call and had to recover.

For a while I had already been versioning our main prompts, but when I extended that same discipline to the MCP tool descriptions, it became immediately clear how much a single word choice could change tool selection behavior.

What full agentic looked like

MCP is what let GenePattern Copilot grow from a question-answering chatbot into something closer to a real agent: one that could run jobs, retrieve results and help a user explore the platform’s available tools.

By the time the project reached its mature state in mid-to-late 2025, the eval suite had grown into a full prompt library, built partly from real usage patterns and partly from the direction we wanted to push the assistant. Each prompt in that library was checked against a known ground-truth answer.

We could also run an eval matrix across combinations of model and RAG source to see how each performed relative to the others. We used LLM-as-judge to score responses against that ground truth, tracked the cost of each configuration and kept every prompt version tied to the eval results it produced.

As for that workshop demo, there was never a dramatic rematch where I proved that the hallucinations had been vanquished for good. Rather, the fix happened quietly, over days and then weeks of iteration, mostly unnoticed by anyone but me and the eval suite.

Orchestrator, planner, six specialists: a multi-agent pipeline that survived production

Thorin Tabor — Thu, 30 Jul 2026 17:59:23 +0000

It's five o'clock on a Wednesday, and I have twelve tabs open across two monitors. Four of them are Google searches. One is a Biostars thread from a few months back that I've had bookmarked for three days now, because it's the closest thing anyone has written to what I'm trying to understand. Another is a conversation with an AI assistant that's now twenty messages deep, patiently walking me through spatial transcriptomic gradients like a TA answering the same question for the fifth time this semester.

I'm not a biologist. I'm a software engineer by training. And I have been assigned to integrate the spatialGE suite of tools into GenePattern, a bioinformatics platform used by researchers who mostly just want their data to turn into an answer to their scientific inquiries.

This isn't a task I can fake my way through. spatialGE consists of a non-linear workflow with at least nine distinct steps and speaks two separate data formats. The modules I'm building have to work together as a coherent, containerized, multi-step pipeline instead of nine stand-alone scripts that happen to share a folder. Getting that right means actually understanding when a researcher would reach for enrichment versus differential expression versus one of half a dozen normalization schemes, because if I get that wrong, the module is wrong no matter how clean the code underneath it is.

The people who wrote spatialGE are not in the room. They're not on Slack. As far as I can tell they're two time zones and a very long email chain away, and I've decided not to bother them until I've exhausted every other option. So it's just me, the documentation and however much of Biostars I can absorb before I have to go make dinner.

This isn't really a story about spatial transcriptomics, though it starts there. GenePattern has been around since the early 2000s and the idea behind it has aged well: researchers don't want to spend their afternoons untangling dependency trees or memorizing the command-line flags for a tool some grad student wrote eight years ago. The platform takes hundreds of independently developed bioinformatics tools, most written by academic labs with no particular interest in software engineering practices, and wraps each one in a user-friendly web form. It's a translation layer between "here is a tool that does interesting science" and "here is something a non-programmer can actually run."

The catch is that every one of those tools has to be integrated by hand: someone sits down with the documentation, or with the scientists who built it if they're lucky enough to get a response, and figures out which parameters actually matter, what the true dependencies are, how to wrap it in a containerized environment and how to test that all of it works.

For spatialGE, that someone was me, and by the time this story is over, that someone was going to become a pipeline of AI agents that eventually pushed our module output up by roughly 500 percent.

The toolkit gets its green light

We were a small team with a long backlog. Every new tool integration was traditionally a multi-day undertaking: read the docs, talk to the lab if you could, write the wrapper, write the manifest, write the tests and hope you haven't missed an edge case a scientist would find obvious.

I'd just come off a different corner of the GenePattern ecosystem: a project called GenePattern Copilot, which was an agentic assistant that could answer platform and genomics questions, as well as execute tasks on a user's behalf. This meant agentic AI was already on my mind. So a week into teaching myself spatial transcriptomics by trial and Biostars, I had the thought that eventually occurs to every engineer stuck doing something repetitive and well-specified: there has to be a better way to do it.

I sketched the architecture on my whiteboard, not a single model doing everything end to end, but a hierarchy: an orchestrator agent at the top, and underneath it, a set of agents each responsible for producing one specific artifact a GenePattern module needs. I brought the sketch to my boss. It got the green light, and the GenePattern Module Toolkit was born.

The idea was simple to state and much harder to build. Point the toolkit at a tool's documentation and its git repository, optionally tell it which user stories or features to prioritize, and let it do the work. What comes out the other end is a module a human can QA in an afternoon instead of building from scratch over three days.

One orchestrator, eight agents

Here's roughly what the toolkit looked like once it settled into a stable shape. At the top sits an orchestrator, managing a pipeline of eight subagents. The first two that get called are a researcher agent, which pulls documentation and context from online sources, and a planner agent, whose entire job is making sure every other agent is working from the same set of assumptions. Execution is then passed to a pipeline of six artifact agents, each responsible for a single component: manifest, wrapper script, parameter groups, documentation, tests and Dockerfile.

The artifact agents use a pattern called chain of verification, though at the time I was just calling it "check the thing before moving to the next thing." Each agent generates its artifact. A linter script then runs against that artifact and checks whether it's actually valid. If it passes, the pipeline moves to the next agent. If it fails, the error gets escalated to the orchestrator, which has to make a judgment call: is this a bad run of this one artifact, or is it symptomatic of an upstream failure, something an earlier agent produced? If it's upstream, the orchestrator kicks the pipeline back to that earlier step and reruns from there. This is gated by a maximum number of iterations before it gives up and flags the run for human evaluation.

Starting with the scaffolding, not the model

GenePattern was old and had grown organically enough over two decades. It never had a formal definition for what a manifest file was supposed to contain, let alone the rest of its artifacts. This was a problem that needed to be solved before I could even think about agents: there was no ground truth to check anything against.

So the first thing I built for an AI-driven project was not AI. It was linter scripts, effectively defining a de facto standard for what a valid GenePattern manifest, and the other artifacts, look like.

Manifests are Java properties files. Every required key has to be present, every value has to be of a type the platform expects and a literal = inside a value has to be escaped or the parser reads it as the start of a new key. None of this is exotic. It's the kind of thing a human engineer internalizes after their third or fourth manifest and then never thinks about again. It is also exactly the kind of detail an LLM will cheerfully get wrong, because nothing about "write a manifest" tells it that an unescaped = is fatal.

The linters caught missing keys, wrong value types, unescaped = characters, dependencies missing from the Dockerfile and property names that quietly drifted between artifacts that were supposed to agree with each other. Those scripts would go on to become the backbone of the chain-of-verification pattern, but at the time they were just me trying to codify rules that had only ever existed as institutional knowledge.

Is it strange to start an agentic AI project by writing rigid, deterministic scripts that have nothing to do with the model? I don't think so. Every project I've built has reinforced the same lesson: the harness around a model matters at least as much as the model itself. Without it you don't have an agent, you have a very confident guess.

Wiring the linters to the AI

With the linters working on their own, the next job was hooking them up to something that could actually write the artifacts they were checking.

For the agent framework I picked Pydantic AI because it's fast, lightweight and good at structured output. This mattered because nearly everything in this system, from planning a module to writing its manifest, depended on getting the model to hand back data in a shape you can work with rather than a paragraph you have to regex apart.

Once that plumbing was in place, the real work started: iterating on prompts, which is a deceptively glamorous phrase for staring at outputs and trying to figure out why an agent decided a parameter should be an integer when it should have been a float. More than once, an artifact would sail through its own generation and then fail the linter for something that looked cosmetic, such as a manifest key spelled slightly differently than the wrapper script expected, and the chain-of-verification loop would kick the run back to the step that actually introduced the problem.

That's the pattern doing its job: catching the kind of small, specific mistake that wouldn't announce itself until a researcher tried to actually run the module and it silently used the wrong default.

Evals before anything else

My last project, GenePattern Copilot, taught me a lesson the hard way: if you don't build a formal evaluation suite from the beginning, you end up debugging by vibes, and vibes do not scale. I started this project determined not to repeat that mistake.

Building a good eval suite from scratch is often a project of its own, but GenePattern had one advantage: it was a mature platform with a large corpus of known-good modules already sitting in production. That corpus became my known-good outputs, and I worked backwards from there, generating synthetic inputs that should produce something close to those known-good modules. Early runs of the toolkit added to that suite too, mostly by failing, with each bad output becoming a documented failure state I could check future runs against.

Where the breakages happened

A single input-and-output eval may be enough for a system with one model doing one job. It isn't enough for a pipeline with eight subagents, each with its own inputs, outputs and failure modes. I needed to evaluate each one independently, or I'd have no way of knowing which agent was actually responsible when something broke downstream. That's where nearly all of my early failures lived, and this project is what taught me that in a multi-agent system, observability isn't optional.

Once one agent's output becomes another agent's input, a small ambiguity introduced three steps upstream can surface as a completely unrelated-looking failure two agents later. Trying to debug that by reading logs after the fact felt like reconstructing a car crash from the skid marks. I needed to watch it happen with better clarity.

I picked Logfire for this, mostly because it's built by the same team behind Pydantic AI and the integration took 10 minutes. This let me trace a single request from the initial prompt through the orchestrator and out through each subagent in order. More importantly, it also let me see exactly where the context handed from one agent to the next started to drift from what the next agent needed. Once I had that visibility, progress on stability went from slow and confusing to fast and specific.

Learning to survive a crash

A full run of the toolkit was not fast. Between the researcher agent pulling context from the web, the linter scripts running their checks, a Docker image building as part of the containerization tests and the LLM calls themselves, a typical run took twenty to thirty minutes. And in the early days, that half hour was not guaranteed to finish.

Inference calls hung. Web requests from the researcher agent hung. Docker builds ran out of disk space partway through and died. Agents hit their maximum number of retries without resolving whatever had gone wrong. None of this was exotic. It was just what happens when you chain together enough steps that each have their own independent failure modes.

The fix started small. I wanted a way to test one step of the pipeline in isolation without rerunning the whole thing from scratch every time, so I started to serialize the current state of the pipeline to disk as a status.json file. This also became the artifact each agent handed off to the next. From there it was easy to implement a flag letting a run resume from an existing status.json. A handful of additional flags let me target specific steps or ranges of steps directly. This made both testing and recovery faster.

What I'd actually built, without ever calling it that at the time, was a crude, hand-rolled version of durable execution: a workflow whose state survives the process running it, so that a crash costs you the time since your last checkpoint instead of the entire run. That wasn't my initial goal. My goal was to get my afternoon back after watching a Docker build die partway through for the third time. But once it existed, I couldn't unsee how much of the pipeline's actual reliability was coming from that one property rather than from anything about the agents themselves.
The planner agent

The planner wasn't part of my original design. It got added later because of a specific, repeated failure: the agents kept producing artifacts that were each individually correct, but which didn't agree with each other. The manifest agent might settle on one set of parameter names while the wrapper script agent settled on a slightly different set. Each would pass its own linter checks in isolation, then fail the moment you tried to assemble them into one working whole.

The planner's job is to make that decision once, upstream of the six artifact agents, instead of letting the six agents make six independent guesses. It looks at the researcher agent's report and figures out the right general approach: an R script gets handled differently than a Python library, which gets handled differently than a raw shell script or a compiled C binary, etc. It decides which parameters are worth exposing, and settles on the actual names and descriptions those parameters will have in GenePattern's interface. All this becomes shared context for every downstream agent.

Handing it to the interns

Once the pipeline was running reliably for me, it was time to see if it worked for anyone else. So we handed it to our interns and asked them to use it for the modules already on their plate.

It broke almost immediately. Runs failed constantly at the planning stage, or hit their maximum iteration count in the first two artifact stages before anything useful came out.

Part of this was a decision made above my pay grade, and a reasonable one. We'd just had an API key leak that racked up a sizable bill before anyone noticed, and the lab had no interest in handing a new set of keys to a room full of interns. Instead, we set them up with a local Qwen model we'd tested ourselves and found passable, albeit not nearly as strong as Claude.

This worked for me. It failed for most of them. I still don't have a clean explanation for the gap between my machine and theirs, but the failure I could actually see and fix was structural: Qwen produced malformed, structurally hallucinated JSON (e.g. invented fields, wrong nesting, values in the wrong shape entirely) far more often than Claude did.

I spent some real effort trying to close that gap by fine-tuning Qwen at the San Diego Supercomputer Center, and in the end achieved something like a 50 percent reduction in the structural hallucination rate. This was good progress, but it wasn't enough.

So I made the call: I stopped trying to optimize the model further, and instead threw together a lightweight hosted frontend on AWS, backed by Claude via Bedrock. After that, the reliability problem which had eaten weeks of intern time mostly stopped being a problem.

Conclusions

Once the interns had their turn and the production hardening was done, we rolled the toolkit out to the rest of the GenePattern team. As a result, module production went up roughly 500 percent. And along the way I learned a few lessons.

The scaffolding around a model—the linters, the schemas, the harness that tells it when it's wrong—deserves to be built before the model touches anything, not bolted on once things are already unstable.

A strong planner sitting upstream of your artifact-generating agents will save you from an entire category of bugs that look like model-quality problems, but which are actually coordination problems. Six correct answers that don't agree with each other just aren't that useful.

Observability isn't optional in anything where one agent talks to another. That's where a lot of non-obvious bugs can hide, rather than in any single agent's output. Before implementing observability, I spent more hours confused than I want to admit.

Don't overlook durability. Any workflow that runs long enough to be worth building is going to run long enough to fail partway through, and if you haven't decided in advance what happens when it does, you'll end up deciding it anyway, at 5pm, under worse conditions than if you'd just planned for it.

Sometimes the scaffolding's real value is proving, with evidence, exactly where a model's ceiling sits, turning the decision to replace it into a documented engineering call instead of a shrug.

None of these lessons are especially profound on their own. But together they compound: Solving the coordination problem with a planner made the durability problem more visible. Better observability made the coordination problem easier to fix. None of it works if the scaffolding underneath is an afterthought.

That's not a spatialGE story, or even a bioinformatics one. It's just what happens when you build something agentic that has to survive contact with real work.