DEV Community: Sowaiba Arshad

My RAG app confidently made up a drug dosage. So I rebuilt it to say 'I don't know'.

Sowaiba Arshad — Thu, 30 Jul 2026 09:49:35 +0000

⭐ Star the repo: ClinicaQuery-AI on GitHub
🔔 Follow me here on Dev.to and on GitHub. I build full-stack AI systems and write up every ugly detail, including the parts that took four hours to fix.

I asked my medical research assistant a question about a paper it had never seen.

It answered anyway.

Fluent, confident, correctly formatted, with a citation and a green "high confidence" badge. Every part of that answer was invented. The paper it cited was about cardiovascular exercise. The question was about antibiotic dosing.

Nothing in my system was broken. That is the terrifying part. Retrieval ran, found the five least-irrelevant paragraphs it had, handed them to the LLM, and the LLM did what LLMs do: it produced fluent text from whatever you give it. There was no error, no exception, no log line. Just a wrong answer wearing a lab coat.

This is the failure mode nobody warns you about when you build your first RAG app. Your system cannot tell the difference between "I found the answer" and "I found five paragraphs." It has no concept of failure, so it can never report one.

I spent two weeks fixing that. Here is everything I learned, including the dependency conflict that had no solution and the DLL error that forced me to build a better architecture than I originally planned.

First, what actually is RAG? (skip if you know)

If you are new to this, here is the whole idea in one paragraph.

An LLM does not know about your PDF. So before you ask it anything, you cut your PDF into chunks, convert each chunk into a list of numbers that captures its meaning (an "embedding"), and store those numbers in a database. When a question comes in, you convert the question into numbers too, find the chunks whose numbers are closest, and paste those chunks into the prompt. The model answers from text you handed it rather than from memory. That is Retrieval-Augmented Generation.

Standard RAG is three steps:

embed the question  ->  find the 5 closest chunks  ->  generate

It works well enough in a demo to make you think you are done. You are not done. Those three steps have at least four serious weaknesses, and I want to walk through each one, because the fixes are the interesting part.

Weakness 1: your chunks don't know where they came from

Here is a real chunk from a paper in my system:

"The reduction was 6.9 mmHg (95% CI: 5.2-8.6, p<0.001)."

Reduction in what? Measured in whom? After which intervention? You cannot tell, and neither can the embedding model. That chunk gets converted into a vector that means roughly "a number went down a bit," which is useless. It will never be retrieved for the question it actually answers.

The fix, from Anthropic's contextual retrieval research: before embedding a chunk, ask an LLM to write one sentence explaining where the chunk sits in its document, and glue that sentence to the front.

The same chunk becomes:

"From a meta-analysis of aerobic exercise trials, section on blood pressure outcomes in hypertensive adults. The reduction was 6.9 mmHg (95% CI: 5.2-8.6, p<0.001)."

Now the embedding actually means something.

_CONTEXT_PROMPT = """<document>
{doc_preview}
</document>

Here is a chunk taken from that document:
<chunk>
{chunk}
</chunk>

Write ONE short sentence (max 25 words) that situates this chunk within the
document, so the chunk can be understood and retrieved on its own."""

It costs one small LLM call per chunk at upload time, and it runs in a thread pool so a 14-chunk paper takes about 6 seconds. Anthropic measured a 49% reduction in retrieval failures from this technique alone.

One detail that matters: I store the original text separately in node.metadata["raw_text"]. The context sentence helps the machine find the chunk. The user should still read what the paper actually said.

Weakness 2: vector search is bad at exact terms

Ask a vector database about "eNOS phosphorylation at Ser1177" and it will cheerfully return a passage about nitric oxide signalling that never mentions Ser1177, because in embedding space those two things sit right next to each other.

Embeddings capture meaning, and meaning is fuzzy by design. That is usually a feature. In medicine it is a liability, because medicine runs on exact strings: drug names, gene symbols, ICD codes, dosages, p-values. Those are precisely the tokens embeddings blur together.

The fix: run a keyword search alongside the vector search. BM25 is the classic algorithm here and it finds literal tokens instantly.

_TOKEN_RE = re.compile(r"[a-z0-9]+(?:[-_][a-z0-9]+)*")

def tokenize(text: str) -> list:
    return [t for t in _TOKEN_RE.findall(text.lower())
            if len(t) > 2 and t not in _STOPWORDS]

That regex is deliberately built to keep ser1177, covid-19, and il-6 intact instead of shredding them into meaningless fragments.

I used rank_bm25, which runs in-process and persists to a pickle file. No Elasticsearch cluster, no Docker, no second service to keep alive. For a single-instance app that is exactly the right amount of infrastructure.

Weakness 3: you cannot just add the two scores together

So now you have two ranked lists. Merging them is where most people quietly break their own system.

The obvious move is to normalise both scores to 0-1 and add them. Do not do this. Cosine similarity is bounded and roughly normal. BM25 is unbounded and heavily skewed by document length. Min-max normalising them means a single outlier BM25 score squashes everything else into a narrow band, and your ranking silently becomes noise.

The fix: throw the scores away and keep only the ranks.

def reciprocal_rank_fusion(ranked_lists, k=60):
    scores = {}
    for results in ranked_lists:
        for rank, item in enumerate(results, start=1):
            nid = item["node_id"]
            scores[nid] = scores.get(nid, 0.0) + 1.0 / (k + rank)
    ...

That is the entire algorithm. Reciprocal Rank Fusion, from a 2009 SIGIR paper. The constant 60 is a damper: rank 1 contributes 1/61 and rank 2 contributes 1/62, so no single list can dominate and a document has to place well in several lists to win.

Elasticsearch, Weaviate, and Azure AI Search all use this. It beats carefully tuned score normalisation in most published comparisons. Six lines.

There is a free bonus here too. If a chunk appears in both the vector results and the keyword results, that is a strong signal, so I track it and surface it in the UI as a "both" badge:

item["retriever_agreement"] = len(ranks[nid])

Weakness 4: your retriever was never trained to compare

This one is architectural and it is the biggest win available.

Your vector database uses a bi-encoder. It embedded the chunk at upload time, long before anyone asked anything, so that chunk's vector is a compressed summary produced with zero knowledge of the future question. Then it embeds your question separately and compares the two vectors. Fast, because everything is precomputed. Lossy, for the same reason.

A cross-encoder reads the question and the chunk together, in one pass, and outputs a single relevance number. It can see that "150 minutes" in the passage answers "how many minutes" in the question. Nothing was compressed away in advance.

The catch is that nothing can be precomputed, so it is one model call per candidate. You never run it over your whole corpus. The pattern is always:

retrieve 30 candidates cheaply  ->  rerank those 30 precisely  ->  keep 5

In my UI, any source that got promoted by the reranker shows a green +4 badge. Expand it and you see rank 7 to 2. That is the reranker catching something vector search ranked seventh and putting it where it belongs.

Weakness 5: the one that actually scared me

Everything above makes retrieval better. None of it helps when the answer simply is not in your documents.

CRAG (Corrective RAG) adds a grader between retrieval and generation, and crucially, gives the pipeline somewhere to go when the grade is bad:

if max_score >= 0.70:
    action = "correct"      # generate now
elif max_score >= 0.35:
    action = "ambiguous"    # rewrite the query, retrieve again, merge
else:
    action = "incorrect"    # do not generate at all

That last branch is the whole reason I rebuilt this thing.

if action == "incorrect" or not documents:
    with trace.span("7_generation") as span:
        span.attributes["skipped"] = True
        span.attributes["reason"] = "crag_graded_retrieval_as_incorrect"

    for word in NO_ANSWER_RESPONSE.split(" "):
        yield word + " ", None

The generator never runs. There is no answer to generate, so none gets generated.

One design note I went back and forth on: I grade on the best document, not the average. One excellent source is enough to answer well, and averaging punishes a good retrieval that also happened to pull in a couple of loosely related chunks.

Ask my cardiology paper about amoxicillin dosing now and it tells you, plainly, that the answer is not in the indexed documents. That is not a limitation. That is the feature.

Bonus: HyDE, the technique that sounds like a bug

There is a quiet asymmetry in dense retrieval. You embed a question and search against embeddings of answers. Look at what those look like:

question: "How much does exercise lower blood pressure?"
passage : "Systolic blood pressure decreased by a mean of 6.9 mmHg
           across 89 trials."

Almost no shared vocabulary. Different grammar, different register. Their embeddings sit further apart than you would like, and the retriever is being asked to bridge a gap nobody trained it to bridge.

HyDE closes it by moving the query into answer-space first. Ask the LLM to write a plausible answer, ignore whether it is true, and embed that.

Yes, you are deliberately generating a hallucination and searching with it. It works because the fake passage is shaped like the real passage, so it lands near it in vector space and drags the true source out of the index.

The hallucination never reaches the user. It is discarded the moment retrieval finishes. Its only job is to have the right shape.

Now the part where everything broke

Six clean layers. Beautiful architecture. Then I ran pip install -r requirements.txt.

Act 1: the DLL that would not load

ImportError: DLL load failed while importing onnxruntime_pybind11_state

ChromaDB imports onnxruntime at module load for a default embedding function I never use, because my embeddings come from Gemini. Did not matter. The import has to succeed for ChromaDB to import at all, and the whole app died at line one.

Pinning onnxruntime<1.20 fixed it. Which immediately caused:

Act 2: the fix that broke something else

opentelemetry-proto 1.42.1 requires protobuf<7.0,
but you have protobuf 7.35.1 which is incompatible.

Installing onnxruntime had pulled the newest protobuf, which broke ChromaDB's telemetry instrumentation. Pin protobuf. Move on.

Act 3: the conflict with no solution

This is the one I want to talk about.

llama-index 0.14.23 requires llama-index-llms-openai>=0.7.0
llama-index-llms-openai-like 0.5.3 requires llama-index-llms-openai<0.6

Read those two lines again. Package A needs version 7 or higher. Package B needs below version 6. Both are in my tree because llama-index-llms-groq depends on openai-like.

There is no version of anything that satisfies both. I watched pip ping-pong between them four times, each install "fixing" the conflict by recreating the other half of it.

I nearly downgraded the whole framework. Then I actually looked at what that package was doing for me.

One thing. One single call:

Settings.llm.stream_complete(prompt)

That was it. An entire branch of dependency hell to wrap an HTTP request I could make myself. And I was already using the official Groq SDK directly in four other files: guardrails, HyDE, CRAG, and contextual chunking.

So I deleted the dependency:

stream = _get_llm().chat.completions.create(
    model=_LLM_MODEL,
    messages=[{"role": "user", "content": prompt}],
    temperature=0.1,
    stream=True,
)
for chunk in stream:
    token = chunk.choices[0].delta.content or ""
    if token:
        full_answer += token
        yield token, None

Conflict gone. Permanently. Fewer packages, less abstraction, more control over the stream.

The lesson I actually took from this: when a dependency conflict has no solution, stop solving it and start asking what the dependency was buying you. Sometimes it is a framework. Sometimes it is a wrapper around six lines you can write yourself, and you have been renting a dependency tree to avoid typing them.

Act 4: the error that improved the architecture

Last one, and it is my favourite, because the error made the system genuinely better.

[reranker] local model unavailable
([WinError 1114] A dynamic link library (DLL) initialization
 routine failed. Error loading torch\lib\c10.dll)

PyTorch would not load on Windows. Which meant my cross-encoder would not load. Which meant the single highest-impact stage in the entire pipeline was dead, on a machine where nothing was actually wrong with my code.

I could have fought the DLL. Instead I asked a better question: why is my most important stage sitting behind a 2GB native dependency with a known platform failure mode?

So I added a third tier. The reranker now picks the best backend available at boot:

Cohere rerank-v3.5 if an API key is set. Best accuracy money can buy.
Local BGE cross-encoder via sentence-transformers. Free, offline, fast.
Groq LLM-as-reranker. Pure HTTP. No torch, no model download, no new key.

Tier 3 scores each candidate with a small parallel LLM call:

with ThreadPoolExecutor(max_workers=8) as pool:
    scores = list(pool.map(
        lambda c: _groq_score_one(query, c.get("text", "")),
        candidates,
    ))

Latency here is entirely network-bound, so 20 sequential calls would add seconds while 20 concurrent calls cost roughly one. It is less precise than a real cross-encoder and burns more API quota, but it works on any machine with an internet connection.

And if all three fail, the pipeline falls back to RRF order rather than crashing, and the trace records which backend actually ran.

That is the real lesson of the whole project. A DLL error on one laptop should degrade one stage, not delete a feature. Enterprise systems are not defined by having the best possible component. They are defined by what happens when the best possible component is unavailable.

You cannot improve what you cannot see

Every stage is wrapped in a span:

with trace.span("5_rerank") as span:
    documents, backend = rerank(question, candidates, top_n=5)
    span.attributes["backend"] = get_reranker_name()
    span.attributes["promoted"] = len(promoted)
    span.attributes["biggest_jump"] = max(...)

Which produces a per-query timeline showing exactly where the milliseconds went:

Stage	Typical
HyDE	400 to 800ms
Vector search	200 to 400ms
BM25	under 10ms
RRF fusion	under 5ms
Rerank	150 to 400ms
CRAG grading	300 to 600ms
Generation	1 to 3s

No LangSmith account, no Phoenix instance. A ring buffer, a JSONL file, and a dashboard page. About 200 lines.

Two things it immediately taught me that I would never have guessed:

BM25 is essentially free. Under 10 milliseconds against 14 chunks. I had been mentally budgeting it as a real cost. It is a rounding error, and it is catching exact-term matches that vector search misses entirely.

HyDE is my second most expensive stage. It costs more than reranking. That is a genuine trade-off I now get to make deliberately, with a number in front of me, instead of guessing.

The thing I am proudest of

I put the pipeline toggles in the product.

There is a panel above the chat input with four switches: HyDE, hybrid search, reranking, CRAG. Turn reranking off, ask the same question again, and watch the source ordering degrade in real time.

That is an ablation study. In the UI. Running on the user's own documents.

class PipelineSettings(BaseModel):
    use_hyde: bool = True
    use_hybrid: bool = True
    use_rerank: bool = True
    use_crag: bool = True

Four booleans on the request body. Anyone can now verify my claims about this architecture on their own data instead of taking my word for it, and I can debug a bad answer by bisecting the pipeline live rather than editing code and restarting.

Where it ended up

1. HyDE              write a hypothetical answer, embed that instead
2. Dense retrieval   vector search, top 20
3. Sparse retrieval  BM25 keyword search, top 20
4. RRF fusion        merge both lists by rank, not score
5. Cross-encoder     rerank the top 30, keep the best 5
6. CRAG grading      score relevance, rewrite and retry, or refuse
7. Generation        stream the answer
8. Guardrails        confidence score from lexical overlap plus LLM check

From three steps to eight. Roughly 3 to 5 seconds per query instead of 1 to 2.

Worth every millisecond, because the system can now do the one thing it could not do before: notice that it does not know.

If you take one thing from this post

Not the six techniques. Those are Googleable and half of them will have better versions in a year.

Take this: your RAG system's most dangerous state is not being wrong. It is being wrong with no mechanism to notice.

Retrieval quality, reranking, hybrid search, all of it exists to reduce how often you land in that state. But reducing it is not the same as detecting it. Until you build something that can grade its own retrieval and refuse, you have not built a system that can fail safely. You have built one that fails silently, which in a medical context is considerably worse.

Try it

Everything here is open source and runs locally.

github.com/Sowaiba-01/ClinicaQuery-AI

Stack: Next.js 15, FastAPI, ChromaDB, rank-bm25, Groq (Llama 3.3 70B), Gemini embeddings.

Papers worth reading if this interested you:

Contextual Retrieval (Anthropic, 2024)
Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE, Gao et al. 2022)
Corrective Retrieval Augmented Generation (Yan et al. 2024)
Reciprocal Rank Fusion (Cormack et al., SIGIR 2009)

If this was useful

⭐ Star the repo so I know it landed
Follow me for the next one. I am working on GraphRAG and multi-document reasoning
Tell me in the comments what your RAG system does when it does not know the answer. I genuinely want to know how other people solved this.

And if you have ever shipped a RAG app that hallucinated confidently at a real user, say so. I would like this comment section to be the place people admit that.

"I built a deepfake detector. Then I built the tool that proves it actually works."

Sowaiba Arshad — Mon, 27 Jul 2026 14:36:58 +0000

There's a moment every ML engineer knows and nobody really talks about.

You finish training. The validation accuracy looks great. You run model.predict(), it works, and you ship it.

Then the quiet questions start.

Is it still accurate, or is it slowly getting worse? Has the incoming data drifted away from what it trained on? When the model says it's "92% confident," is it actually right 92% of the time, or is it just loud? Is the input data even valid, or is some silent schema change quietly poisoning every prediction? And the one that actually keeps you up at night: how would you even know if something broke?

Building a model is mostly a solved problem now. Knowing whether you can trust it once it's in production is not. That gap is the thing I wanted to close.

Meet ModelSentinel

ModelSentinel is an open-source Python toolkit for everything that happens after model.predict(). It evaluates, monitors, and stress-tests ML models through one consistent API.

import modelsentinel as ms

monitor = ms.Monitor(task="classification", name="my-model")

monitor.evaluate(y_true, y_pred, y_score)       # accuracy, F1, ROC-AUC, confusion matrix
monitor.calibration(y_true, y_score)            # are the probabilities trustworthy?
monitor.detect_drift(reference_df, live_df)     # has the data shifted since training?
monitor.profile_data(live_df)                   # missing values, dupes, outliers, schema

print(monitor.health_score())
# {'overall': 99.2, 'grade': 'EXCELLENT', 'components': {...}}

monitor.generate_report("report.html")

Everything rolls up into a single Model Health Score, one number you can set an alert on instead of squinting at five dashboards.

I didn't want to demo it on toy data, so I audited my own model

Talk is cheap, so I pointed ModelSentinel at a real system I'd already built: DeepGuard, an EfficientNet-B4 deepfake detector. Not a toy example. The actual PyTorch model, the actual weights, the actual face images.

And I didn't just run it once. I tested it against two different datasets built with two different face-generation methods: a held-out split of the 140k Real and Fake Faces set (StyleGAN style), and a separate set I generated myself with inswapper_128 face-swapping. If the detector had only memorized one kind of fake, this would have caught it.

Here's what ModelSentinel reported, straight from the model:

Metric	140k test split (held-out)	inswapper_128 set
Accuracy	0.9975	0.9938
F1	0.9975	0.9938
ROC-AUC	0.99999	0.9989
Brier / ECE	0.0028 / 0.0075	0.0086 / 0.0147
Health Score	99.65 (EXCELLENT)	99.23 (EXCELLENT)

Confusion matrices (rows = true [real, fake]): [[398, 2], [0, 400]] and [[396, 4], [1, 399]].

Two things jumped out that a plain accuracy number would never have told me. First, the detector stays above 99% on both generation methods, so it's generalizing across techniques rather than just fitting its training distribution. Second, the very low ECE (0.0075) means the confidence scores are actually calibrated. When DeepGuard says 95%, you can more or less believe it. That's a claim you can only make after you measure calibration, which most projects just never do.

I'll be honest about one thing. One of those datasets is the same domain the model trained on, so treat 99% as an upper bound rather than a promise about the messy real world. The point is that ModelSentinel gives you the numbers to reason about it at all.

What's actually under the hood

No magic, and nothing that pretends "AI is monitoring your AI." It's just solid, well-understood statistics wired together cleanly.

Evaluation: classification and regression metrics, plus probability calibration (Brier, ECE, MCE) and decision-threshold tuning (F1 and Youden's J).
Drift detection: Kolmogorov-Smirnov and Population Stability Index for numeric features, chi-square and Jensen-Shannon divergence for categorical ones, all aggregated into a single dataset-level verdict.
Data quality: missing values, duplicates, constant columns, IQR outliers, and schema validation between training and production.
Health score: a weighted roll-up that renormalizes gracefully when one of the checks is missing.

The metrics match a hand computation to six decimal places, because underneath it leans on scikit-learn and scipy, the boring and battle-tested stuff. ModelSentinel's job is just to make all of it effortless to use.

Built like a library, not a notebook

This is the part I care about most. A reliability tool that isn't reliable itself is a bit of a joke, so I built ModelSentinel like a real open-source project. It's fully typed and documented, has a pytest suite, passes ruff linting cleanly, runs CI across Python 3.9 to 3.12, ships as a buildable wheel, includes real benchmarks with timings, and keeps a semantic-versioned changelog. You install it with pip install -e ".[dev]" and the tests go green on a clean machine.

Where it's going

ModelSentinel ships new versions regularly, and the roadmap is deliberately ambitious.

v0.4: explainability (Grad-CAM and SHAP) and framework adapters for scikit-learn, PyTorch, and TensorFlow.
v0.5: a FastAPI monitoring server with real-time drift.
v0.6 and beyond: LLM and RAG evaluation, including hallucination, faithfulness, and toxicity checks.

The goal is one toolkit that can answer "can I trust this model?" whether the model is a classifier, a vision network, or an LLM.

Try it, break it, build on it

ModelSentinel is MIT-licensed and open to contributions. If you've ever shipped a model and felt that little flicker of doubt about whether it's still working, this one is for you.

Star the repo, fork it, and tell me which reliability check you'd want next. New versions are landing regularly, and honestly the best ideas tend to come from people who have been burned by a silent model failure.

Built by Sowaiba Arshad. Feedback and PRs genuinely welcome.

Sowaiba-01 / ModelSentinel

Open-source Python toolkit that checks whether a trained ML model is still healthy: metrics, data drift, calibration, data quality, and a weighted health score.

ModelSentinel

AI reliability & observability toolkit — monitor, evaluate, explain, and protect machine-learning models with a single, consistent Python API.

Shipping a model is easy. Knowing whether it is still trustworthy in production is not. ModelSentinel answers the questions that come after model.predict():

Is my model still accurate, or is performance quietly degrading?
Has the incoming data distribution drifted away from training?
Is the input data even valid — missing values, duplicates, schema changes?
Are my predicted probabilities calibrated, or overconfident?
What single number tells me if this model is healthy right now?

Install

pip install modelsentinel          # from PyPI (once published)
pip install -e ".[dev]"            # from source, with dev tools

30-second quick start

import modelsentinel as ms
monitor = ms.Monitor(task="classification", name="DeepGuard-B4")

# 1. How good are the predictions?
monitor.evaluate(y_true, y_pred

…

View on GitHub

I Opened a GitHub Issue and Four AI Agents Fixed It Without Me Touching a Single Line of Code

Sowaiba Arshad — Sun, 26 Jul 2026 07:30:52 +0000

I want to be honest with you. When I started this project I did not actually believe it would work.

Not in a humble way. I mean I genuinely thought the idea was too ambitious for one person to pull off with free API tiers and a laptop. Fully autonomous AI that reads a GitHub issue,understands a codebase, writes working code, runs tests, gets the code reviewed by another AI, and opens a pull request. No human in the loop. No clicking approve at any step.

But it works. And the first time it actually completed a full run I just sat there staring at the pull request on GitHub for a solid minute.

The moment that started it

I was reviewing a bug ticket at 1am. It was a simple fix. Like embarrassingly simple. Change a function name, update two tests. Fifteen minutes of actual work, forty minutes of context switching, finding the right file, remembering where the tests live, writing the PR description.

And I thought, what if I just did not have to do that.

Not "what if AI helped me type faster." What if I described the problem in plain English and walked away.

What I built

Four agents. Each one with a specific job.

The Architect reads the repo, opens the most relevant files, understands the codebase structure, and produces a concrete implementation plan. It posts that plan as a comment on your GitHub issue so you can see its thinking.

The Coder spins up a fresh cloud sandbox, clones your repo, writes the code following the architect's plan, runs the linter, commits, and runs the test suite. If tests fail it reads the error, adjusts, and tries again. It gets three attempts.

The Reviewer reads the git diff line by line. Checks for hardcoded secrets, SQL injection risk, missing input validation, bad error handling. Returns APPROVED or sends it back with a specific reason.

The PR Agent opens a draft pull request with a full description of what changed, why it changed, and what the test output was.

The whole thing runs in about four minutes on a simple feature request.

What makes this different from just calling GPT

A lot of "autonomous AI" demos are one agent calling a few tools in a loop until something works. This is different in a way that matters.

The agents have completely separate roles, separate tools, and they cannot see each other's internal reasoning. The Architect does not know what the Coder is thinking. The Reviewer does not know what the Architect planned. Each one gets only what it needs.

The supervisor that routes between them is pure Python if/else logic. No LLM deciding what to do next. That was a deliberate choice. I wanted the control flow to be predictable, not something that could hallucinate its way into an infinite loop.

The code actually runs. This is the part people underestimate. It is not simulated. The Coder has a real Ubuntu sandbox with real git, real compilers, real test runners. When it runs tests, those are actual test results. When a test fails, it is a real failure. The agent reads the real error output and has to actually fix it.

The stack I used

All of this runs on free tiers.

Groq gives you 500,000 tokens per day on Llama 3.1. That is enough to run the swarm several times a day without paying anything. E2B gives you cloud sandboxes that spin up in seconds and run arbitrary code safely. LangGraph handles the agent graph. FastAPI serves the backend. Next.js streams live agent logs to the browser over WebSocket so you can watch every thought the agents have in real time.

The whole thing deploys with one command.

The part nobody talks about

Building autonomous agents sounds clean in blog posts. The reality is that AI models at this scale do genuinely weird things.

The model tried to call a tool that did not exist. Eleven times in a row. Because I told it to in the system prompt and forgot to actually add the tool.

It hit a 13,000 token request on a 12,000 token limit because the message history grew with every tool result appended.

The first git clone always failed because the cloud sandbox was still booting when the agent tried to use it.

The test runner looped forever on a Java project because there was no Java branch in the detection logic and the model kept retrying pytest on a codebase with zero Python files.

These are not theoretical edge cases. These all happened on the first real test run. Fixing them one by one until the swarm ran clean is what actually took the most time.

What it feels like when it works

I created a GitHub issue on my old JavaFX Brick Breaker game asking for a high score tracker. I typed the description, hit submit, and watched the logs in the browser.

The Architect opened BrickBreak.java, read 800 lines of game code, understood how scoring worked, and posted a plan to the GitHub issue. The Coder created ScoreManager.java from scratch, wired it into the game, compiled both files, verified they linked correctly. The Reviewer checked the diff and approved. A pull request appeared on GitHub with a description that explained exactly what was done and why.

I did not write a single line of that.

There is something genuinely strange about watching a machine understand your code and extend it correctly. It does not feel like autocomplete. It feels like something crossed a line.

What this actually is

This is a personal project. Not a product, not a startup, not something with an uptime SLA. I built it to see if I could, and to understand how multi-agent systems actually work under the hood rather than just reading about them.

It is not production ready. The agents still make mistakes on complex codebases. The Architect sometimes misreads project structure. The Coder occasionally writes code that compiles but misses the intent. That is fine. The point was never a perfect system.

The point was to build something real, hit the actual problems, and solve them one by one until it worked. And it works.

I learned more about LLM context management, token budgets, deterministic vs LLM-driven routing, and sandbox execution environments from this one project than from anything else I have built. That is the real value.

It is open source

If you want to dig into the code, run it locally, or steal ideas for your own agent project:

github.com/Sowaiba-01/devops-swarm

You need three free API keys: Groq, GitHub, and E2B. Docker Compose handles the rest. The live log stream in the browser is genuinely fun to watch.

Why My Medical AI Took 6.4 Seconds Per Scan and How I Got It to 3.1.

Sowaiba Arshad — Sat, 25 Jul 2026 08:31:16 +0000

I built a chest X-ray diagnostic platform called ThoraxNet. It detects 14 thoracic pathologies from a single image, reports how confident it is using Monte Carlo Dropout, draws **GradCAM **heatmaps over the regions that drove each prediction, and writes a structured radiology report with an LLM.

The model was the interesting part to build. It was not the part that decided whether the thing felt like a product. That came down to one unglamorous number: how long a user waits after they hit "analyze."

This post is about how I took that wait from 6.4 seconds to 3.1 seconds, why the honest answer is "about 2x and not 10x," and the two bugs I only found because I stopped trusting the happy path and read the logs.

Live demo: https://thorax-tho.vercel.app
Code: https://github.com/Sowaiba-01/ThoraxNet

The problem: a good model wrapped in a slow request

The model worked. The demo worked. But every scan took roughly six and a half seconds, and a six second spinner makes anything feel broken no matter how good the output underneath it is.

My first instinct was the same wrong instinct most people have: "the vision transformer forward pass must be the bottleneck, so let me go optimize the model." I have shipped enough regressions chasing that kind of hunch to know it is worth nothing until it is measured.

So before changing a single line of model code, I instrumented the pipeline to report how long each stage actually took, and returned that breakdown on every single response:

"stage_timings_ms": {
  "preprocess": 16,
  "mc_dropout": 2868,
  "gradcam": 6,
  "report": 92
}

The result was not what I expected, and that is exactly why measuring first matters. A single forward pass through the transformer is about 15 milliseconds. The model was never the problem. Monte Carlo Dropout was, and it was eating nearly all of the server side budget.

The root cause: twenty forward passes, one at a time

Monte Carlo Dropout estimates uncertainty by running the model many times with dropout left switched on, then measuring how much the predictions move. My original implementation did the obvious thing:

samples = []
for _ in range(n_samples):        # n_samples = 20
    logits = model(x)             # one image, batch size 1
    samples.append(torch.sigmoid(logits))

Twenty sequential forward passes, each with a batch size of one. Every pass pays the full per call overhead, and the hardware spends most of its time waiting between launches instead of computing.

The fix is to stop asking twenty separate times and ask once. Tile the single image into a batch of twenty and run one forward pass:

tiled = x.repeat(n_samples, 1, 1, 1)   # (20, 3, 224, 224)
logits = model(tiled)                  # ONE forward pass
probs = torch.sigmoid(logits).view(n_samples, batch, -1)
mean, std = probs.mean(0), probs.std(0)

The correctness argument is the part worth understanding, and it is the exact question a good interviewer will ask you. Are those twenty copies not identical? No. Dropout samples a fresh mask for every element in the batch. So the twenty tiled copies each get an independent dropout mask, which means they are exactly the twenty independent stochastic samples the estimator needs. Same statistics, one launch instead of twenty.

I did not want to trust that reasoning on faith, so I wrote a test that runs both the old sequential version and the new batched version four hundred times each and asserts their means agree within Monte Carlo error:

assert torch.allclose(batched_mean, sequential_mean, atol=0.05)

That test is the difference between "I think this is equivalent" and "I proved this is equivalent."

The results, measured on identical hardware

Same host, same image, same protocol of thirty requests with warmup discarded. The only variable is the code.

Metric	Before (v1.0.0)	After (v1.1.0)	Improvement
p50 latency	6,387 ms	3,127 ms	1.9x faster
p95 latency	7,525 ms	3,862 ms	1.9x faster
p99 latency	8,026 ms	4,768 ms	1.7x faster
Throughput	0.15 req/s	0.30 req/s	2.0x

Every percentile improved by roughly 2x, with no change to the model weights and no change to accuracy. This is purely a change in how the model is executed.

Two smaller changes cleaned up the rest of the request path. The LLM report call was moved off the critical path onto a worker thread, so the user gets their pathology result without waiting on report generation. And GradCAM heatmaps, which were being recomputed on every request, are now cached per image and class.

The honest part: why it is 2x and not 10x

Here is where it would be easy to lie, and where lying would eventually cost me an offer.

On a GPU, this exact change is often close to 10x. The entire win comes from keeping an accelerator busy that was otherwise sitting idle between twenty tiny kernel launches.

ThoraxNet runs on a free tier CPU host with two virtual cores. There is far less idle parallelism to reclaim there. So the same code change gives roughly 2x, not roughly 10x.

I could have written "10x faster" in the title and most readers would never have checked. But the number that survives a technical interview is the one you can explain. The honest version is: 2x on CPU, because the batching win is bounded by how much idle parallelism there is to reclaim, and Monte Carlo Dropout on CPU is still the dominant cost, so the next real lever is GPU inference or quantization, not more batching. That sentence is worth more than a bigger number I cannot defend.

Two bugs I found by reading the whole path, not the ticket

Profiling forced me to read the entire request path from HTTP call to response. Two problems fell out that had nothing to do with latency.

GradCAM had never worked. The route handler read an attribute on the pipeline that was never actually assigned anywhere. The inference code built the heatmaps into a local variable and then dropped them when the function returned. Every heatmap request came back as a 404. Nothing logged an error, and the frontend simply rendered an empty panel, so it shipped and sat broken because no code path ever raised. I fixed it and pinned it with a regression test that fails if the overlays are not recorded, because a silent bug earns a loud test.

Every radiology report was quietly failing. While watching the deploy logs I saw this on repeat:

The model `llama3-70b-8192` has been decommissioned and is no longer supported.

The LLM provider had retired that model months earlier. Every report request was returning a 400 and falling back to a template, so users were getting canned text instead of a generated report, and nobody noticed because the fallback made it look fine. The fix was one line to point at the current model, plus an environment variable so the next deprecation is a config change and not a code change.

Neither of these was in my task list. Both were only visible because I stopped trusting the happy path and actually read what the server was doing.

What I would do next

Monte Carlo Dropout still dominates at roughly 2.8 seconds. Batching took the easy win. The remaining levers are honest about the hardware rather than clever in the request path:

GPU or quantized inference to cut the per request cost of the twenty passes. The export and evaluation scripts are already written, and the accuracy delta table is the homework I still owe.

A shared cache for the GradCAM session store, since it currently lives in process memory that will not survive a restart or a second replica.

The takeaway

The model was 15 milliseconds. The product was 6.4 seconds. The entire gap lived in how the model was called, not in the model itself, and I only found that because I measured before I touched anything and read the logs instead of the ticket.

The unglamorous work is the work. Profiling first, proving equivalence with a test, and being honest about a 2x instead of inflating it to a 10x are the habits that hold up when someone actually reads your code.

If you found this useful, follow me on GitHub for more write ups like this: https://github.com/Sowaiba-01

The full project is open source. Fork it, build on it, and if it helped you or you just think it is cool, a star means a lot: https://github.com/Sowaiba-01/ThoraxNet

ThoraxNet is for research use only. It is not FDA cleared and is not a substitute for a radiologist.