Two Bugs That Almost Shipped in My Agentic RAG Assistant

#rag #langgraph #llm #python

I built an agentic RAG assistant — upload a PDF, ask questions, get streamed answers grounded strictly in the document. It uses LangGraph for tool orchestration, Groq for inference, Qdrant Cloud for retrieval, and Streamlit for the UI, and it's self-contained: no separate backend service to keep alive for the live demo. If the answer isn't in the document, it says so instead of guessing.

It worked. Mostly. Every so often, a stray 【source】 tag — an internal citation marker never meant for the user — leaked straight into the answer on screen. Not always. Not predictably. That inconsistency turned out to be the whole puzzle, and chasing it down taught me more about streaming systems than the rest of the project combined.

That was the first of two bugs that surfaced during demo prep — the second one had nothing to do with streaming and everything to do with where I was storing state. Fixing it changed the deployment architecture entirely, which is why the finished version looks different from what I originally built.

(Screenshots below use a fictional test document — not a real product or person.)

Watching it decide — the agent retrieves, streams the answer token by token, and stays grounded in the uploaded document throughout.

What the project does

Ask a question, and the agent:

Retrieves relevant chunks from a Qdrant Cloud vector store built from your uploaded PDFs
Answers only from retrieved content — if nothing relevant is found, it says so instead of guessing
Routes queries between llama-3.1-8b-instant and openai/gpt-oss-120b based on complexity, so short factual questions don't pay large-model latency or cost

A LoRA fine-tuned resume screener also exists as a separate service, callable via screen_resume — it's not wired into the live demo's default flow, so I'm calling that out here rather than letting someone discover the gap themselves.

Stack: LangGraph (ReAct-style agent) · Groq running gpt-oss-120b (primary) and llama-3.1-8b-instant (routed, low-complexity queries) · Qdrant Cloud for the vector store (originally local Chroma — more on that below) · HuggingFace bge-small-en-v1.5 embeddings · Streamlit frontend, fully self-contained.

The assistant answering only from the uploaded document — quoting specifics like latency numbers and tech stack directly from the source, and correctly refusing to answer when the document has no relevant info ("What is Arjun's favorite hobby?").

Architecture — and where each bug lived

A user question goes into the Streamlit app, the LangGraph agent either retrieves from Qdrant Cloud or routes to a small or large Groq model depending on complexity, and the answer streams back. Bug #2 lived at the storage step — it's why Qdrant Cloud replaced local Chroma entirely. Bug #1 lived at the streaming step, one layer downstream.

Bug #1: a marker that only sometimes leaked

The agent tags its retrieval citations internally with a marker like 【source】, which is supposed to get stripped before the answer reaches the user. Simple enough — clean each chunk of streamed text with a regex as it comes in.

Except sometimes it didn't get stripped.

The cause: streaming breaks text into arbitrary token chunks, and the marker was getting split across two of them. One chunk would end with 【sour, the next would start with ce】. Since I was running the regex on each chunk in isolation, neither half ever matched the full pattern — both leaked straight through, untouched.

The fix wasn't a smarter regex. It was buffering: hold back any text that could be the start of an incomplete marker, and only release it once you're sure you're not sitting mid-tag.

_CITATION_ARTIFACT_RE = re.compile(r"【[^】]*】")

def stream_agent(question: str):
    """Yield the answer as it streams in, with citation markers
    stripped even when a marker is split across chunk boundaries."""
    buffer = ""
    for chunk in agent.stream(question):
        buffer += chunk

        while True:
            start = buffer.find("【")
            if start == -1:
                if buffer:
                    yield buffer
                    buffer = ""
                break

            if start > 0:
                yield buffer[:start]
                buffer = buffer[start:]

            end = buffer.find("】")
            if end == -1:
                break

            buffer = buffer[end + 1:]

Streaming still feels instant to the user. Nothing incomplete ever reaches them — the buffering adds negligible overhead, invisible in normal use.

The broader lesson: streaming systems fail in ways batch systems simply can't. A batch-mode version of this same assistant would never have hit this bug — it processes the full response as one string before cleaning it. This bug only existed because I was processing partial state in real time, and for a doc-QA tool, a leaked internal marker is exactly the kind of thing that quietly erodes trust in the answer itself, even when the answer underneath it is correct.

Bug #2: it felt broken because it was, structurally

Bug #1 was a parsing problem. Bug #2 turned out to be an architecture problem wearing a parsing-problem costume.

Once the streaming issue was fixed, a different problem showed up during demo prep: every time the backend went idle and spun back down, the app "forgot" everything. Users had to re-upload the same PDF and re-ingest it before asking anything, even though nothing about the workflow suggested that should be necessary.

The cause here wasn't a code bug — it was an architecture mismatch. At the time, the backend was deployed on Render's free tier, which uses an ephemeral filesystem: anything written to local disk, including the Chroma vector store, gets wiped on every restart or spin-down. Free-tier services spin down automatically after 15 minutes of inactivity, which is basically guaranteed for a portfolio demo that isn't getting constant traffic.

One giveaway made this obvious in hindsight: a response that came back in 0.86 seconds. That's far too fast for a real embed → retrieve → LLM round trip. It wasn't a fast correct answer — it was an empty vector store returning nothing, instantly.

The fix was to stop treating the vector store as something that lives on a server's local disk at all, and move it to a hosted vector database that persists independently of any backend's lifecycle:

# src/vectorstore.py — shared, persistent vector store client
import os
from langchain_qdrant import QdrantVectorStore
from langchain_huggingface import HuggingFaceEmbeddings
from qdrant_client import QdrantClient

embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")

client = QdrantClient(
    url=os.getenv("QDRANT_URL"),
    api_key=os.getenv("QDRANT_API_KEY"),
)

def get_vectorstore(collection_name: str = "documents"):
    return QdrantVectorStore(
        client=client,
        collection_name=collection_name,
        embedding=embeddings,
    )

Ingestion writes to Qdrant instead of a local Chroma folder, and retrieval reads from the same remote collection — so a restart no longer wipes anything out. Once retrieval no longer depended on local state surviving a restart, the "upload every time" problem disappeared along with it.

This fix ended up reshaping the whole deployment. With persistence handled by Qdrant Cloud, the Streamlit frontend no longer needed a standalone backend to talk to at all — it calls Groq and Qdrant directly. The FastAPI backend (src/api.py) still exists in the repo for local development and HTTP access to the agent, but the live demo doesn't depend on it being up. One bug fix turned into a simpler, more resilient architecture than the one I started with.

One more thing that broke after the fact: a model deprecation

Not a bug in my code, but worth logging alongside the other two: this project originally ran llama-3.3-70b-versatile. Groq deprecated that model on 2026-06-17, with a shutdown date of 2026-08-16. The agent is now migrated to openai/gpt-oss-120b — Groq's recommended replacement, with full tool-calling support and a higher free-tier token budget (200K TPD vs 100K TPD).

Same lesson as the Groq deprecations I hit on a different project: never assume a hosted model string is permanent. If a production system depends on one, it belongs in a config value, not hardcoded — the day it gets deprecated shouldn't be a surprise.

What I'd tell someone building something similar

Don't clean streamed text chunk-by-chunk if the pattern you're removing can span chunk boundaries. Buffer first, clean second.
A suspiciously fast response is a bug signal, not a win. If retrieval-augmented generation ever responds faster than a real embed-plus-inference round trip should take, check whether it actually retrieved anything.
Free-tier hosting often means ephemeral disk. If your app's state needs to survive restarts, it can't live on the same filesystem as the compute — put it in a database or hosted store that's decoupled from the app's lifecycle. In my case, fixing this also let me drop a backend service entirely.
Hosted model strings aren't permanent. Keep them in config, not hardcoded, and expect at least one deprecation on any project that lives long enough.

Known limitations

Being upfront about what isn't polished yet:

gpt-oss-120b's tool-calling via Groq occasionally malforms a function call on ambiguous questions, surfacing as a 500 error — not yet hardened.
screen_resume expects a separate deployed service and fails gracefully if it's unreachable; this feature isn't live in the deployed demo.
route_query's complexity gate is a word-count heuristic, not a trained classifier — it will occasionally misroute a short-but-hard question to the cheaper model.