I built an agentic RAG assistant — upload a PDF, ask questions, get streamed answers grounded in the document. It uses LangGraph for tool orchestration, Groq for fast inference, Qdrant for retrieval, and Streamlit for the UI.
It worked. Mostly. Every so often, a stray 【source】 tag — an internal citation marker never meant for the user — leaked straight into the answer on screen. Not always. Not predictably. That inconsistency turned out to be the whole puzzle, and chasing it down taught me more about streaming systems than the rest of the project combined.
That was the first of two bugs that surfaced during demo prep — the second one had nothing to do with streaming and everything to do with where I was storing state.
What the project does
Upload a PDF, ask a question, and the agent:
- Retrieves relevant chunks from a vector store
- Answers only from retrieved content — if nothing relevant comes back, it says so instead of guessing
- Can route queries between a cheap model and a larger one based on complexity
- Can screen resumes through a separate fine-tuned LoRA model
Stack: LangGraph (ReAct-style agent) · Groq running Llama 3.3 70B for inference · Qdrant for the vector store (originally Chroma — more on that below) · HuggingFace embeddings · FastAPI backend · Streamlit frontend.
The assistant answering only from the uploaded PDF, with sources quoted directly. (nimbus_overview.pdf is a fictional test document I wrote for the demo — not a real product.)
The bug: a marker that only sometimes leaked
The agent tags its retrieval citations internally with a marker like 【source】, which is supposed to get stripped before the answer reaches the user. Simple enough — clean each chunk of streamed text with a regex as it comes in.
Except sometimes it didn't get stripped.
The cause: streaming breaks text into arbitrary token chunks, and the marker was getting split across two of them. One chunk would end with 【sour, the next would start with ce】. Since I was running the regex on each chunk in isolation, neither half ever matched the full pattern — both leaked straight through, untouched.
The fix wasn't a smarter regex. It was buffering: hold back any text that could be the start of an incomplete marker, and only release it once you're sure you're not sitting mid-tag.
_CITATION_ARTIFACT_RE = re.compile(r"【[^】]*】")
def stream_agent(question: str):
"""Yield the answer as it streams in, with citation markers
stripped even when a marker is split across chunk boundaries."""
buffer = ""
for chunk in agent.stream(question):
buffer += chunk
while True:
start = buffer.find("【")
if start == -1:
if buffer:
yield buffer
buffer = ""
break
if start > 0:
yield buffer[:start]
buffer = buffer[start:]
end = buffer.find("】")
if end == -1:
break
buffer = buffer[end + 1:]
Streaming still feels instant to the user. Nothing incomplete ever reaches them — the buffering adds negligible overhead, invisible in normal use.
The broader lesson: streaming systems fail in ways batch systems simply can't. A batch-mode version of this same assistant would never have hit this bug — it processes the full response as one string before cleaning it. This bug only existed because I was processing partial state in real time, and for a doc-QA tool, a leaked internal marker is exactly the kind of thing that quietly erodes trust in the answer itself, even when the answer underneath it is correct.
The second bug: it felt broken because it was, structurally
Bug #1 was a parsing problem. Bug #2 turned out to be an architecture problem wearing a parsing-problem costume.
Once the streaming issue was fixed, a different problem showed up during demo prep: every time the backend went idle and spun back down, the app "forgot" everything. Users had to re-upload the same PDF and re-ingest it before asking anything, even though nothing about the workflow suggested that should be necessary.
The cause here wasn't a code bug — it was an architecture mismatch. The backend was deployed on Render's free tier, which uses an ephemeral filesystem: anything written to local disk, including the Chroma vector store, gets wiped on every restart or spin-down. Free-tier services spin down automatically after 15 minutes of inactivity, which is basically guaranteed for a portfolio demo that isn't getting constant traffic.
One giveaway made this obvious in hindsight: a response that came back in 0.86 seconds. That's far too fast for a real embed → retrieve → LLM round trip. It wasn't a fast correct answer — it was an empty vector store returning nothing, instantly.
The fix was to stop treating the vector store as something that lives on the server's local disk at all, and move it to a hosted vector database that persists independently of the backend's lifecycle:
# src/vectorstore.py — shared, persistent vector store client
import os
from langchain_qdrant import QdrantVectorStore
from langchain_huggingface import HuggingFaceEmbeddings
from qdrant_client import QdrantClient
embeddings = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5")
client = QdrantClient(
url=os.getenv("QDRANT_URL"),
api_key=os.getenv("QDRANT_API_KEY"),
)
def get_vectorstore(collection_name: str = "documents"):
return QdrantVectorStore(
client=client,
collection_name=collection_name,
embedding=embeddings,
)
Ingestion writes to Qdrant instead of a local Chroma folder, and retrieval reads from the same remote collection — so a Render restart no longer wipes anything out. Once retrieval no longer depended on local state surviving a restart, the "upload every time" problem disappeared along with it. A cold-start query against a previously ingested PDF now correctly retrieves chunks and returns a grounded answer, even after 30+ minutes idle — no re-upload needed.
What I'd tell someone building something similar
- Don't clean streamed text chunk-by-chunk if the pattern you're removing can span chunk boundaries. Buffer first, clean second.
- A suspiciously fast response is a bug signal, not a win. If retrieval-augmented generation ever responds faster than a real embed-plus-inference round trip should take, check whether it actually retrieved anything.
- Free-tier hosting often means ephemeral disk. If your app's state needs to survive restarts, it can't live on the same filesystem as the compute — put it in a database or hosted store that's decoupled from the app's lifecycle.
- Timeouts should scale with your actual worst-case chain, not a default guess. A request that hops through a cold backend, a cold embeddings API, and an LLM call needs more budget than a single quick round trip.
Try it / see the code
- Code: https://github.com/ayush-s-tomar/agentic-rag-research-assistant
- Live demo: https://agentic-rag-groq.streamlit.app
- API docs: https://agentic-rag-research-assistant-jjch.onrender.com/docs (read-only browsing recommended — it shares the same free-tier API keys as the live demo, so please go easy on it)
⚠️ Heads up if you try the live demo: it's on Render's free tier, so the first request after a period of inactivity can take 30–60 seconds to wake the backend up. That's expected, not a bug.
If you've hit a bug that only showed up because of streaming, chunking, or an ephemeral filesystem, I'd love to hear about it in the comments.
Top comments (0)