Building a ChatGPT-style assistant looks easy in a demo and breaks in three places when it goes to production: the responses don't stream (users stare at a spinner), the model hallucinates (it has no idea what's in your docs), and your costs explode. The difference between a demo and a shippable assistant is four techniques composed correctly.
I built a real SaaS "ask the docs" assistant in React and measured it. This is the condensed version; the full guide (the streaming route, the RAG pipeline, the vector-store decisions, and every metric) is on my site 👇
Full guide: https://prepstack.co.in/blog/react-ai-chatgpt-like-application-streaming-openai-rag-vector-databases-guide
The result (real SaaS docs assistant, ~3,000 docs, ~50k questions/mo)
| Metric | Before (just LLM) | After (streaming + RAG + pgvector) |
|---|---|---|
| Hallucination rate | 18% | 4% |
| Irrelevant answers | 35% | 8% |
| Answer cites a source | 0% | 96% |
| "This was helpful" | 51% | 84% |
| Time to first token | 3,500ms | 250ms |
| pgvector retrieval (p95) | — | 65ms |
| Cost per query | — | $0.003 |
| Support tickets deflected | — | 23% of product Qs |
The four techniques
1. Streaming — non-negotiable UX, ship it first. A full response is a 3–13s wait; the first token lands in ~250ms. That took time-to-first-content 3,500ms → 250ms and abandon rate 14% → 3%. Use Server-Sent Events, not WebSocket — one-way server→client is all chat needs, and it's what OpenAI's own API uses.
2. OpenAI behind a server route — never the browser. The API key stays server-side, and the route is where rate limiting (per tenant + per user), prompt validation, cost caps, and observability live. Edge runtime streams best; log usage after the stream closes, never blocking on telemetry.
3. RAG stops hallucinations. Retrieve the relevant chunks from your docs, put them in the prompt, instruct the model to answer only from them. Hallucinations 18% → 4%, "helpful" 51% → 84%, and it deflected 23% of incoming support questions. The underrated lever is chunking + retrieval quality, not the model — a tuned RAG pipeline on gpt-4o-mini beats a naive one on gpt-4o, for less money.
4. pgvector is enough for almost everyone in 2026. The Postgres you already run, with an HNSW index → sub-100ms retrieval (p95 65ms) up to ~10M vectors. Resist adding a managed vector DB before you've tried Postgres; reach for Pinecone/Qdrant only at higher scale or specific latency needs.
The architecture
React (useChat) --POST /api/chat (SSE stream)--> Next.js API route
| 1. rate-limit + validate
| 2. embed question (OpenAI)
| 3. retrieve top-k (pgvector cosine)
| 4. build grounded prompt
<---- tokens stream back ----------------------- 5. stream from gpt-4o-mini
6. log usage + cache
The first SSE event sends the retrieved sources so the UI shows citation chips within ~100ms while the answer streams in.
The economics
Cost per query landed at $0.003. A response cache on the question hash (34% of questions are repeats) saves ~$4,200/month at 1M queries — without it the bill is 50% higher. One warning: watch the cost dashboard daily the first month. A single tenant looping the API can run a five-figure bill before you notice. Set hard per-tenant caps.
The mental model
A production AI assistant is four boring components composed correctly: a streaming UI, a server-side LLM call, retrieval over your own data, and a vector store to make that retrieval fast. None is exotic in 2026 — the engineering is making them work together: fast enough to feel instant, accurate enough to be trusted, cheap enough to scale. Ship streaming + RAG on day one; everything else is optimization.
The full guide has the complete streaming Next.js route, the manual + Vercel-AI-SDK React useChat, the RAG pipeline (chunking, embeddings, pgvector storage & retrieval), the HNSW vs IVF decision, prompt-injection defenses, cost monitoring, and the full latency/cost/quality/reliability tables:
Originally published on PrepStack.
Top comments (0)