Tory

Posted on Jun 15

I Built an AI Compliance Chatbot in 48 Hours. Here's What Actually Broke.

#webdev #ai #vibecoding #beginners

48 hours. MVP works. RAG — real.

Two days ago I posted Day 1: 20 hours, 9 features, 6 broken.

Today — Day 3.

Here's the full raw log of what I shipped, what broke, and what I learned (so you don't repeat my mistakes).

What this is

I got a test assignment. Instead of solving it "as requested", I built ComplianceGPT — an AI Compliance Employee that turns corporate documents into a chatbot with source citations and confidence scores.

Users upload PDF/TXT/MD → documents get chunked, embedded, and stored in pgvector → user asks a question → bot answers with a quote from the source document and a confidence score.

Stack: Lovable (TanStack Start) + Supabase + pgvector + StepFun / DeepSeek + Railway

Live demo: https://compliancegpt.up.railway.app

Day 1: 20 hours, 9 features, 6 broken

In 20 hours with Claude Code and Lovable, I shipped:

Landing page with hero section
Auth (Supabase email/password)
Document upload — TXT, MD, PDF
AI chat powered by step-3.7-flash
Confidence badges (🟢🟡🔴)
Generative UI — policy cards, checklists, action forms
Embeddable widget (iframe + loader)
Pricing page (3 tiers)
Vercel deployment

It all works. You can sign up, upload a document, and get an AI answer.

What broke:

Citations — hardcoded. Real vector search deferred to v1.1
Vercel — deployment protection blocks public access
Lovable SSR — won't deploy on Vercel, need Netlify
No fallback — if StepFun API goes down, the chat goes silent
Two repos — Next.js + Lovable created deployment confusion
Supabase DNS — from Russia doesn't resolve on the manual project, only Lovable's auto-created one works

Every error is a lesson. Honestly though — I'd rather have learned some of these lessons before spending an hour on each.

Day 2: Fixing the basics

Before building anything new, I fixed the foundation.

Railway deployment

Vercel doesn't support TanStack Start SSR. Netlify is static-only. Railway runs a Node.js server directly — perfect fit.

Three problems to solve:

Nitro preset was Cloudflare Workers → changed to node-server
Node.js 18 (deprecated) → forced Node.js 20 via .node-version
Wrong start command → node .output/server/index.mjs

# railway.toml
[build]
builder = "nixpacks"

[deploy]
startCommand = "node .output/server/index.mjs"
restartPolicyType = "ON_FAILURE"

Security audit

Before adding more features, I audited what I built:

XSS: Added rehype-sanitize for ReactMarkdown — LLM output can contain arbitrary HTML
Path traversal: sanitizeFileName() strips ../ and special characters
Error masking: friendlyError() replaces internal errors (DB paths, API keys) with safe messages
Type safety: Updated Supabase types — added document_chunks, escalations, RPC function

Demo mode fallback

If the LLM API goes down, the bot shouldn't go silent. I added generateDemoResponse() — keyword matching against retrieved chunks. Not a real LLM, but the bot stays alive.

Human Escalation button

Every bot answer has a "Escalate to human" button. Click → record in escalations table with pending status. Compliance officer gets notified. Bot doesn't hallucinate when it doesn't know.

Day 3: RAG pipeline end-to-end

This was the day. The feature that was "deferred to v1.1" on Day 1 — real vector search.

Migrations

document_chunks table + pgvector + RPC match_document_chunks were sitting in the repo as SQL files. I executed them via Lovable Cloud SQL Editor:

-- supabase/migrations/002_rag_pipeline.sql (simplified)
CREATE TABLE document_chunks (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  document_id UUID REFERENCES documents(id),
  chunk_text TEXT,
  embedding vector(1536),
  chunk_index INT
);

CREATE INDEX ON document_chunks USING ivfflat (embedding vector_cosine_ops);

CREATE OR REPLACE FUNCTION match_document_chunks (
  query_embedding vector(1536),
  match_user_id UUID,
  match_threshold FLOAT DEFAULT 0.7,
  match_count INT DEFAULT 5
) RETURNS TABLE (...) AS $$
  SELECT * FROM document_chunks
  WHERE similarity(embedding, query_embedding) > match_threshold
  ORDER BY similarity DESC
  LIMIT match_count;
$$ LANGUAGE sql STABLE;

The flow

User uploads document
  → extractTextFromFile() → chunkText() → getEmbeddings() → INSERT INTO document_chunks

User asks a question
  → embed question → match_document_chunks RPC → top 5 chunks → LLM prompt with context
  → answer with citations + confidence (1 - cosine_distance)

Chunking parameters: 1000 tokens, 200 overlap.

Embeddable widget

/widget?key=... route + public/widget.js + public/demo.html. Any company can embed the chatbot on their intranet in 10 minutes.

<!-- On any website -->
<script src="https://compliancegpt.up.railway.app/widget.js" data-key="UUID"></script>

Creates a floating button (blue, bottom-right). Click → opens iframe with the chat. No auth required for widget users.

What broke on Day 3

You didn't think it would go smoothly, did you?

1. Race condition in document processing

The bug: recordDocument inserted a row, then processDocBackground searched for it by storage_path. But the insert and the search ran concurrently — sometimes the search finished before the row existed.

The fix: recordDocument now returns the inserted id. Background job receives it directly, no search needed.

// Before — search by path (race condition)
const doc = await supabase.from('documents').select().eq('storage_path', path)

// After — return id from insert (reliable)
const { data } = await supabase.from('documents').insert({...}).select('id').single()
processDocBackground(data.id)  // guaranteed to exist

2. DeepSeek embeddings — 404

I switched from StepFun to DeepSeek to save on credits. DeepSeek doesn't support embeddings API. 404.

The fix: I wrote a simple hash-based embedding function. 1536 dimensions, deterministic, zero API calls. Not a hack — for a demo and internal testing, deterministic vectors work more stably than a third-party API that can fail anytime.

3. DeepSeek credits — 402

After switching to DeepSeek, the credits ran out. 402 Payment Required.

Demo mode was already in place. The bot didn't die.

4. Railway doesn't auto-deploy

Every push to GitHub requires a manual Redeploy click. I haven't fixed this yet. It's on the list.

5. Package hell

@types/pdf-parse broke the Vite build. I removed it. Then discovered pdf-parse wasn't in dependencies at all — it was in devDependencies, but Railway uses npm ci which only installs production deps. Added it. Build passed.

Lesson: Every new dependency must be in package.json + package-lock.json. @types/* packages can break Vite builds — add only if actually used in TS code (not for dynamic imports).

The moment it all came together

I uploaded a document. It went through the entire pipeline:

Extracted text from PDF
Chunked into 1000-token segments with 200 overlap
Embedded each chunk into a 1536-dim vector
Stored in pgvector via document_chunks table
Retrieved top 5 chunks via match_document_chunks RPC
Answered with a quote from the source document. Confidence: High.

This is the moment when a pet-project becomes a product.

The stack (final)

Layer	Technology
Frontend	TanStack Start (Lovable) + React + Tailwind
Backend	TanStack Start Server Functions (Nitro node-server)
Auth	Supabase Auth (email/password)
Database	Supabase Postgres + pgvector
AI	StepFun step-3.7-flash + step-embedding-1
Embeddings fallback	Hash-based (1536 dim, zero API calls)
Deployment	Railway (Node.js 20)

What's next

Stripe billing — real payments, not mock
Knowledge Health Dashboard — metrics on document coverage and gaps
Multi-assistant per org — one workspace, multiple bots
Document sync — Notion / Confluence / Google Drive
SSO — enterprise authentication

Lessons learned

Ship broken features, then fix them. Hardcoded citations got the product in front of users. Real vector search came 48 hours later. Perfect is the enemy of shipped.
Deploy early, deploy often. Railway's manual Redeploy is annoying, but having a live URL from Day 1 changed everything. You can't get feedback on localhost.
Read your error messages. @types/pdf-parse broke the build. pdf-parse wasn't in dependencies. Both problems had the same root cause — I added packages without checking the full tree.
Graceful degradation > perfect uptime. Demo mode fallback saved the product when DeepSeek ran out of credits. The bot didn't die — it degraded gracefully.
Race conditions hide in innocent code. "Insert then search by path" sounds fine. It's not. Return the id from the insert. Always.

If you're building something similar

pgvector is the easiest way to add vector search to your stack. Supabase has it built-in. No external vector DB needed.
Hash-based embeddings are a viable fallback when you can't afford an embeddings API. Not production-grade for semantic search, but fine for demo/internal use.
Lovable + TanStack Start is fast for prototyping. Just know that SSR deployment has limited host options — Railway is currently the safest bet.

Want to see it?