My RAG App OOM-Crashed on a 512MB Free Tier — So I Rebuilt It, Then Broke It Five More Ways on AWS on Purpose

torch was quietly eating my entire memory budget, and the only symptom was a service that said "live" while every real request just hung and restarted. That was bug #1. It wasn't the last one.

What I built

Document Assistant is a RAG (Retrieval-Augmented Generation) app: upload a PDF, ask it questions in plain language, and get an answer streamed back token-by-token that's actually grounded in that document — not a hallucinated guess from a model that's never seen it. It's the standard pattern for Q&A over private documents.

• Repo: https://github.com/0Abhijeet/Document-assistant
• Live demo: https://document-assistant-okwx.onrender.com

Architecture, and why each piece is there

The first version was a single-file FastAPI app: FAISS for vectors saved to local disk, Ollama running a local model, no database, no tests, no deployment story. It looked like a working prototype and had a real correctness bug baked in — FAISS's local index got wiped and rebuilt on every upload, so only one document could exist in the system at any time. Ollama also assumes a persistent local model process, which doesn't fly on any stateless container host.

The rebuild:

Browser (chat UI) → FastAPI (Render)
                         │
                         ▼
                  ingest.py: PyPDFLoader → RecursiveCharacterSplitter → fastembed (384-dim)
                         │
                         ▼
                  Postgres + pgvector (Neon): documents / chunks / query_logs
                         │
                         ▼
                  retrieve.py: cosine similarity search
                         │
                         ▼
                  generate.py: Groq API (streamed)

Key decisions:

• Postgres + pgvector instead of a dedicated vector DB. One database holds document metadata, chunk embeddings, and query logs — simpler than running a second stateful system just for vectors.
• Neon for hosting. Free tier is permanent rather than a time-boxed trial, pgvector works out of the box, and there's none of Cloud SQL's VPC/networking overhead for what's a portfolio project at this stage.
• Groq instead of Ollama. Free tier, no credit card, an OpenAI-compatible client that's close to a drop-in replacement, and rate limits generous enough (30 req/min, 14,400/day) not to matter for a demo.
• fastembed instead of sentence-transformers. This was a decision I made after a production crash, not before — more on that below. Same 384-dimension output, so no schema changes were needed to switch.
• Alembic migrations, with the first migration also enabling the pgvector extension.
• pytest against a real Postgres instance, not SQLite. pgvector's Vector column type has no SQLite equivalent — you either test against the real thing or you're not really testing the schema.
• GitHub Actions CI that spins up a throwaway pgvector/pgvector:pg16 service container and runs the full suite on every push.

Ingestion is additive by design — uploading a new document never deletes existing ones. That's a direct fix for the FAISS bug, and it's backed by a dedicated regression test.

The interesting part: four bugs that actually taught me something

1. The production OOM crash-loop

The app deployed to Render and briefly reported "live," but every real request either hung indefinitely or the instance silently restarted. Render's free tier caps memory at 512MB, and sentence-transformers pulls in torch, which alone can approach that ceiling. I found this by reading Render's "Instance failed" event log, which showed the repeated silent restarts — the app logs themselves gave no useful signal.

Fix: swap sentence-transformers for fastembed, which runs on ONNX instead of torch and produces the same 384-dimension vectors at a fraction of the memory footprint. No schema migration required.

Lesson: a dependency you picked for convenience can quietly become your memory budget. Check what it drags in, not just what it does.

2. The container that was "running" against the wrong database

Later, on AWS, I hit a version of this bug that's arguably worse: the container started with no errors and looked completely healthy — but it was talking to a throwaway local Postgres container instead of the external database I intended (first Neon, then RDS).

Two things were compounding. DATABASE_URL was hardcoded inline in docker-compose.yml under app.environment, which takes precedence over the same key set in .env. Separately, depends_on: db: condition: service_healthy was auto-starting the local db service regardless of which service I actually asked docker compose up to run.

Fix: changed the compose file to reference ${DATABASE_URL} instead of a hardcoded value, and used --no-deps to stop the local db service from auto-starting.

Lesson: "container running" and "healthy" are not proof it's talking to the backend you think it is. Verify against the actual data source, not process status.

3. RDS "Connection timed out" — with both instances in the same VPC

psql hung and failed even though RDS showed "Available" and both instances lived in the same VPC. The RDS security group's inbound rule was IP-based (briefly, and wrongly, widened to 0.0.0.0/0 as a shortcut) instead of referencing the EC2 security group directly. EC2's traffic to RDS originates from its security-group identity inside the VPC, not from any external IP — so an IP-based rule can never match it, no matter which IP you put there.

Fix: deleted the IP-based rule and added a new one with the source set to the EC2 security group itself, selected from autocomplete rather than typed as an IP.

Lesson: SG-to-SG referencing and IP-based rules solve different problems and aren't interchangeable on the same rule. Also worth saying plainly: briefly opening a database's inbound rule to the whole internet to "just get it working" is a meaningfully worse mistake than doing the same on SSH — it's direct data exposure, not just a login prompt.

4. DATABASE_URL=DATABASE_URL=postgresql://...

After the RDS cutover, the app crashed with "Could not parse SQLAlchemy URL." The .env line had silently become DATABASE_URL=DATABASE_URL=postgresql://... — the key got typed twice while editing in nano. SQLAlchemy was trying to parse a string that literally started with DATABASE_URL=.

Fix: rewrite the line as a single clean assignment.

Lesson: a malformed .env line fails silently at the shell level and only shows up once the app actually tries to use the value. cat the file back and read it after every edit, before restarting anything.

Why AWS at all, if Render already works

The AWS deployment isn't a second production environment — Render remains the permanent, always-on demo. The AWS side exists specifically to close a resume/JD gap: RDS, VPC security groups, and IAM are named directly in target job descriptions, and the networking work involved (like the SG-to-SG fix above) is itself the transferable skill, not just a keyword to have on a resume.

The setup: EC2 (t2.micro, Ubuntu 22.04) running the app in Docker with a restart: unless-stopped policy, verified against a real instance reboot with no manual intervention; RDS PostgreSQL (db.t3.micro, PostgreSQL 16.x) with pgvector enabled, migrated from Neon via pg_dump / pg_restore --no-owner --no-privileges; a 2GB swap file on the EC2 instance since 1GB of RAM is tight for the embedding model; and an Elastic IP so the demo URL survives stop/start cycles. IAM used a dedicated console user instead of root, with MFA on root and a small cost budget with an alert threshold.

I deliberately scoped out Nginx/HTTPS termination for this deployment — it demonstrates general reverse-proxy skills, not AWS-specific ones, and doesn't address the actual goal.

Known limitations — stated plainly

• Thread-pool concurrency, not true async. FastAPI runs blocking def routes in a worker thread pool automatically, which avoids freezing the event loop but doesn't scale to heavy concurrent load. Fine for a demo, not for production traffic.
• No authentication. Anyone with the URL can upload or query documents. Fine for a portfolio demo, not for real private data.
• Deployment is manual, not part of CI — a deliberate scope cut, not an oversight.
• Cold starts on Render's free tier: the service spins down after inactivity, and the first request after idle time takes 30–60s.
• ivfflat index (lists = 100) is a reasonable default that hasn't been benchmarked against real data volume.
• On the AWS side, the SSH security group rule was temporarily widened to 0.0.0.0/0 to unblock EC2 Instance Connect — an open item to be scoped back to "My IP" before the exercise is considered closed.

What I'd do at scale

• Move to true async DB and embedding calls (asyncpg, async-compatible inference) instead of thread-pool concurrency
• Add authentication and per-user document scoping
• Automate deployment in CI instead of the current manual step
• Benchmark and tune ivfflat against real data volume, or evaluate HNSW indexing
• Add rate limiting on upload/query endpoints
• Add structured logging/observability beyond the current query_logs table
• On the AWS side: close the SSH rule back to "My IP," then tear down EC2/RDS/the Elastic IP once documentation is captured, so nothing keeps running unattended against the free-tier credit

Closing

If you're evaluating RAG architecture decisions, hit something similar with Docker Compose environment precedence, or want to talk through the AWS networking side — I'd like to hear from you.

• Repo: https://github.com/0Abhijeet/Document-assistant
• Live demo: https://document-assistant-okwx.onrender.com

---

About me: I'm Abhijeeth, an AI/ML engineer based in Chennai. I build and ship production RAG/LLM systems — FastAPI, pgvector, Docker, AWS. Open to full-time AI/ML Engineer roles. github.com/0Abhijeet

rag #aws #postgres #python

Comments

[Lisha Choudhary]

The fastembed/ONNX swap-out is a smart move a lot of people skip until they get hit with the same OOM crash in prod. Did you benchmark retrieval quality before/after the switch, or was it purely a memory-footprint call? Also appreciated seeing the actual RDS migration steps laid out — most writeups stop at "and then I deployed it," this one didn't.