Testing showed a true exact-symbol match — the strongest possible signal you can get in a code search — losing to weaker results that just happened to show up in more ranked lists. That's not a crash. It's worse: the system runs fine and quietly gives you the wrong answer.
What I built
Codebase Q&A Assistant lets you point it at a GitHub repository and ask natural-language questions about the Python code inside it — answers come back grounded in the actual retrieved source, not general knowledge about "what code like this usually does." It clones the repo, parses it with Python's AST, chunks it into meaningful semantic units, embeds those chunks, and answers questions by retrieving and citing the real source lines.
- Repo: (https://github.com/LISHA-tech222/codebase_qa)
- Live demo: (https://codebase-qa-jgle.onrender.com)
Architecture, and why each piece is there
GitHub repo (source input)
│
▼
AST chunking + FastEmbed embeddings
│
▼
PostgreSQL + pgvector storage
│
▼
User question ──▶ Hybrid retrieval (exact match + semantic + RRF)
│
▼
Groq LLM
│
▼
Answer + validated citations
AST chunking instead of line-based chunking. Functions, classes, and methods are meaningful semantic units; arbitrary line ranges can split a function in half. A question about a specific method should retrieve that actual method, not two fragments of it.
Classes stored whole, and methods stored again separately. A class chunk gives context; a method chunk gives focused recall. The tradeoff, stated plainly: this deliberate overlap can produce near-duplicate results in retrieval.
A synthetic module-level chunk for imports and constants, so nothing indexed silently disappears — at the cost of an approximate line range, since module-level lines aren't always contiguous.
Trivial-function detection went through two designs. The first attempt flagged any function with two or fewer non-blank lines as trivial — and it wrongly flagged legitimate one-line functions. The fix was to stop counting lines and instead inspect the AST body directly for actual Pass, Ellipsis, or docstring-only content. Lesson: structural information beats line-count heuristics.
PostgreSQL + pgvector, one datastore for both relational metadata and vector search, instead of running a second vector-specific system.
Hybrid retrieval, not pure semantic search. Code identifiers matter enormously — a question like "what does parse_config do?" should strongly favor the symbol literally named parse_config. But semantic search is still necessary for questions like "how does the application read configuration?" where the exact identifier never appears. So retrieval runs a tiered exact/keyword search (exact name → prefix → substring → content match) alongside pgvector cosine similarity, and merges the two ranked lists with Reciprocal Rank Fusion (k = 60) — RRF combines rank positions rather than raw scores, which matters because exact-match scores and cosine similarity live on completely different scales and a weighted blend would need arbitrary normalization to combine them.
Groq over Anthropic, chosen for the free-tier-friendly stack and an OpenAI-compatible chat completions interface that made integration close to drop-in.
The interesting part: four things that actually taught me something
1. RRF was ranking the best possible signal below noise
Once hybrid retrieval was running, testing surfaced a real problem: a true, case-insensitive exact symbol match — the hardest signal retrieval can produce — could get outranked by mediocre results that simply happened to appear in both the exact and semantic ranked lists and accumulate a better combined RRF score. Pure rank-fusion, in other words, was treating "showed up twice, weakly" as more important than "matched exactly, once, perfectly."
Fix: pin true exact symbol matches above the RRF-merged remainder — exact tier-1 matches + RRF(everything else) — instead of feeding them into RRF as just another input.
Lesson: exact symbol matching is a hard signal and semantic similarity is a soft one. A ranking algorithm that treats them as interchangeable inputs will occasionally let the soft signal win, and in code search specifically, that's the wrong failure mode to allow.
2. Alembic and SQLAlchemy fighting over the same ENUM
A migration failed with DuplicateObject: type "symbol_type" already exists. The project uses a PostgreSQL ENUM for symbol_type specifically to keep invalid symbol types out of the database — but the explicit .create() call in the Alembic upgrade step and SQLAlchemy's automatic ENUM creation during table creation were both trying to create the same type.
Fix: remove the explicit upgrade-side .create() call and let table creation handle it, while keeping the explicit drop behavior on downgrade.
Lesson: when an ORM auto-creates a Postgres type as a side effect of table creation, an explicit migration step doing the same thing isn't redundant safety — it's a conflict waiting to happen.
3. AWS's RDS setup wizard defaults to the paid option
While provisioning RDS for the AWS deployment exercise, "Managed in AWS Secrets Manager" was the pre-selected credentials option — which bills per secret and isn't part of the RDS free tier. RDS Proxy, sitting right next to it, carries the same dependency.
Fix: switch to "Self managed" credentials before creating the database, and leave RDS Proxy disabled.
Lesson: several AWS wizards default to their most "managed"/convenient option, and that option is frequently the paid one. Worth a deliberate second look at credentials, monitoring, and proxy settings specifically for hidden cost during database creation — not just at launch time.
4. A "failed" migration that had actually already succeeded
pg_restore returned a wall of "already exists" and "duplicate key" errors — the kind of output that reads as a failed migration at a glance.
Root cause: the restore had already completed successfully on an earlier run. Every single error was a duplicate-object or duplicate-key error, which by definition only happens when the schema and data are already present — it's the signature of a second run against already-migrated data, not a broken one.
Fix: stopped trusting the error list and verified directly — \dt to confirm the tables existed, then SELECT COUNT(*) FROM chunks; checked against the known source row count.
Lesson: pg_restore error output needs to be read for what kind of error it is, not just whether errors exist. Verify against actual data, not exit status or error text.
Why deploy to AWS at all, if Render already works
The AWS deployment (EC2 + RDS, region eu-north-1) wasn't a second production environment — Render remains the permanent, always-on version. It exists because RDS, VPC security groups, and IAM are named directly in target job descriptions, and the SG-to-SG networking work is itself the transferable skill, not just a keyword to list. EC2 ran the existing Docker image directly (docker build + docker run --restart unless-stopped, no compose file, consistent with the project's existing single-Dockerfile setup) rather than ECS/Fargate — Fargate has no free-tier allocation and bills per vCPU-second immediately, so a single EC2 instance demonstrates the same containerized-deployment skill without that cost; that's a cost-driven tradeoff, stated plainly, not a technical limitation being hidden. RDS's only inbound rule referenced the EC2 security group directly rather than any IP range, and an Elastic IP kept the demo URL stable for the duration of the exercise. Everything — EC2, RDS, the Elastic IP — was verified working end-to-end in a browser, documented, and torn down in the same session.
Known limitations — stated plainly
- Python only. The parser targets Python specifically; Tree-sitter would be the natural path to multi-language support.
- Approximate module-level citations, since the synthetic module chunk's line range is a min/max over lines that aren't always contiguous.
- Invalid citations are silently stripped, but the underlying unsupported statement in the answer text remains — the UI doesn't yet distinguish verified from unverified claims.
-
Synchronous ingestion. Large repositories can make
/ingestslow, since processing happens inline during the HTTP request rather than as a background job. - No repository size limits or filtering yet for very large codebases.
- Public repositories only — cloning currently assumes an accessible repo; private repo support would need GitHub OAuth or App-based auth.
What I'd do at scale
- Multi-language support via Tree-sitter instead of Python's
astmodule - Background ingestion jobs with progress reporting and incremental indexing, instead of processing inline on the request
- Distinguish verified source-backed claims from unsupported ones in the UI, instead of silently stripping invalid citation tags
- Rate limiting and repository size limits on
/ingest - Streaming answers, chat history, and syntax-highlighted source in the UI
- On the AWS side: HTTPS termination (reverse proxy or ALB), a move to ECS/Fargate once cost isn't a constraint, CI/CD deploying straight to EC2 (GitHub Actions → ECR → SSH deploy), and basic Terraform for the EC2/RDS/security-group resources instead of manual console setup
Closing
If you're working on retrieval ranking for code search, have hit the same Alembic/SQLAlchemy ENUM conflict, or want to compare notes on the AWS side — I'd like to hear from you.
- Repo: (https://github.com/LISHA-tech222/codebase_qa)
- Live demo: (https://codebase-qa-jgle.onrender.com)
About me: I'm Lisha, An AI/ML engineer based in Chennai. I build production RAG systems with hybrid retrieval, FastAPI, PostgreSQL/pgvector, Docker. Open to full-time AI/ML Engineer roles. https://github.com/LISHA-tech222
Top comments (1)
The RRF implementation for hybrid retrieval is the part I'd have gotten wrong first try — combining exact and semantic scores without one dominating is genuinely fiddly. Curious what your fusion weights ended up being and whether you tuned them empirically or against a benchmark set. Also good call documenting the citation validation piece — hallucination mitigation rarely gets a real writeup, usually just a footnote.