Every time I solved one problem, another one appeared. Looking back, that's what made this project worth building.
I built Octogen: github.com/SplinterSword/octogen
When I started building Octogen, I had a very simple idea.
"What if I could ask questions about any GitHub repository instead of spending hours searching through files?"
It sounded straightforward.
Clone a repository.
Generate embeddings.
Store them somewhere.
Pass the retrieved context to an LLM.
Done.
...or at least that's what I thought.
Within a few days I realized the AI part was actually the easiest part of the project.
Everything around it was where the real engineering happened.
Why I Started Building It
Every internship or new project starts the same way.
You clone a repository.
Open fifty files.
Search for "auth".
Search for "login".
Search for "middleware".
Eventually you understand the architecture...after wasting half your day.
I wanted something different.
Instead of searching for filenames, I wanted to ask questions like:
"Where is authentication implemented?"
or
"Which file handles Stripe webhooks?"
That became the goal.
Not to replace documentation.
Not to replace GitHub.
Just reduce the time it takes to understand someone else's code.
The First Version Worked...
...until I tested it on real repositories.
Small repositories were great.
Large repositories completely broke everything.
The more realistic my test cases became, the faster I discovered all the assumptions I had made.
Problem 1: Embeddings Were the Easy Part
Most tutorials stop after saying:
Generate embeddings and perform similarity search.
That part actually worked.
The difficult question was:
What exactly should I embed?
Entire files?
Functions?
Paragraphs?
Random chunks?
If I embedded entire files, retrieval became noisy.
If I split everything into tiny chunks, the model lost context.
Eventually I settled on splitting documents intelligently and generating summaries before storing embeddings.
The summaries became just as valuable as the code itself because retrieval quality improved dramatically.
My biggest takeaway was simple:
Better context beats bigger models.
Problem 2: AI Costs Increase Faster Than You Think
I also wanted AI-generated commit summaries.
The first implementation was embarrassingly simple.
Every commit diff went directly to the same model.
It worked...
Until I tested repositories with hundreds of commits.
Then everything started failing.
Not because of bugs.
Because I was smashing into Groq's TPM (Tokens Per Minute) limits.
I had accidentally built a denial-of-service attack against my own API key.
The Fix
Instead of treating every request equally, I built a routing system.
Small commit?
→ Use a lightweight model.
Medium commit?
→ Use a larger Groq model.
Huge commit?
→ Send it to Gemini with its massive context window.
On top of that I added:
- Filtering to remove lock files and generated code.
- Batching instead of firing hundreds of requests simultaneously.
- Automatic fallback between providers.
The result wasn't just fewer failures.
It was dramatically cheaper.
Sometimes solving infrastructure problems saves more money than optimizing prompts.
Problem 3: The Repository Was Full of Noise
Another issue surprised me.
The AI wasn't hallucinating.
It was answering based on terrible context.
Large repositories contain things like:
package-lock.jsonbun.lockdist/.next/- Generated files
Those files contribute thousands of useless tokens.
The model happily consumed them anyway.
I realized I wasn't building an AI problem.
I was building a filtering problem.
After aggressively removing generated files before indexing, retrieval quality immediately improved.
Sometimes the best prompt engineering is deleting data.
Problem 4: Production Behaves Nothing Like Development
Everything looked perfect locally.
Then production builds started failing.
TypeScript errors.
ESLint errors.
Middleware behaving differently.
One particularly frustrating issue turned out to be almost funny.
Authentication simply refused to work.
Hours later I discovered the file was named:
proxy.ts
instead of
middleware.ts
One filename.
Several hours gone.
Framework conventions are much stricter than they look.
It was a painful reminder that sometimes the biggest bugs have the smallest fixes.
Problem 5: PostgreSQL Isn't Really a Vector Database
I decided against using Pinecone or another managed vector database.
I wanted one database.
PostgreSQL.
Adding pgvector sounded simple.
It wasn't.
Prisma doesn't natively understand vector columns.
That meant raw SQL.
Unsupported field types.
Custom similarity queries.
The documentation covered pieces of it, but putting everything together took much longer than expected.
In the end, though, keeping relational data and embeddings together made the architecture much simpler.
Sometimes avoiding another service is worth a little extra work.
The Biggest Lesson
When people look at AI projects they usually focus on the model.
The model was probably 20% of this project.
The other 80% looked like this:
- Handling API failures
- Managing rate limits
- Optimizing token usage
- Designing retrieval
- Reducing costs
- Dealing with framework quirks
- Making everything reliable
That's the part people don't usually post screenshots of.
But it's where most of the engineering happens.
Building AI products is much closer to systems engineering than prompt engineering.
If I Built It Again...
Now that I've gone through the process once, there are several things I'd approach differently.
I'd implement incremental indexing instead of rebuilding repositories from scratch.
I'd add GitHub webhooks so repositories stay synchronized automatically.
I'd introduce background job queues much earlier rather than handling long-running operations synchronously.
Most importantly, I'd invest in observability before adding more AI features.
It's much easier to improve a system when you can clearly see where it's breaking.
Final Thoughts
Building Octogen taught me something unexpected.
The hard part of AI products isn't calling an LLM.
Anyone can do that.
The hard part is building a system that continues working after thousands of files, hundreds of commits, API limits, production traffic, and every edge case you didn't think about on day one.
The AI was only one piece of the puzzle.
The real challenge was building the infrastructure around it.
And honestly...
That's what made building it fun.
Thanks for reading!
If you've built an AI application recently, I'd love to hear:
What engineering problem surprised you the most?
Project link: github.com/SplinterSword/octogen
Top comments (0)