Octocode is our open source semantic code search engine built in Rust (Apache-2.0). It's the code-search layer behind Octomind, and you can grab it at https://github.com/muvon/octocode.
Last month we added an LLM reasoning step to the retrieval pipeline. The first version made things worse.
Hit@10 dropped 7 points. Recall@10 dropped 7.5 points. The model was pruning away true positives because it thought they weren't relevant.
Here's what we learned, what we shipped, and why "add contextual retrieval" is not the universal win everyone claims.
The Problem: Similarity Is Not Relevance
Octocode uses hybrid retrieval — vector search plus keyword overlap. Works well. But there's a gap: the snippet that shares the most tokens with your query is often not the code that answers it.
Ask "where do we decide a request is retryable" and the similarity ranker hands you:
- The retry config struct
- The retry constant
- The test that names the word four times
The actual function that makes the call? Rank seven.
Vector search ranks by similarity. Hybrid adds keyword matching. But neither reads the code. Neither knows what the code does.
First Attempt: Pure LLM Reranking
We added an LLM reasoning step after hybrid retrieval. The model reads the candidate code bodies and re-ranks by whether they actually answer the query. No new index, no reindexing — just a reranker sitting between retrieval and results.
Tested on 127 queries. Here's what happened:
| Metric | Before | After | Change |
|---|---|---|---|
| MRR | 0.595 | 0.752 | +0.157 |
| NDCG@10 | 0.658 | 0.758 | +0.100 |
| Hit@10 | 0.913 | 0.843 | −0.071 |
| Recall@10 | 0.886 | 0.811 | −0.075 |
MRR and NDCG went up. The ranking quality improved. But Hit@10 and Recall@10 dropped.
The LLM was being too aggressive. It returned only what it judged relevant and quietly discarded true positives at ranks 6–10. Better ranking head, worse recall floor.
The Fix: Fuse, Don't Replace
Don't let the LLM replace the ranking — fuse it with the hybrid ranking via Reciprocal Rank Fusion (RRF).
score = 1/(k + hybrid_rank) + reasoning_weight * 1/(k + reasoning_rank)
Hybrid rank always contributes — that's your recall floor. Reasoning rank drives the head.
Final fused results on the same 127 queries:
| Metric | Before | After | Change |
|---|---|---|---|
| MRR | 0.595 | 0.809 | +36% |
| NDCG@10 | 0.658 | 0.833 | +27% |
| Hit@5 | 0.827 | 0.953 | +0.126 |
| Hit@10 | 0.913 | 0.969 | +0.056 |
| Recall@5 | 0.777 | 0.924 | +0.147 |
| Recall@10 | 0.886 | 0.944 | +0.058 |
Every metric up. Hit@5 at 0.953 means nineteen times out of twenty the answer is in the top five.
What We Tuned (and What Didn't Matter)
Reasoning weight: Best at 2.0. Weight 5 buys nothing and costs recall.
Candidates: 25 is the sweet spot. We tried 40 — worse across the board.
Context level: Full code bodies win by a lot. Signatures-only was the worst performer.
LLM temperature: 1.0 beat 0.3 and 0.0. Counterintuitive, but the model reasons better with normal sampling. Stiff decoding made it dumber.
What Did NOT Work: Contextual Retrieval
Anthropic's contextual retrieval approach — adding descriptions at index time — gets recommended everywhere. We tested it.
Hit@5: −0.008
Recall@10: −0.019
It trades recall for ranking. On code, that's a net loss. "Add contextual retrieval" is cargo-culted advice — not a universal win. We didn't ship it.
How to Turn It On
The feature is in Octocode's master branch now (not yet released — landing soon). Off by default, one config flag:
[search.reasoning]
enabled = true
model = "deepseek:deepseek-v4-flash"
max_candidates = 25
context_level = "full"
reasoning_weight = 2.0
final_top_k = 10
Or swap in any other provider:model:
[search.reasoning]
enabled = true
model = "openai:gpt-4o-mini"
max_candidates = 25
context_level = "full"
reasoning_weight = 2.0
final_top_k = 10
Any provider:model works. One LLM call per semantic search. structural_search stays pure grep. No reindex needed.
The Takeaway
LLM reranking works – but only if you keep the hybrid ranking as a floor. Fusion beats replacement. And don't cargo-cult "best practices" without testing them on your actual workload.
Octocode is open source under Apache-2.0. The benchmark suite, ground truth dataset, and raw results are all in the repo: https://github.com/muvon/octocode
This feature is merged to master now, not yet released – landing soon.
Full writeup with all the numbers and tuning details: https://muvon.io/blog/reasoning-retrieval-code-search
Top comments (0)