The Challenge
Three months ago, I participated in the Redrob India Data & AI Challenge. Track 1: given 100,000 candidate profiles and a job description for a Senior ML/AI Engineer role, build an AI system that ranks candidates the way a great recruiter would not by matching keywords, but by actually understanding who fits.
The output needed to be a ranked CSV of the top 100 candidates, with reasoning for each pick.
I'm a cloud and backend engineer, about 2 years into my career. I've never formally studied machine learning. I learned what I needed by building things. This felt like exactly that kind of problem.
Why Keyword Matching Fails
Before jumping into what I built, here's the thing that bothered me about standard systems.
A candidate who writes "built dense retrieval pipeline serving 50M+ queries" and another who writes "implemented embedding-based search system" are saying the same thing. BM25 the industry standard keyword matcher sees two completely different documents because the words don't overlap.
Meanwhile, a verbose mediocre engineer who keyword-stuffs their resume scores higher than a terse brilliant engineer who just ships.
And humans doing this at scale have their own version of the same problem: juggling 20+ signals inconsistently across hundreds of profiles. Notice period, location, GitHub activity, company prestige, career trajectory, skill depth no one can hold all of that in their head fairly.
I wanted to build something that combines signals the way a thoughtful recruiter actually would.
The Architecture: Don't Depend on Any Single Signal
The core insight I started with: no single signal is trustworthy alone.
BM25 misses semantic matches. Embedding similarity misses keyword-heavy roles. Skill matching without trajectory context rewards buzzword collectors. Trajectory scoring without behavioral signals misses the person who's perfect on paper but wants 6 months notice.
So I built a 4-signal system where every signal checks the others.
The Formula
score = (0.20 × BM25 + 0.30 × FAISS + 0.20 × Skill + 0.30 × Trajectory)
× yoe_factor
× ml_product_years_factor
× gate_multiplier
× behavioral_gate
Let me explain each piece.
Stage 1: Precompute Everything (once, ~1.7 hours)
I split the pipeline into two stages. Stage 1 is the heavy lifting run once, save everything to disk. Stage 2 is pure math runs in under 2 minutes on CPU.
Signal 1: BM25 (weight: 0.20)
Classic keyword search using BM25Okapi. I built a query from the JD specific terms like "vector search", "FAISS", "learning to rank", "production ML", "retrieval" and scored all 100K candidates against it.
BM25 is fast and good at catching explicit keyword matches. It's also easily gamed, which is why it only gets 20% weight.
Signal 2: FAISS Cosine Similarity (weight: 0.30)
I encoded all 100K career text descriptions using BAAI/bge-small-en-v1.5 into 384-dimensional vectors, then searched with the JD vector using FAISS IndexFlatIP.
Why BGE-small over the more common all-MiniLM-L6-v2? On MTEB retrieval benchmarks, BGE-small scores 61.7 vs MiniLM's 56.9. That's an 8.4% improvement and with the same 384 dimensions and similar inference speed. For a retrieval task specifically, it matters.
BGE also uses asymmetric encoding: the JD (query) gets a prefix:
jd_text = "Represent this sentence for searching relevant passages: " + jd_embed_text
Candidate texts are encoded without prefix. This is the correct BGE usage for retrieval and it makes a real difference in practice. Meta's Lead AI Engineer jumped from rank 13 to rank 9 after switching from MiniLM to BGE-small.
Signal 3: Skill Quality (weight: 0.20)
For every skill in the dataset, I computed cosine_similarity(skill_embedding, jd_vector) to get a relevance score. Then for each candidate:
skill_quality = Σ (skill_relevance × proficiency_weight × min(1.0, duration_months / 24))
So "Expert in Vector Search for 36 months" beats "Beginner in Vector Search for 48 months". And only JD-relevant skills count Figma proficiency doesn't help a ranking engineer.
Signal 4: Trajectory (weight: 0.30)
This is the most novel signal and the one I'm most proud of.
trajectory = 0.35 × production_score
+ 0.25 × pre_llm_score
+ 0.15 × still_coding
+ 0.10 × title_progression
+ prestige_bonus
Production score: I scanned career descriptions for production markers "shipped", "deployed", "serving N users", "A/B test", "latency", "99th percentile". Engineers who actually shipped things use this vocabulary. Those who only researched don't.
Pre-LLM ML depth: Total months spent on scikit-learn, PyTorch, XGBoost, TensorFlow, FAISS, Spark MLlib. JD explicitly says "if you learned ML after the LLM wave, this role isn't for you." I implemented a soft ramp to 48 months instead of a hard cutoff so someone with 3 skills × 18 months each scores properly instead of getting zeroed out.
Still coding: GitHub activity score from the platform signals + coding keywords in current role. Penalizes engineers who've moved entirely into management.
Title progression: Senior/Lead/Principal/Staff titles get a 1.30× multiplier on the title progression component.
Prestige bonus: This is data-driven from the candidates' own profile fields. Instead of a hardcoded list of "good companies", I read industry and company_size directly from the JSONL. Software + 10000+ employees → 0.15 bonus. Fintech/SaaS/AI + mid-size → 0.10 bonus. Only the FAANG tier (Google, Meta, Apple, etc.) gets a hardcoded 0.20 override. This means Razorpay, Zomato, Paytm automatically get classified correctly based on their industry + size no manual list maintenance.
The Gates: Hard Filtering Before Ranking
Some candidates shouldn't be in the top 100 regardless of score. I implemented hard gate multipliers:
| Condition | Multiplier |
|---|---|
| Honeypot (impossible profile data) | × 0.00 |
| Anti-title (Operations Manager, HR, Accountant...) | × 0.05 |
| Consulting-only career (TCS, Infosys, Wipro entire career) | × 0.20 |
| CV/Speech primary domain | × 0.15 |
| Pure research (no production evidence) | × 0.15 |
| Framework-only (only LangChain/LlamaIndex, no real ML) | × 0.20 |
And a behavioral gate (range 0.50–1.20) that accounts for:
- Notice period (≤30 days → +0.15, >120 days → -0.10)
- Location (Pune/Noida → +0.20 per JD)
- Willingness to relocate (+0.10)
- Overseas without relocation intent (-0.30)
- GitHub activity signals
Years of Experience Soft Multipliers
The JD says 5–9 years. I implemented:
- Under 3.5yr → hard exclude (hard gate)
- 3.5–5yr →
max(0.40, yoe / 5.0)soft penalty - 5–9yr → 1.0× (ideal)
- Above 9yr → taper:
max(0.85, 1.0 - (yoe - 9.0) × 0.02)
And separately: ml_product_years years spent in applied ML roles at product companies (not consulting). JD explicitly requires "4-5 years in applied ML at product companies." This was a direct scoring multiplier: min(1.0, 0.50 + (ml_yrs / 4.0) × 0.50).
The Reasoning Layer
Every candidate gets a written explanation in the CSV. No LLM involved all facts come directly from the raw profile.
Three tiers based on rank:
Ranks 1–10: Full evidence company, ML years at product companies, GitHub activity score, and a recent work description snippet.
5.9yr exp, currently Senior AI Engineer at Apple; top skills: scikit-learn,
TensorFlow, Python; 5.8yr at ML product companies github_activity=97;
Trivandrum, Kerala; immediately available (30d notice). Recent: Built and
shipped a production recommendation system at a marketplace product, going
from offline experimentation to live A/B test in 5 months.
Ranks 11–50: Summary with top skills, location, availability, and the most recent work description where not duplicated.
Ranks 51–100: Gap analysis honest about why they ranked lower, with specific concerns (job-hop rate, limited ML product tenure, semantic mismatch).
I also de-duplicated reasoning synthetic data had identical career descriptions, so I tracked seen descriptions and skipped the "Recent:" snippet for duplicates, preventing identical reasoning entries in the CSV.
Problems I Actually Hit
1. ml_product_years was computed but never wired in
I had a whole function that computed years in applied ML at product companies. It was stored in features.pkl. It was never referenced in rank.py. The JD literally says this is a requirement. I caught it during a cross-read of the scoring formula. Fixed by adding it as a multiplier.
2. Prestige was hardcoded
My first version had a list of "good companies" hardcoded. That's not data-driven it's bias. Switched to reading industry + company_size from the JSONL. Razorpay (Fintech + 5000+ employees) now auto-classifies correctly. Infosys (IT Services) auto-penalizes. No manual list maintenance.
3. Pre-LLM 24-month hard cutoff
I had if pre_llm_months < 24: score = 0. A candidate with 3 skills × 18 months each = 54 months total but no single skill over 24 months got zeroed out. Wrong. Switched to min(1.0, deep_months / 48.0) a soft ramp to 48 months.
4. HuggingFace rejecting binary files via git
HF now uses Xet storage for binary files and rejects them via regular git push. I had 154MB FAISS index and 175MB BM25 index. Solution: uploaded them via huggingface_hub Python API to a separate HF Dataset repo (Haripvelu/redrob-artifacts), then used hf_hub_download() in the demo to fetch them on first run.
5. TODAY hardcoded
I hardcoded TODAY = date(2026, 6, 13) during development and forgot to change it. It was used for computing "days since last active" in the behavioral gate. Fixed to TODAY = date.today().
6. Reasoning cut mid-sentence
I was truncating reasoning at 200 characters with a word-boundary cut. "Built and shipped a production recommendation system at a marketplace product, going from..." got cut after "going" sometimes. Switched to sentence-boundary detection using rfind('.') within 400 chars.
Results
Top 10: Apple, Salesforce, Mad Street Den, Zomato, Amazon, Razorpay, Ola, Microsoft, Meta, Netflix. All product companies. All India-based. All 5–9yr YoE range. All ML/NLP/search roles.
After switching to BGE-small:
- Meta Lead AI Engineer: rank 13 → rank 9 ✅
- Zomato Senior ML Engineer: rank 8 → rank 4 ✅
- Google Search Engineer: rank 17 → rank 40 ✅ (their actual skills were SAP, YOLO, Kubeflow not a semantic match for NLP/search despite the "Google" halo)
Evaluation:
- 62/62 custom eval checks passing (format validation, artifact integrity, signal sanity, adversarial probes, ranking quality)
- 48/48 unit tests passing
- 0 honeypots in top 100
- 0 consulting-only in top 100
- validate_submission.py (official checker) passes
Runtime: ~1.7 hours precompute (one-time), under 2 minutes to rank 100K candidates on CPU.
What I'd Do Next
Learning to Rank: Replace the hand-tuned weights with LambdaMART trained on historical recruiter feedback. The 0.20/0.30/0.20/0.30 weights were arrived at through experimentation and intuition real feedback data would make them rigorous.
LLM Re-ranker on top 50: Add a small local model (Phi-3 Mini) to re-rank the top 50 after the initial retrieval. Still CPU-friendly, but adds semantic reasoning the embedding similarity can't capture.
Dynamic JD embedding: The JD embed text is currently hardcoded in precompute.py. The pipeline should accept any JD and re-embed at precompute time. That makes it a general-purpose ranking system, not a one-role system.
Three Months Later
I submitted everything. GitHub, HuggingFace Space with live demo, ranked CSV, PDF deck answering all the required questions. I waited.
Still waiting.
I don't know if results were announced privately, if the challenge was extended, or if it quietly ended. I'm not bitter I got something valuable regardless: a real end-to-end ML pipeline I built from scratch under pressure, a working understanding of hybrid retrieval systems, and a few bugs caught that I'd have shipped to production.
If you're building something similar a hiring system, a recommendation engine, a search ranker the core lesson is: don't depend on any single signal. Ensemble them. Gate the obvious failures hard. Be honest in your reasoning. And build the eval suite before you think you need it.
Code
GitHub: github.com/Harivelu0/redrob-ranker
Live demo: huggingface.co/spaces/Haripvelu/redrob-ranker
The Space runs the full BM25 + FAISS pipeline (not a fake demo) the large indexes are fetched from HF Dataset on first click. Takes ~45 seconds to download, then ranks instantly.
I'm Haripriya (hp). I'm a software engineer focused on cloud, backend, and ML infrastructure. I write about building real things.
Top comments (0)