DEV Community

Cover image for Before we had users, we wrote the CVs ourselves
Jagriti
Jagriti

Posted on Originally published at kandidate.ai

Before we had users, we wrote the CVs ourselves

kandidate.ai matches an engineer's CV against live tech jobs in Germany, in English or German. Early on we hit the problem every search product hits: how do you know the results are good before people use it? On day one you have job ads and nothing else. So we turned job ads into test candidates. We took a real job, built a strong-fit candidate from its own title and skills, and checked whether the matcher put that job near the top. A CV built from an ad should find that ad.

Making test data without users turned out to be the easy part. Keeping the measurement tied to what we actually serve was harder, and our own numbers fooled us three times:

  1. A test harness printed results for four scoring versions while measuring a ranker no user was served.
  2. Four in five rows in our served lists shared their score with another row, so retrieval order, not the score, decided their place.
  3. Giving the ranker 500 candidate jobs instead of 300 made the top 100 worse.

How the matcher works

CV ──► parsed profile (titles, skills, years, German level)
    │
    ├─► dense leg:   embedding, pgvector cosine   (top 300)
    ├─► keyword leg: Postgres full-text search    (top 100)
    └─► domain leg:  jobs in the same tech stack  (top 150)
          │
          ▼
    reciprocal rank fusion (k = 60) → 300 candidates
          │
          ▼
    re-rank: (cosine + capped skill bonus + small boosts,
              clamped at 1.0)
             × German-level and seniority multipliers
             × your own 👍/👎, if any
             sort key adds a small recency lift
          │
          ▼
    diversity: max 3 per employer (backfilled),
               repeated ads collapsed
          │
          ▼
    the list you see
Enter fullscreen mode Exit fullscreen mode

We fuse by rank because cosine similarity and Postgres's ts_rank_cd can't be compared on one scale. Fusion only decides which 300 rows get scored. After that, its order is supposed to be thrown away.

The re-ranker is where the product's opinions live. Cosine similarity says "this ad is about the same things as this CV". It doesn't know that the ad asks for fluent German and the reader has A2, or that the reader holds 11 of the 12 tools it lists. The re-ranker does.

The keyword leg came from a measurement. On 2 July 2026, for 4 test personas, about 14% of the jobs our judge rated relevant were missing from the embedding's top 150. A simple keyword match on 4 of the persona's skills found them, and for one persona the figure was 23%. No re-ranker weight can rescue a job that never enters the pool. Our bet on why: embeddings blur exact identifiers like PROFINET or EN ISO 13849, which cross from an English CV to a German ad unchanged.

Test data before users

CVs built from job ads

eval:synthetic covers every region × tech-stack cell in the corpus:

  1. Pick a real job (the "anchor").
  2. Build a candidate who fits it well, but not perfectly.
  3. Run that candidate through the same embed, retrieve, rank and diversify functions that /api/match calls. It starts from a ready-made profile, so CV parsing isn't under test.
  4. Check where the anchor lands in the 20 jobs a reader would be shown.
// simplified
function profileFromJob(job: Job, rng: () => number) {
  const keepShare = 0.6 + rng() * 0.2;   // 60–80% of must-haves
  return {
    titles: [job.title],
    skills: [
      ...sample(job.requiredSkills, keepShare, rng),
      ...pickUpTo(job.niceToHave, 2, rng),   // 0–2 nice-to-haves
    ],
    years: yearsFor(job.seniority),
    germanLevel: meetsBar(job.germanRequired), // no language penalty
  };
}
Enter fullscreen mode Exit fullscreen mode

A seeded PRNG (mulberry32) makes the candidates repeatable, but the corpus changes nightly. Two same-seed runs three hours apart (23 September 2026) differed by up to 0.0045, so we treat a change under 0.005 as drift.

This is a self-retrieval test. The candidate carries the anchor's exact title and is built to meet its German bar, so mostly it catches a broken pipeline, not a weak ranker. An early baseline (13 June 2026) read recall@5 = 1.000 over 208 anchors. That told us the test was easy, not that the ranker was good.

Personas and an LLM judge

Self-retrieval can't tell you whether the other 19 jobs in a list are any good. For that we wrote 14 synthetic personas: an embedded C++ engineer, a Berlin full-stack developer, an IC/ASIC designer, someone pivoting careers, and others.

For each persona we build a judged pool from the embedding's top 150 plus 100 keyword hits. Gemini 2.5 Flash, at temperature 0, judges every job in it on one question: would this person be a credible applicant? It sees the persona's titles, years, skills and German level, plus the ad's title, skills, language fields and first 900 characters. It returns a verdict, a confidence and a written reason, and a job counts as relevant only at confidence ≥ 0.6. It isn't given posting date, salary or visa-related fields, which the ranker does weigh. We haven't measured how often it agrees with human raters.

Later came golden cases: fixed pools of jobs with relevance labels, some written by hand and some exported from 👍/👎 ratings. npm run eval re-ranks them offline and reports P@5, nDCG@10 and MRR.

Metric Definition Blind to
Anchor recall@5 Share of anchors in the top 5 of the 20 served Whether the other 19 jobs are any good
MRR Mean of 1/rank of the anchor; 0 if it isn't in the 20 served The same
Precision@k Share of the top k the judge rated relevant Everything the judge isn't given
Pool recall Share of judged-relevant jobs that reach the 300-job pool Jobs retrieval never returned. The labels come from retrieval's own results, so it reads high

Mistake 1: copies of the ranker drift

Two harnesses were meant to rank exactly like production. Neither did.

  • The golden-case eval, until 13 August 2026, scored without the production weights and without the candidate's saved preferences. Dropping the weights moves the aggregate, and the gate catches it. Dropping the preferences moved nothing: P@5, nDCG@10 and MRR were identical, and the gate printed ✓ no aggregate regression.
  • The synthetic harness. Production moved to a 300-job pool and the employer cap on 2 July 2026, and to tuned weights on 3 July. Until 10 October, the harness kept a 100-row pool, scored without weights and skipped the diversity pass. It printed numbers for four scoring versions, and no change to a weighted rule could move them.

The same drift happened outside the evals. Two implementations of "which tech stack is this job?" disagreed on a third of live jobs (10 October 2026). The home page's own search could reach only about half of a reader's results-page top 8 (9 October).

A copied ranker doesn't make an eval fail. It makes it blind, and nothing in the output says so.

The fix:

  1. One ranking function. Every page, email and eval ranks through the same rankPool().
  2. Fixtures that only one rule can reorder. Every row carries the same skills and language fields, so dropping a rule changes the order and fails the test, where the aggregate gate saw nothing. The synthetic harness also gets a structural check: the test parses it with the TypeScript compiler and fails if it imports the scorer or the diversity pass itself. A text search would pass on a comment.
  3. Stamped baselines. Every baseline records what it was measured under, and the harness refuses to compare across a mismatch:
// simplified
const STAMPS = [
  "scoringVersion", "retrievalArm", "seed", "pool",
  "servedLimit", "cluster", "perDomainCap",
] as const;

function comparable(baseline: Stamps, run: Stamps) {
  const differs = STAMPS.filter(
    (k) => baseline[k] === undefined || baseline[k] !== run[k],
  );
  return differs.length === 0
    ? { ok: true }
    : { ok: false, differs }; // prints "deltas suppressed"
}
Enter fullscreen mode Exit fullscreen mode

A missing stamp counts as a mismatch. Once the harness was fixed, this check refused to compare the new baseline with the old one, because seven scorer versions had landed in between. The 18 October 2026 run (recall@5 0.970, MRR 0.932) became a fresh start. The cost wasn't a number. It was three months of numbers about a system no one was served.

Mistake 2: four in five rows were tied

On 9 October 2026 we read back the lists we had served over the previous 90 days. 81.5% of served rows shared their score with another row in the same list, and so did 54.7% of rows in the top 5.

The score was rounded to two decimals inside the scoring step, then sorted with a stable sort, so tied rows kept their input order: retrieval-fusion order. In 28% of adjacent tied pairs, the lower-ranked row had the higher cosine. Removing the rounding (v10) made exact ties rare but not impossible. Arrival order still broke them, and the home page was about to re-rank the same pool in a different arrival order. So v12 made the order total.

// before (v9): rounded during scoring, then a stable sort
row.score = Math.round(raw * 100) / 100;
rows.sort((a, b) => b.score - a.score);   // ties keep input order

// after (v10, v12): full precision, then a total order
rows.sort((a, b) =>
  b.rankScore - a.rankScore ||   // score + recency lift (v15+)
  b.cosine - a.cosine ||
  (a.id < b.id ? -1 : a.id > b.id ? 1 : 0), // code units
);
// round only for display
Enter fullscreen mode Exit fullscreen mode

The tie-break avoids localeCompare on purpose, because a locale-dependent order is another hidden input. Both fixes were about determinism, not relevance (for v12, the golden eval didn't move). The list is now sorted by the number we say it's sorted by.

Mistake 3: more recall, worse lists

This happened twice.

July 2026. The keyword leg raised pool recall, but recall@20 after re-ranking stayed flat. The cosine-anchored re-ranker re-buried what the keyword leg had rescued.

November 2026. It seemed obvious that 500 candidates would beat 300. On 13 November we tested it on 8 of our 14 personas: the relabelling hit its $3 spend cap, and the other 6 had stale labels. The judge rated every job in the re-ranked top 100 at both pool sizes, so no unjudged row counted as a miss.

(mean of 8 personas, 13 Nov 2026) Pool 300 Pool 500
Pool recall 0.852 0.930
Precision@20 after re-ranking 0.638 0.625
Precision@100 after re-ranking 0.546 0.510

The top 20 barely moved. The top 100 got worse: the 169 rows that entered at pool 500 were 31% relevant, and the 169 they pushed out were 48% relevant. We kept 300.

Limits: 8 personas is small, and the judge isn't given recency, salary or visa-related fields. But two earlier, cheaper methods pointed the same way. One was a sampled replay on 10 November, where jobs new to the top 100 were 37% relevant against 39% for the positions they competed with: too close to call. Judging every row removed the bias from unjudged rows. The study cost $3.68, about 80% of it thinking tokens. Gemini 2.5 Flash thinks by default unless you set a thinking budget.

Both times, the bottleneck was ranking inside the pool, not the size of the pool.

What the metrics didn't catch

Every scoring change bumps SCORING_VERSION, and every baseline records it. On 20 November 2026 we're on v21. Some of the most useful versions came from people reading the output, the code or the logs:

  • v15: in July we had left recency out, partly because a test showed it cost 0.048 nDCG@20. That test dated ads by when we first saw them, not by the employer's posting date. Since v15, a small lift on the employer's date affects the order only.
  • v17: an ad's "RTOS (FreeRTOS)" counted as missing for a CV listing FreeRTOS, while the requirement list on the same page said "Shown in your CV".
  • v19: while recording a demo, we saw the learning path tell a CV listing Embedded Linux and Yocto to "start with Linux". Now a specific tool satisfies the general one.

What we'd do differently

  1. Make every eval import the production ranker, and guard it with fixtures only the rule under test can reorder. An aggregate gate won't notice a rule the golden set never exercises.
  2. Stamp everything you measure, and refuse to compare across a mismatch, even when that costs you your history.
  3. A perfect self-retrieval score means the test is easy.
  4. Never round a value you sort on, and make the order total.
  5. Judge every row in the slice you decide on. Samples gave us "too close to call". Full labels gave us a decision.

Here's what the matcher looks like from the reader's side, from uploading a CV to applying (97 seconds):

A question for anyone who has done it: if you've checked an LLM relevance judge against human raters, how large a sample did you need before you trusted it?

If you're an engineer looking at jobs in Germany, kandidate.ai matches your CV, in English or German, and ranks the results for your German level. There's a free plan: kandidate.ai.

Top comments (0)