DEV Community

VANSH ARORA
VANSH ARORA

Posted on

Re-Ranking Context with Explicit Feedback, Without Letting It Take Over

The previous post covered TokenCap's local learning ledger: a file of explicit judgments about context packs. This one covers what reads it. tokencap ask "<task>" --adaptive uses that evidence to re-rank the files retrieval found. It is opt-in, it is bounded, and it explains itself.

Running it

tokencap ask "invoice rounding" --adaptive
Enter fullscreen mode Exit fullscreen mode

Before the usual task-context summary, the CLI prints one extra line. It says whether matching evidence was found and how many events were used, and it confirms that all baseline candidates were preserved. With no matching feedback, it says "no matching explicit evidence" and no file is boosted.

Which feedback counts

The ranking code lives in src/learning/adaptiveRank.js. A ledger event is considered only if its task has the same subject as the current task.

Both task labels are stemmed and stripped of function words and generic task words: fix, bug, add, update, refactor, review, implement, workflow and similar. What remains is the subject. Two tasks match when:

  • one subject is contained in the other ("auth" and "auth flow"), or
  • at least half of their combined subject terms are shared.

A single shared word between two longer tasks is not enough. So "fix the login bug" does not pull in evidence from "fix the payment bug", and "invoice rounding" does match "rounding in invoice totals".

What gets boosted

Two kinds of evidence produce a boost:

  • files selected in a pack you marked helped (+0.75 per matching event)
  • files you reported missed, whatever the outcome (+0.5 per matching event)

Files of neutral or harmful packs are never boosted. The total boost per file is capped at 2. A file with many helpful reports still cannot outrun a much stronger retrieval match by feedback alone.

What it never does

It never adds a file. The boost applies only to files already in the baseline candidate list. A missed file that retrieval did not find at all is not inserted. Feedback is not a back door into the pack.

It never removes one either. Every baseline candidate is still in the result, and the output includes candidatesPreserved. Each re-ranked result also keeps its baselineRank and baselineRelevanceScore, plus an explicit-feedback match reason when a boost applied. Run with --json and you can see exactly what moved and why.

Inspecting the evidence

tokencap analyze --learning
tokencap analyze --patterns
Enter fullscreen mode Exit fullscreen mode

--learning lists recent events and outcome counts. --patterns reports file pairs that appeared together in two or more packs you judged helpful or harmful. Neutral records are left out, including Missed reports from the companion. A pattern is advisory. It is not a dependency edge and not a rule, and nothing acts on it.

Quality gates

Two npm scripts test the feature itself:

npm run eval:adaptive
npm run eval:ledger
Enter fullscreen mode Exit fullscreen mode

eval:adaptive runs a versioned, controlled corpus. For each case it compares baseline and adaptive recall within the top-K window, and the tokens needed to cover every expected file. It fails if adaptive recall drops below baseline on any case, if candidates are not preserved, or if the average token saving is below the corpus minimum.

eval:ledger runs a held-out check on your own repository's feedback, to see whether the boosts hold up on records they were not tuned on. It needs 20 or more records.

The controlled corpus is a regression gate, not a benchmark claim. It tells us the re-ranker did not break recall. It does not tell you how much better your packs will get.

More: tokencap.vansharora.app

Top comments (0)