The biggest improvement was learning when to stay quiet.
This is our experience while integrating Jev to a use case that's both cost effective and accurate when it comes to llm. The hard part wasn't getting AI to say something smart. It was getting it to stay quiet when it wasn't sure.
Your resume score is probably lying to you.
Nobody needs another tool that gives a resume 62 out of 100 and then says:
Strengthen your bullet points.
I know because that's roughly what our own reviewer used to do.
The fix wasn't a better prompt.
It was changing what we asked the model to do — and, more importantly, what we allowed it to say.
The reviewer is now built around Jev, a decision model designed around structured judgments rather than generating another paragraph of critique.
Instead of asking it to write advice, we ask it narrower questions and get probabilities back.
That distinction turned out to matter a lot more than I expected.
The honest version of this story isn't:
We added AI to our resume builder.
It's that we shipped a reviewer, watched people bounce off it, and realised that the difficult part of AI review isn't getting a model to say something intelligent.
It's getting it to stay quiet until it has earned the right to speak.
That's probably the one idea from this project I'd keep if I had to throw everything else away.
The problem with asking a model to "review my resume"
The obvious implementation looks something like this:
resume
↓
large prompt
↓
LLM
↓
long critique
Paste in the resume.
Ask for feedback.
Render whatever comes back.
A lot of AI review tools follow some version of this pattern, and it creates a few problems.
1. Confidence becomes invisible
Ask a model for prose and you get prose with roughly the same authoritative tone throughout.
A 60% hunch and a 99% certainty can sound almost identical.
The person reading it has no idea which advice actually matters.
So they either trust everything or start trusting nothing.
2. Holistic judgment is difficult to test
Consider:
Rate this resume against what a hiring manager looks for.
That's one enormous question.
What's the correct answer?
There isn't really one.
Run the same resume twice and the review may move around.
If the result changes, what do you tune?
Which part failed?
What test should fail?
You don't really know.
3. Models really want to be helpful
Ask a model to critique something and it will generally find something to critique.
Give it a bullet like:
Responsible for maintaining the product workflow.
and a sufficiently helpful model may suggest turning it into something like:
Improved the product workflow, increasing efficiency by 25%.
Where did 25% come from?
Nowhere.
It sounds better.
It is also fiction.
That's a particularly nasty failure mode when the output eventually ends up in a document someone sends to an employer.
4. A vague truth is still useless
Your bullets under the Product role could be stronger.
Maybe that's true.
But it's a weather report.
The user still has to figure out:
- Which bullet?
- What's wrong with it?
- What should change?
- Is this a real issue or generic advice?
The machine has technically provided feedback while leaving the difficult part to the human.
The failure, I think, is structural.
We gave the model both the job of judging and the job of writing.
For this product, most of the writing turned out not to need a model at all.
The reframe: Jev decides, the code speaks
The biggest change was surprisingly simple:
Stop letting the model write the feedback the user reads.
The old shape was:
Resume
↓
LLM
↓
One long generated critique
The new shape is closer to:
Resume
↓
Deterministic checks + bounded judgments
↓
Evidence + uncertainty
↓
Decide whether anything is worth surfacing
↓
Point at the exact editable line
The flow today is roughly this.
A parser turns the PDF or DOCX into structured resume data.
Normal code handles anything that doesn't require intelligence:
- page count
- missing sections
- duplicated lines
- repeated bullet openers
- date calculations
- structural problems
- other deterministic checks
Then the redacted resume text goes to Jev.
Instead of asking:
Is this a good resume?
we ask smaller questions about particular pieces of text.
Jev returns structured judgments rather than writing the explanation itself.
Application code then decides whether those signals add up to something worth telling a person.
If they do, the system attaches an explanation that was written beforehand.
The UI points to the relevant line.
The person can act on it.
That's it.
Because the model gives us decisions rather than paragraphs, we can have a rule for when the system should say something at all.
That's the whole game.
Four decisions that did most of the work
1. Break broad questions apart
We stopped asking things like:
Is this a good bullet?
That's almost impossible to calibrate.
Instead, the problem gets broken into smaller judgments.
For example:
- Does the line mostly describe a responsibility?
- Does it provide any sense of scale?
- Can a reader tell what the person actually did?
The important part isn't those particular questions.
It's the decomposition.
Once the judgments are structured, ordinary code can decide whether there's enough evidence to surface something.
That part isn't AI.
It's just logic.
And that's exactly why it's useful.
You can write tests for it.
You can watch it fail.
You can change the rule.
You can compare the result before and after.
You can test narrow questions. Broad opinions are much harder to test.
2. The model doesn't invent the evidence
Every finding the user sees is built from roughly three pieces:
- an explanation written beforehand
- guidance written beforehand
- the user's own words, quoted from the resume
The model doesn't invent metrics.
It doesn't manufacture an example accomplishment.
It doesn't decide that you somehow improved revenue by 31%.
It contributes a signal.
That's it.
This turned out to be one of the simplest anti-hallucination techniques in the whole system.
You can put:
Do not invent metrics.
inside a prompt.
That's a request.
Or you can design the system so the model cannot put invented prose in front of the user in the first place.
That's a constraint.
I trust the second one more.
3. Silence is a real output
This took me a while to appreciate.
If the signals don't agree strongly enough, we don't show anything.
No card.
No weak suggestion.
No:
This could potentially maybe benefit from...
Nothing.
At first this feels wrong.
You built an AI reviewer. Surely it should review things.
But a system that always produces feedback has a strange incentive: it has to find something wrong even when there isn't much worth saying.
So we deliberately made silence meaningful.
If the system is uncertain, the user sees less.
When it is confident enough to surface something, we show the lines that seem most worth looking at first.
The goal isn't:
Here are all the problems with your resume.
It's closer to:
If you're going to fix one thing, start here.
A strong resume should produce less criticism.
That shouldn't be considered a failure of the reviewer.
It should be considered success.
4. We killed the middle state
The first version had three outcomes:
Passed
Unclear
Could improve
"Unclear" seemed intellectually honest.
It was awful.
Users didn't read it as:
The model is responsibly communicating uncertainty.
They read it as:
The tool doesn't know what it's doing.
Worse, once they saw enough uncertain rows, they started distrusting the confident ones too.
So we removed it from the UI.
Now a visible result is generally either:
Passed
or:
Could improve
If the evaluator can't make a sufficiently useful call, the row doesn't need to exist.
The uncertainty still exists internally.
We just don't turn uncertainty into useless UI.
A calibration bug that had nothing to do with resumes
This was probably my favourite bug in the project.
We were evaluating one part of a resume on an ordered scale.
Strong examples were occasionally being flagged while some weaker ones were getting through.
My first assumption was obvious:
The questions must be badly written.
They weren't.
The mistake was in how we interpreted confidence.
Suppose the possible result is an ordered scale:
1 2 3 4 5
A model can be uncertain between level 2 and level 3.
That sounds like low confidence.
But imagine that the decision your application actually cares about is here:
1 2 3 | 4 5
^
boundary
If almost all of the probability is split between 2 and 3, does it really matter that the model can't decide which exact level it is?
For the application-level decision, not much.
Both answers are on the same side of the boundary.
So we stopped thinking only in terms of:
How confident is the model in the winning answer?
and started thinking more about:
Where does the probability mass sit relative to the decision we actually care about?
Those are not the same question.
That sounds obvious after you've found the bug.
It wasn't obvious before.
If you're building anything around graded model output, this distinction is worth remembering:
A confidence field is not necessarily a verdict.
Sometimes the distribution tells you far more than the winning label.
Privacy wasn't something I wanted to bolt on later
A resume reviewer has an uncomfortable property:
It needs to read fairly personal text.
But it doesn't need all of it.
The reviewer needs to see your sentences.
It doesn't need to know your name.
Before evaluation, we create a separate sanitized representation with identifying information removed.
Things like:
- name
- contact details
- location
- profile links
The original is what the interface uses when it needs to show the user their own line or edit it.
The evaluator works against the redacted representation.
That separation also made another feature easier to reason about: rechecking.
When somebody comes back after fixing a finding, the question should be:
Did I fix the thing you told me was wrong?
Not:
Can you inspect my resume again and discover another twelve things to complain about?
So the recheck is deliberately scoped to the previous findings.
It can say the issue appears fixed.
It can say it isn't fixed.
It can say it looks partially fixed.
It can admit that it can't tell.
What it shouldn't do is turn every revision into another endless round of criticism.
What we got wrong
This is probably the more useful part.
The first version was basically a dashboard
It reported broad areas of resume quality.
Impact.
Clarity.
Structure.
That looked nice.
It also wasn't very actionable.
Telling somebody:
Your experience section could demonstrate more impact.
doesn't solve much.
Which sentence?
Why?
What do I change?
The biggest product improvement wasn't a better model.
It was changing the unit of feedback.
From:
Something in this section is weak.
to:
This specific line is worth looking at.
The model wasn't necessarily the problem. The unit of feedback was.
We judged summaries without enough context
This produced some wonderfully wrong results.
A resume summary might say that somebody led a certain type of project or worked across a particular domain.
If you evaluate that sentence alone, those claims can look unsupported.
And the evaluator may correctly notice that.
The problem is that the evidence is sitting three inches lower on the page.
Once the summary was evaluated with the relevant experience available as context, a lot of those weird results disappeared.
The lesson wasn't:
The model needs to be smarter.
It was:
We didn't give the decision enough evidence.
There's a difference.
Career gaps are still uncomfortable territory
A six-month contract, a parental leave, time spent caring for somebody, unemployment and a deliberate break can all create superficially similar timeline patterns.
Text alone doesn't always tell you what happened.
This is exactly the kind of area where being "helpful" can become harmful quickly.
So the system is deliberately conservative here.
It shouldn't tell somebody that a gap itself is a problem.
It definitely shouldn't tell them to invent experience to fill it.
At most, there are situations where a short and honest line of context may make something easier for a human reader to understand.
I still consider this an area where false positives matter more than coverage.
We haven't finished evaluating it
This is the boring but important answer.
We're still going through representative resumes and recording:
- false positives
- missed findings
- cases where the evaluator is uncertain
- places where our own rules behave badly
The thresholds aren't sacred constants delivered from a mountain.
They're informed guesses backed by tests, and they'll move as we learn more.
I'd rather say that than give the system a level of scientific authority it hasn't earned.
Five things I'd keep from the experiment
If I were building another system like this tomorrow:
1. Ask the model to judge before asking it to write
Generation isn't automatically the valuable part.
Sometimes the useful intelligence is the decision.
2. Break vague judgments into smaller ones
Smaller decisions are easier to inspect, test and combine.
3. Let "nothing" be a valid result
A system doesn't become smarter by always having an answer.
4. Ground criticism in something editable
If you can't point to what the person can actually change, think carefully about whether the criticism is useful.
5. Treat uncertainty as data
Don't automatically translate uncertainty into a paragraph full of hedge words.
Use it.
The system described here is the reviewer behind FreeResume's "What's Wrong With My Resume" tool.
I'm much more interested in the cases where it gets something wrong than the ones where it gets something obviously right.
Because if you can identify exactly why a decision was wrong, you have something you can improve.
And that's ultimately the difference I care about.
A model that always has an answer is impressive.
A system that knows when the answer isn't good enough to show is useful.
Top comments (1)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support