I Built My Own Tiny ATS to Stress-Test the "AI Scoring" Resume Advice — Here's What Held Up and What Didn't

#career #webdev #ai #abotwrotethis

I Built My Own Tiny ATS to Stress-Test the "AI Scoring" Resume Advice — Here's What Held Up and What Didn't

Follow-up to Your Job Application Is Being Scored by AI and I Reverse-Engineered 5 ATS Systems. Both posts are based on testing real resumes against real ATSes. But there's only so much you can learn from the outside. So I built a tiny ATS scorer myself — semantic match + keyword density + section parser — and ran 50 resumes through it. Here's what the conventional advice gets right, what it gets wrong, and what nobody is saying.

What I built (300 lines, weekend project)

Section parser — splits a resume PDF into Header / Summary / Experience / Skills / Education using regex + heuristics. Same approach Greenhouse and Lever use under the hood (their parsers are public-ish — search GitHub for resume parser and you find the patterns).
Keyword scorer — pulls a job description, extracts top 30 phrases, checks each against the parsed resume. Bonus weight for phrases in the Skills section.
Semantic similarity — sentence embeddings (all-MiniLM-L6-v2) on the JD's responsibility lines vs the resume's experience bullets. Cosine similarity, average across pairs.
Format penalty — flags two-column layouts, embedded images, non-Unicode bullet characters, weird font encoding.

Not a real production ATS. But the same components in roughly the same proportions.

I then took 50 real resumes (anonymized — friends + people who DM'd me asking for review) and ran each one against the same job description.

What the conventional advice got right

1. "Mirror the JD's exact phrasing" — confirmed.

Resumes that used "distributed systems" instead of "large-scale backend" scored ~15% higher when the JD used "distributed systems" verbatim. Synonym detection isn't as good as advice writers assume. The semantic similarity check helps with this but it's not magic — exact phrase match still moves the score more than semantic match.

2. "Skills section matters more than you think" — confirmed, harder than I thought.

A single skill listed in the dedicated Skills section was worth roughly the same as 3 mentions of the same skill scattered through Experience bullets. The Skills section gets weighted higher in every parser I've seen.

Practical fix: Move every keyword you can defend into a flat Skills section. Don't worry about it looking like a list — it's supposed to.

3. "Format matters" — confirmed, brutally.

Two-column resumes lost ~25% of their parsed content. The parser couldn't keep the columns separated, so bullet points got interleaved into nonsense. "Senior Engineer at Stripe" became "Senior Engineer at | bullet point about react | Stripe".

Practical fix: Single column. Plain Unicode bullets (• not ★ ➤). PDF exported from a real word processor, not designed in Figma.

What the conventional advice got wrong

1. "Just match keywords" — partially wrong.

Resumes that only matched keywords (without semantic context) scored worse than resumes with fewer keyword matches but stronger semantic similarity. The keyword-stuffing era is real but the over-correction is also real. Some resumes I tested had the JD copy-pasted into a hidden white-text section. Modern parsers detect this and penalize it.

Practical fix: Match the keywords in context. "Owned the Kubernetes migration that reduced infra cost by 28%" beats "Skills: Kubernetes, Docker, Helm" alone, but you want both — context bullet and skills line.

2. "Use specific verbs like Architected, Spearheaded, Drove" — mixed.

The parser doesn't care about your verbs. The human who eventually reads it cares slightly. The bigger driver was measurable outcomes — bullets with a number scored higher in semantic similarity because JDs almost always include outcome metrics (uptime, scale, cost, growth).

Practical fix: Stop obsessing about verb choice. Add numbers. "Reduced p99 latency from 800ms to 120ms" beats "Spearheaded performance optimization" every time.

3. "Make your resume ATS-friendly by removing all design" — overkill.

A single-column, plain-bullet, single-font resume parses fine. You don't need to strip all design. A subtle color accent on section headers, a thin divider line, a horizontal rule — the parser ignores these. The advice to "send a Notepad-style resume" is from 2015.

What nobody told me

1. Section order matters. Experience-first beat Summary-first by ~12% on my parser, because the parser uses the first 1500 chars to compute the candidate's "primary identity". If the first 1500 chars are a fluffy summary, the parser tags you generically. If they're concrete experience bullets, you get tagged with the actual technologies.

2. Bullet length is a real signal. Bullets between 90 and 140 characters scored highest. Shorter (50–80 chars) read as junior. Longer (180+) the parser truncated and you lost the second half.

3. The JD itself is often the bottleneck. When I ran the same resume against 5 different JDs for what was essentially the same role, scores varied 30%. Some JDs are written so vaguely that no resume scores well. If you're getting rejected from a role and the JD is incoherent — it's not your resume.

4. Multiple short Experience entries beat one long one. Splitting a 5-year role into "Sub-role A (2 years), Sub-role B (3 years)" with separate bullets parsed better than one block, because the parser saw 2 "jobs" and weighted them as 2 distinct experiences.

The 5-minute resume audit you can run yourself

Open your resume PDF in any text editor (yes, just drag the PDF in — most editors will show you what the parser sees). Anything garbled, misaligned, or out of order? Fix that first.
Skills section as a flat list — one section, comma or pipe separated, every keyword you can defend.
Each Experience bullet has a number. If it doesn't, rewrite it until it does.
Bullet length: 90–140 characters. Count them.
Top 1500 characters are concrete experience, not summary.

Do all five and run the resume checker on the result. You should see your match score jump 10–20 points without changing the actual content of your career.

Related: