DEV Community: John Zacharia

Evaluating a Recruiting Platform Like an Engineer

John Zacharia — Mon, 21 Sep 2026 18:01:51 +0000

Most ATS buying guides are written for HR. This one is for whoever on the team can read an API doc - because the questions that actually separate recruiting platforms are the same ones you’d ask of any system you’re going to depend on for three to five years.
HR sees "40+ features." An engineer sees a workflow engine, a data layer, and a UI - and knows the checkbox lies about all three. So evaluate it the way you’d evaluate any vendor SaaS.

1. The workflow engine: is it configurable or is it a services contract?

Every ATS claims "customizable workflows." The real question is whether you can model a new hiring stage, approval, or role-family variation - or whether every change is a ticket to the vendor’s professional-services team.
Test it live: configure your weirdest workflow during the demo. If it needs a services engagement once, it needs one every time. That’s not configuration; that’s a dependency you don’t control.

2. Automation: where the hours actually live

Stage-triggered emails, scheduling links, task assignment, SLA nudges. This is the ATS equivalent of CI: nobody notices it until it’s missing, and then everyone loses hours a week to manual work.
Test: build a rule yourself in the demo - "when a candidate enters stage X, send email Y and create task Z." If you can’t construct it without help, your recruiters won’t either.

3. Integrations: read the docs, not the logo wall

A partner logo on a marketing page is not an integration. What you need to know is the same thing you’d check for any third-party API:
•Is it native and supported, or a brittle Zapier hop?
•Is it bidirectional, or does data only flow one way?
•What are the rate limits and sync frequencies?
•Is there real documentation, or a "contact us" button?
Ask for the integration docs and a reference customer running your exact stack (HRIS + calendar + assessment tools). Vague answers here are a hard signal.

4. The data layer: reporting and export

Two questions decide this. First: can you reproduce your five core hiring metrics - funnel conversion, time-to-fill, source effectiveness - inside the tool, on realistic data, without exporting to a spreadsheet? This single test eliminates more vendors than any other.
Second, and engineers always forget to ask this until it’s too late: what does data export cost at the end of the contract? Negotiate the exit before the entrance. Export format, cost, and timeline should be contract clauses signed at purchase, when your leverage is highest - not a five-figure surprise delivered as an unusable dump on your way out.

The evaluation method: weight by pain, score from tests

Don’t score vendors on feature count. Do this instead:
1.List your five biggest hiring pains.
2.Map each to the feature depth that fixes it.
3.Weight by severity (worst pain’s features 5x, mildest 1x).
4.Score each vendor 1-5 per weighted row - from live tests only, never the proposal.
5.Multiply, sum, rank.
It’s the same discipline as writing acceptance tests before you trust a dependency: you define what "works" means, then you verify it yourself.
The full breakdown - all eight decisive features, the scorecard template, and the post-go-live metrics to track - is here: this weighted scorecard method.
If your team is evaluating platforms, the checklist that separates depth from checkbox theater lives in hiremore AI’s ATS guide.

Inside the Resume Parsing Pipeline: Where Extraction Breaks

John Zacharia — Sat, 19 Sep 2026 13:04:52 +0000

If you’re building or evaluating an ATS integration, resume parsing looks simple from the outside and gets genuinely interesting once you look at where it fails. Here’s the pipeline, stage by stage, and the specific failure mode at each step.

Stage 1: Text extraction.

Digital documents (real PDF/DOCX with selectable text) yield a text stream directly. Scanned or photographed documents route through OCR. This is where layout does the most damage before you’ve even started tagging entities: a two-column resume can interleave unrelated text mid-sentence if the extractor reads left-to-right across the full page width instead of column-by-column, and tables scramble reading order in a similar way. Layout-aware extraction (detecting column and table boundaries before running text extraction) is the standard mitigation, and it’s the single highest-leverage fix for this whole pipeline.

Stage 2: Section detection and entity recognition.

The extracted text gets segmented into sections (experience, education, skills) and tagged: employer names, job titles, date ranges, degrees, certifications. This is where LLM-based parsers meaningfully outperform the regex-and-dictionary approach that dominated a decade ago. Older parsers choke on unconventional phrasing (“Led the thing that shipped the thing” as a job title, unfortunately real). LLM-based entity recognition handles semantic variation far better, at the cost of a new failure mode covered below.

Stage 3: Normalization and profile assembly.

Tagged entities get standardized: dates into one format, titles mapped to a taxonomy, skills deduplicated against a skills ontology. This is the stage that should route low-confidence fields to a human review queue instead of guessing. “Flag, don’t guess” isn’t a nice-to-have design principle here, it’s the difference between a visible 2-minute fix and an invisible data corruption that silently poisons every downstream feature: search, ranking, and analytics.

The failure mode worth architecting around specifically:

confident LLM hallucination. LLM-based parsers handle messy, non-standard input brilliantly, and occasionally fabricate a plausible-looking field that isn’t actually on the page, a tidy job title synthesized from surrounding context rather than extracted from text. This is more dangerous than a garbled OCR output precisely because it looks clean. Mitigations that actually work: ground every extraction with a source-position reference (so you can trace any field back to its exact location in the document), run spot audits comparing extracted fields against source text on a regular sample, and surface confidence scores in the reviewer UI rather than hiding them.

On accuracy benchmarks:

vendor claims of “99% accuracy” measured on a clean internal test corpus tell you almost nothing about your production traffic. If your applicant pool includes a meaningful share of scans, two-column CVs, or non-Western date formats, test on your own 100-document sample before trusting any published number. Field-level accuracy on your real mix is the only number that matters, and it’s cheap to measure: parse the sample, audit against source documents, and compute correct-fields-over-total-fields per field class.

On downstream compliance:

if parsed fields feed any automated knockout rule (years of experience, certification checks), gate that rule behind a confidence threshold. An auto-rejection caused by a parsing error is legally indistinguishable from a screening decision, and “the parser misread the date” is not a defense anyone wants to test in front of a regulator.
The rest of the failure modes, plus real accuracy deltas from production case studies (one team saw field accuracy jump from 71% to 91% just from switching parsers and adding flag-don’t-guess normalization), are in the full guide.

Evaluating ML-Based Hiring Tools: An Engineer's Checklist

John Zacharia — Wed, 16 Sep 2026 13:05:43 +0000

If your company is buying an ML-based hiring tool, there is a decent chance someone will forward the technical due-diligence to an engineer. This is a checklist for that engineer. The business folks will evaluate the demo; your job is to evaluate the model, the data path, and the integration surface — the parts a demo is specifically designed to hide.

1. Explainability is an architecture question, not a UI feature

The first thing to establish: can the system produce a per-decision explanation, and is that explanation faithful to the model, or is it a post-hoc rationalization bolted on for the UI?
Ask what method generates the "why this candidate ranked here" output. If the answer is a real feature-attribution approach tied to defined criteria, good. If the answer is vague — "the model considers many factors" — you are likely looking at a black box with a narrative layer. A useful probe: ask them to show two candidates who ranked closely and explain the delta. Faithful explanations produce a crisp, criteria-linked difference. Rationalizations get mushy.
Why you care: a keyword matcher with a neural-net press release cannot explain rankings in criteria terms, because there is no model to explain. The explainability question is the fastest way to expose capability theater.

2. Training data provenance and the bias question

Ask what the model was trained on. Models trained on a company's historical hires learn to replicate historical hiring patterns — including whatever bias was in them. "The model is bias-free by design" is a sentence no serious ML practitioner says; bias is measured, not designed away.
What you want to see:
•An independent bias audit (not a self-assessment), reasonably recent.
•Reported four-fifths / adverse-impact ratios (lowest group pass rate ÷ highest; ≥ 0.80 is the conventional threshold).
•A clear statement of whether the tool scores against criteria you define or patterns it infers from your past hires. Prefer the former; the latter quietly automates the status quo.
This is not academic. Under NYC Local Law 144 the audit is legally required, and the EU AI Act puts hiring models in its high-risk tier. The compliance obligation sits with the employer, so your engineering sign-off has legal weight.

3. Integration is where these tools go to die

The most common fate of a recruiting tool is shelfware, and the most common cause is integration that looked fine on a partners page and fell apart in production. "API available" is not integration.
Check for:
•Bidirectional sync with your specific ATS version, documented, ideally with a reference customer on the same stack.
•Webhook / event support vs. polling, and rate limits that survive your actual application volume.
•Where the source of truth lives when the two systems disagree.
Weight this heavily. One healthcare staffing firm re-ran a selection with integration weighted double after their previous tool never synced; adoption went from 25% to 85% at 90 days.

4. Data portability — negotiate it before you sign

Your leverage is highest before the contract exists and drops to near zero afterward. Get, in writing:
•Full export of candidate data, scores, and audit logs in standard formats.
•No per-export fees, no proprietary-format lock-in.
•A defined data-handoff on exit.
Lock-in economics are how a mediocre tool becomes a three-year hostage situation. Contract for the exit while you can.

5. Make the pilot the only accuracy benchmark

Ignore vendor accuracy numbers. "95% accurate" against an undefined benchmark is unfalsifiable. Define your own measure before the pilot: agreement rate between the tool's rankings and your best recruiters' judgments, on your candidates, on live roles, run in parallel with the current process for ~30 days. Track agreement rate, time saved, override rate, and pass-rate stability across groups.

The checklist, condensed

•Per-decision explanations are faithful and criteria-linked
•Training-data provenance disclosed
•Independent bias audit with four-fifths ratios
•Scores against defined criteria, not inferred patterns
•Documented bidirectional ATS integration on your version
•Contractual data + audit-log export, standard formats, no fees
•30-day parallel pilot with a pre-defined success metric

If you want the business-side version of this — the 7 vendor questions framed for a procurement call rather than a code review — it pairs well with the checklist above.

I work on a recruiting AI platform, and my bias is toward tools that can survive this checklist. If you want to see one built for it, here is how we approach explainability, auditing, and export

ATS Integration Architecture: What to Map Before You Sign

John Zacharia — Wed, 02 Sep 2026 12:54:11 +0000

Most guides on choosing an applicant tracking system talk about features. This one is about the layer that actually determines whether the platform survives contact with your stack: integrations.

If you're the technical person pulled into an ATS evaluation, this is the part your non-technical colleagues will underestimate — and the part that generates the most post-contract pain. Here's how to map it properly.

1. Inventory every data flow, not every tool

Don't list “tools.” List directional data flows. For each system, write down what data moves, which direction, and how often:
●HRIS — candidate → employee record handoff on hire (bidirectional ideally)
●Payroll — new-hire data push (one-way, event-triggered)
●Background screening — order + status callback (bidirectional, webhook-driven)
●Calendar (Google/Microsoft) — interview scheduling + availability sync
●Video interview tools — scheduling links out, recordings/scores back
●Job boards / distribution — posting syndication out, application ingestion back
The “direction + trigger” framing exposes gaps that a flat tool list hides.

2. Classify each integration by mechanism

Not all “integrations” are equal. Push the vendor to tell you which of these each connection actually uses:
●Native pre-built integration — maintained by the vendor, lowest overhead
●Public REST API — you (or middleware) build and maintain it
●iPaaS / middleware (Workato, Merge.dev, etc.) — flexible, adds cost + a dependency
●Flat-file / SFTP batch — a red flag in 2025 for anything real-time
●“On the roadmap” — treat as does not exist
A “yes, we integrate with X” that turns out to be a nightly CSV export is a very different thing from a webhook-driven bidirectional sync.

3. Interrogate the API itself

If any integration will run through the public API, evaluate it like you'd evaluate any dependency:
●Is it REST/GraphQL, documented, and versioned?
●Rate limits — do they survive a high-volume hiring event?
●Webhooks for state changes, or are you stuck polling?
●Auth model (OAuth 2.0 vs. long-lived API keys)?
●Sandbox environment for testing before go-live?
●Data model for candidates — how are duplicates and merges handled?

4. Make compatibility a contractual gate

The most expensive integration gaps are the ones discovered after signing. Build a formal integration checklist and require vendor sign-off on compatibility before finalizing terms. A vendor's response to your integration map tells you as much about their support quality as their docs do.

The business context

This technical mapping isn't a side quest — it's stage two of a five-stage selection framework (requirements → integrations → scored shortlist → structured pilot → ROI). The full non-technical playbook, including the weighted vendor scoring matrix and implementation guidance, is here:

👉 How to Choose the Right ATS for Your Hiring Team

If you've survived an ATS migration, what integration surprised you post-contract? Drop it in the comments — the failure modes are weirdly consistent across teams.