We Used an AI Agent to Screen 200 Applicants. Here's What We Learned.

We posted a senior engineer role. 200 applications in 48 hours. Our team of 4 could not possibly read all of them carefully.

So we used an AI agent to do the first screen.

What the agent did

For each application:

1. Read the resume/CV and cover letter
2. Checked against our requirements (5+ years experience, Python proficiency, distributed systems background)
3. Scored on a 1-5 scale with written reasoning
4. Categorized: strong match, possible match, or no match
5. For strong matches: drafted a personalized interview invitation
6. For no matches: drafted a respectful rejection

The results

Out of 200 applications:

• 23 scored as strong match (11.5%)
• 41 scored as possible match (20.5%)
• 136 scored as no match (68%)

We manually reviewed all 23 strong matches and a random sample of 20 possible matches. The agent's assessment agreed with ours on 21 of 23 strong matches (91% agreement) and 17 of 20 possible matches (85% agreement).

The 2 strong-match disagreements: one was a career changer with non-traditional experience (agent undervalued, we liked them). One had an impressive resume but our team felt the cover letter showed misalignment (agent overvalued).

Time saved

Manual screening of 200 applications: estimated 20-25 hours (6-8 min each for careful reading).

AI agent screening: 45 minutes total processing time + 3 hours of human review on the top candidates.

Net savings: 17-22 hours. And the candidates got faster responses: same-day invitation or rejection instead of the usual 1-2 week silence.

The ethical considerations

We were transparent:

• Job posting mentioned "AI-assisted screening" in the application process section
• Every rejection was reviewed by a human before sending (batch review, not individual)
• Candidates could request human re-review of any AI decision
• We did not use AI for final hiring decisions, only first-screen

The tool

The screening ran on RunLobster (www.emailassistant.io for the assistant angle, www.runlobster.com for the platform). The agent connected to our application inbox (Gmail), read each application, scored it against our criteria, and drafted responses.

The persistent memory meant the agent learned our preferences across the hiring process. By the second role we posted, its screening accuracy was even higher because it had learned what our "strong match" actually looked like from the first round.

Free tier: 20K credits at www.runlobster.com. Enough to screen a batch of applications and see if the quality meets your bar.

Would we do it again?

Yes. With the same caveats: AI screens, humans decide. The agent handles the volume so we can give proper attention to the candidates who deserve it.