DEV Community

Ayush Sharma
Ayush Sharma

Posted on

Exploring HackerRank's Open-Source AI Hiring Agent: A Hands-on Walkthrough (Guide)

Can an AI evaluate a resume the way a recruiter would?

That question made me explore HackerRank's open-source Hiring Agent (thanks to Hardik Goyal). I wanted to understand how it works, how difficult it is to set up, and what kind of feedback it actually provides. This article is a walkthrough of that journey-from setting it up on WSL Ubuntu to debugging a few unexpected issues and finally running my resume through the evaluation pipeline.

Why I Tried This

A few days ago, while discussing resume reviews with one of my seniors, Hardik Goyal, I asked him if there was a good way to get an estimate of how an AI-based hiring system might evaluate my resume.

His response was simple:

"I think HackerRank recently open-sourced their ATS, try that."

That got me curious enough to try it myself.

After looking into the repository, I realized it wasn't actually a complete ATS, but an open-source AI resume evaluation pipeline published by HackerRank's InterviewStreet GitHub organization.

Since I was already planning to improve my resume, I thought this would be a good opportunity to understand how the project works, set it up locally, and see what kind of insights it could provide.

This article is a walkthrough of that entire experience-from the initial setup to debugging a few unexpected issues, and finally running the evaluation successfully.

What This Project Is (and Isn't)

Before jumping into the setup, it's worth clarifying one thing.

This project isn't the complete hiring system used by HackerRank, nor should its score be treated as an industry-standard ATS score.

Instead, it's an open-source project that demonstrates one part of a hiring workflow—AI-assisted resume evaluation.

It combines resume parsing, GitHub analysis, and LLM-based reasoning to produce a structured report instead of simply matching keywords. src

That transparency is what caught my attention.

Overall Architecture

The entire pipeline is surprisingly straightforward.

Instead of only reading the PDF, it also tries to understand your GitHub profile before evaluating your overall engineering profile.

My Setup

I used the following environment:

  • Windows 11
  • WSL2 (Ubuntu)
  • Nano (for quick terminal edits)
  • Python 3.12
  • Google Gemini API (AI Studio)
  • GitHub API (pre-configured)

The setup itself wasn't difficult, but I did run into a few issues along the way.

Cloning the Repository

The first step was simply cloning the project.

git clone https://github.com/interviewstreet/hiring-agent.git
cd hiring-agent
Enter fullscreen mode Exit fullscreen mode

After that I created a Python virtual environment.

Or at least... I tried to.

Issue #1 — Python Wasn't Found

My first attempt was:

python -m venv .venv
Enter fullscreen mode Exit fullscreen mode

which immediately failed with:

Command 'python' not found
Enter fullscreen mode Exit fullscreen mode

Nothing serious.

Ubuntu ships with python3 instead of python.

So I switched to:

python3 -m venv .venv
Enter fullscreen mode Exit fullscreen mode

I thought that would solve it.

It didn't.

Issue #2 — Missing venv Package

The next error looked like this:

It turns out Ubuntu doesn't install the virtual environment package by default.

Installing it solved the problem.

sudo apt update

sudo apt install python3.12-venv
Enter fullscreen mode Exit fullscreen mode

After that:

python3 -m venv .venv

source .venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

worked perfectly.

Installing Dependencies

Once the virtual environment was ready, installing everything else was straightforward.

pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

The repository installs everything required for:

  • PDF parsing
  • Gemini
  • Ollama
  • GitHub integration
  • Jinja templates
  • Pydantic models

No surprises here.

Configuring Gemini

The project supports two providers:

  • Ollama
  • Google Gemini

Since I already had a Gemini API key, I decided to use that.

While editing the .env file, I also ended up learning two Nano shortcuts that I'll probably keep using:

-> Alt + U → Undo
-> Alt + E → Redo

The .env file looked like this:

LLM_PROVIDER=gemini
DEFAULT_MODEL=gemini-2.5-pro
GEMINI_API_KEY=YOUR_API_KEY
Enter fullscreen mode Exit fullscreen mode

Simple enough.

Or so I thought.

Organizing Resume Files

Instead of keeping resumes scattered across different folders, I created a dedicated directory inside the project.

(mkdir resumes)

Since my resume was stored on my Windows W: drive, I first verified it was available inside WSL using ls /mnt, then copied it into a dedicated resumes folder while giving it a shorter filename for easier execution.

ls /mnt

cp "/mnt/w/ResumeS/<PATH_TO_MY_RESUME>/Ayush_Sharma_Resume.pdf" \
resumes/resume_NEW_NAME.pdf
Enter fullscreen mode Exit fullscreen mode 
hiring-agent
│
├── resumes
│     ├── resume_general.pdf
│     ├── resume_google.pdf
│     └── resume_microsoft.pdf
Enter fullscreen mode Exit fullscreen mode

This makes it easy to evaluate different resume versions without changing commands every time.

First Run

With everything configured, I finally ran:

python score.py resumes/resume_microsoft.pdf
Enter fullscreen mode Exit fullscreen mode

The logs started scrolling.

The Gemini provider initialized successfully.

Everything looked good...

Until it suddenly stopped.

At this point, I wasn't sure whether the problem was with my API key, my project configuration, or the repository itself.

Issue #3 — Gemini Quota Error

The error looked something like this:

429

Quota exceeded

limit: 0
Enter fullscreen mode Exit fullscreen mode

At first, I assumed I had entered the wrong API key.

Then I checked Google AI Studio.

Everything looked normal.

I even verified that the project existed.

Still the same error.

Debugging the API

Instead of guessing, I decided to verify each assumption one by one.

The first thing I checked was whether my API key actually worked.

I listed all available Gemini models using a simple curl request.

It worked.

Next, I tested a small prompt directly against the API.

Again, it worked.

That confirmed:

  • my API key was valid
  • the Gemini API was working
  • networking wasn't the problem

The issue had to be somewhere else.

The Actual Fix

After comparing the responses, I noticed something interesting.

The repository was configured to use:

gemini-2.5-pro
Enter fullscreen mode Exit fullscreen mode

But my direct API request used:

gemini-2.5-flash
Enter fullscreen mode Exit fullscreen mode

Changing one line inside .env fixed everything.

DEFAULT_MODEL=gemini-2.5-flash
Enter fullscreen mode Exit fullscreen mode

The project started working immediately.

Sometimes debugging is simply about eliminating possibilities one by one instead of assuming the first error message tells the whole story.

GitHub Enrichment

One feature I genuinely liked was the GitHub integration.

Instead of relying only on the resume PDF, the project also looks at your GitHub profile.

It fetches repositories, classifies them, and then asks the LLM to identify the projects that best represent your work before generating the evaluation.

That adds another layer of context which many simple resume checkers completely ignore.

No doubt, this was actually my favorite feature of the project.

How the Evaluation Works

Once everything is collected, the evaluation pipeline looks like this.

The final report isn't just a number.

It also explains why certain scores were assigned.

What I Liked

I wasn't particularly interested in chasing a higher score.

I was more interested in understanding the evaluation process.

One thing I really appreciated was the idea of bonus points.

Instead of only looking for missing things, the system also rewards additional effort, such as maintaining a portfolio website or having a LinkedIn profile.

I also liked that deductions weren't hidden.

Rather than silently reducing the score, the report explicitly mentions why points were deducted.

Whether you agree with those deductions or not, at least you know what influenced the result.

The recommendations section was another pleasant surprise.

They weren't just generic resume advice copied from the internet.

They were generated based on the overall profile the system had built using both the resume and GitHub.

Not every suggestion perfectly matched my situation, but several of them were genuinely useful and worth thinking about.

That kind of transparency makes the feedback much more actionable.

For reference, this is the report generated for my resume:

Things That Could Be Better

Like any open-source project, there is room for improvement.

I think project ranking could become smarter by considering more GitHub signals instead of relying primarily on LLM selection.

It would also be nice if the system automatically detected deployed applications from platforms like GitHub Pages, Vercel, Netlify, or Render instead of assuming every project without an explicit demo deserves a deduction.

Another improvement could be different evaluation rubrics for different roles.

The expectations for a backend engineer, machine learning engineer, or frontend developer are obviously different, so role-specific scoring could make the evaluation even more meaningful.

What I Learned

I originally started this just to test a resume.

Instead, I ended up learning much more than I expected.

Along the way I got to:

  • configure Python environments inside WSL
  • debug missing Ubuntu packages
  • work with the Gemini API
  • investigate API quota issues
  • understand GitHub rate limits
  • explore how LLMs can be used for structured information extraction
  • see how AI-assisted resume evaluation pipelines are designed

Sometimes the setup process teaches you just as much as the project itself.

Final Thoughts

I don't think anyone should treat the final score from this project as an absolute measure of their resume.

Hiring decisions involve far more context than any single AI system can capture.

But I do think it's a great project to explore if you're curious about how AI-assisted resume evaluation works behind the scenes.

It offers a transparent look into how an AI-powered resume evaluation pipeline can be built, and I appreciate that it explains its reasoning instead of hiding everything behind a single percentage.

If you're interested in AI tooling, resume evaluation, or simply want to understand how these systems work under the hood, it's definitely worth exploring.

And if you're into open source, it also looks like a fun project to contribute to.

Resources

If you've tried this project—or know of similar open-source tools—I'd love to hear your experience in the comments.

If you enjoyed the article, consider leaving a ❤️. It really helps!

Top comments (0)