12 Hours, 15 CVEs, 5 That Mattered: Building TrackBack at Hacktoberfest Hack Day

At 6:30 PM on 31 October, my teammate Sambhav and I walked into Hacktoberfest Hack Day Sonipat at Newton School of Technology, Rishihood University with no idea what we would build. Twelve hours later, at 6:30 AM, our repository was frozen with 13 merged pull requests, 64 tests, a green CI pipeline and a tool we are genuinely proud of.

This is the story of those twelve hours: what we built, how we built it like a real open-source project, and what broke along the way.

Picking the problem

The organizers gave us a list of problem statements. We spent the first part of the night not coding, but reading the evaluation criteria for each one. We picked Problem 03: a Python Vulnerability Reachability Analyzer, because it had something rare for a hackathon: a clear, testable definition of "correct".

The problem it solves is one every developer knows. You run a security scanner and get a wall of alerts:

requests 2.25.1 has 4 known vulnerabilities. urllib3 1.26.4 has 11.

But most of those alerts never matter. A CVE in a function your code never calls cannot hurt you. Scanners work at the package level, so they flag everything, developers stop reading the alerts, and the real ones get lost. That is alert fatigue.

Our question was simple: which vulnerability alerts can actually reach your code?

What we built: TrackBack

We named the tool after our team, TrackBack, because it tracks the call path back from a vulnerable function to your own code. For every alert in a Python project it gives one of three verdicts:

• REACHABLE: a call path from your code reaches the vulnerable function, and we show every hop with file and line
• UNCERTAIN: no proven path, but something hides the answer (dynamic code, an object of unknown type), and we say exactly why
• NOT AFFECTED: the package is installed, but the vulnerable code never runs

Here is the moment the project clicked for us. Our sample app never imports urllib3. It only calls requests.get. TrackBack still finds this:

REACHABLE     CVE-2021-33503  urllib3==1.26.4  Catastrophic backtracking in URL authority parser
                -> app.<module>  (app.py:18)
                  -> app.main  (app.py:14)
                    -> app.fetch  (app.py:9)
                      -> requests.api.get  (requests/api.py:76)
                        -> ... five more calls inside requests ...
                          -> urllib3.util.url.parse_url  (urllib3/util/url.py:330)
                fix: upgrade urllib3 to 1.26.5

On that app, a normal scanner reports 15 alerts. TrackBack says 5 are reachable, and gives a reason for each of the other 10.

How it works

TrackBack runs six steps, all on your own laptop:

1. Read dependencies: exact == pins from requirements.txt and pyproject.toml
2. Find advisories: from the OSV database, merging the GHSA, PYSEC and CVE records for the same issue
3. Load the real source: the exact pinned version from PyPI, only the .py files, parsed with Python's ast module. Nothing is installed or executed
4. Build the call graph: across the app and its dependencies, following imports, aliases, classes, self, super() and more
5. Identify the vulnerable function: with an open-source model (more below)
6. Trace the paths: a breadth-first search from the app's entry points to the vulnerable function

Two design ideas made the biggest difference.

A vulnerable function is a name plus a condition. PyYAML's yaml.load is only dangerous without a safe loader, and yaml.safe_load is literally load(stream, SafeLoader). A plain call graph would flag every safe_load user. So TrackBack checks the actual argument at each call: yaml.load(text) is reachable, yaml.safe_load(text) is not affected.

Never a wrong "safe". Python is dynamic. When we cannot see the answer, because of eval, getattr with a variable name, or a method called on an object of unknown type, the verdict is UNCERTAIN, never NOT AFFECTED. A false alarm costs a developer a few minutes. A false "safe" can cost a breach.

The open-source AI part

The hardest part of the problem is that advisories are written in English. "Catastrophic backtracking in URL authority parser" does not tell you the function is urllib3.util.url.parse_url.

So we use Qwen2.5-Coder 7B, an open-weight model running locally through Ollama. TrackBack downloads the fix commit for each advisory, and the model reads the advisory text plus the diff and names the vulnerable functions. Then comes the part we care about most: every name the model returns is checked against the real source code. For one urllib3 advisory, Qwen suggested a method called read1, which does not exist in that version. It was dropped automatically.

The model suggests. Static analysis decides. That way the same input always gives the same verdict, and a hallucination can never turn into a "safe".

Working like a real open-source project

The hack day had rules that felt strict at first and turned out to be the best part of the night:

• The first commit was only HACKATHON.md. After that, main was PR-only
• Every PR ran GitHub Actions: install, tests, and our evaluation script
• Every PR had to name the models the product uses and any AI coding assistant that helped write it

We used a PR template for that last rule. Our product uses only Qwen2.5-Coder; we used Claude Code as a coding assistant, and we named it in every PR. Being honest about that from the first PR made the questions at the end easy.

By the morning we had 13 merged pull requests, each one small and focused: inputs, engine, output, model, evidence, docs, a web interface, and so on. Reviewing your own work one PR at a time at 3 AM is surprisingly good for catching mistakes.

What broke along the way

Plenty.

• PyYAML's old wheel was Python 2 code. Our parser choked until we taught the fetcher to prefer Python 3 wheels.
• Dead code was reported as reachable. A function nobody called still counted as an entry point. We added a script-versus-library rule, and now the report says "nothing in the app calls this function".
• The model download stalled. I closed my laptop lid in the hostel and came back to five stuck downloads. Restarting Ollama fixed it, about half an hour before the 11 PM mentor check.
• A real bug from a real project. We scanned a sandboxed Jinja2 app and TrackBack said "not affected". That was wrong: Jinja compiles templates into Python code and runs it, so calls are invisible to static analysis. We added a rule: if the vulnerable package generates code at runtime, the answer is UNCERTAIN. Testing on real repositories found a problem our own examples never would have.
• We renamed the tool at 4 AM. Our first name turned out to belong to an existing company, so we renamed everything to TrackBack: package, command, reports and docs. The tests made that safe.

Proving it works

We did not want to just claim it works, so we measured it. The repository has 8 small example apps, each built around one trap (aliases, wrappers, dead code, safe loaders, getattr, dynamic imports), each with its expected answer and the tool's real output committed:

$ python scripts/evaluate.py
expected \ got   reachable     uncertain     not_affected
reachable        7             0             0
uncertain        0             2             0
not_affected     0             0             6

15/15 correct, 0 wrongly reported as not affected

We also built a local web interface for the demo. It shows a headline like "5 of 15 alerts can actually reach your code", every call path step by step with your own code highlighted, and a "How TrackBack decided" panel that walks through all six steps of the scan.

What we learned

• Read the evaluation criteria before writing code. It changed which problem we picked and what we built first.
• Small PRs and CI are not bureaucracy. At 4 AM, a green check is the only thing you can trust.
• Use AI where it is strong, and verify it. The model is great at reading advisories and diffs. It is not a source of truth.
• "I don't know" is a feature. Saying UNCERTAIN with a clear reason is more useful than a confident wrong answer.
• Test on real code early. Our biggest bug came from someone else's repository.

Try it

TrackBack is open source under the MIT license:

github.com/Arjun421/Python-Vulnerability-Reachability-Analyzer_

pip install -e ".[dev]"
trackback scan path/to/your/project
trackback serve --root examples --cache examples/cache --offline

Thanks to the organizers and mentors of Hacktoberfest Hack Day Sonipat at Newton School of Technology and Rishihood University for a night we will remember, and to my teammate Sambhav Kumar for staying up through all twelve hours of it.

Team TrackBack: Arjun Singh and Sambhav Kumar