# 216 Tests Passed, but Our GitHub Build Failed: Lessons From Our First Security Hackathon

Arnav Goyal — Sun, 01 Nov 2026 00:52:50 +0000

This is a submission for the MLH x DEV Writing Challenge

Imagine spending hours building a project, fixing bugs, and finally seeing all 216 tests pass.

You feel like everything is ready.

Then you open GitHub and see a red cross next to your pull request.

That happened to us while building pkgtruth at Hacktoberfest Hack Day 2026, hosted at Rishihood University, Sonipat.

And surprisingly, it became one of the most useful parts of the experience.

How it started

Our team, Commit-tee, was working on a software supply chain security problem.

The task was to build a tool that could compare published software packages with their original source code.

At first, the idea sounded straightforward.

Take two versions of the code, compare them, and report the differences.

But as we started exploring Python packages, we realized there was much more to it.

A published Python wheel is not always identical to the source repository. Some differences are expected because of the packaging process, while others could introduce security risks.

So the real challenge wasn't just finding differences.

It was understanding which differences actually mattered.

Turning an idea into a working tool

We called our project pkgtruth.

Our goal was to build something developers could actually use, not just a demonstration that worked with one example.

We created a Python analysis engine and connected it to two interfaces.

One was a command line tool for developers who prefer working in a terminal.

The other was a browser interface where users could investigate packages and explore reports.

Over time, we added more features.

Release DNA lets users compare different releases and see what changed.

Proof Capsule generates a report containing the evidence collected during an investigation.

Release Gate evaluates the findings using configurable security rules.

We also integrated a local Qwen model through Ollama to help explain technical findings in simpler language.

One thing we were careful about was not allowing AI to change the actual security results.

The analysis engine finds the evidence. AI only helps explain it.

Why the small details mattered

One of our biggest challenges was making the reports understandable.

A tool might correctly identify a modified Python file, but what does that tell someone who doesn't know much about package security?

Should they be worried?

Was the change expected?

Could it affect what happens when the package is installed?

We wanted our reports to answer these questions without making claims we couldn't prove.

For example, detecting a potentially executable ⁠ .pth ⁠ file is important, but it doesn't automatically mean the package is malicious.

Similarly, a valid package RECORD file doesn't guarantee that the package came from a trustworthy publisher.

We learned that building a security tool isn't only about detecting problems.

It's also about explaining uncertainty.

The moment our tests weren't enough

Near the end, we ran our Python test suite.

216 tests passed.

We checked the web application, command line interface, and package installation.

Everything looked good.

We committed our changes and opened a pull request on GitHub.

Then our automated CI check failed.

At first, that was frustrating because the same project had passed every test locally.

After checking the GitHub Actions logs, we found the reason.

Our application generates example security reports. During development, we had improved the report format and added information to the Release DNA output.

But the example files committed to GitHub still contained the older output.

GitHub regenerated those files, compared them with the committed versions, and detected a mismatch.

The solution was simple once we understood the problem.

We regenerated the examples, reviewed the changes, committed the updated files, and pushed them to GitHub.

The next CI run passed.

It taught us something important.

A project isn't finished just because the code works on your own laptop.

It should also build, install, and behave consistently in another environment.

What we learned about building software

Before this project, it was easy to think of software development mainly as writing code and getting the expected output.

But pkgtruth involved much more.

We had to think about how users interact with the tool, how errors are reported, how files are handled safely, and how different parts of the application communicate.

We also used Git branches, pull requests, automated testing, and continuous integration throughout development.

These things might sound less exciting than adding a new feature, but they made the project much more reliable.

Another lesson was that a simple interface can take just as much thought as a complicated algorithm.

We spent time trying to turn technical findings into something a developer could understand in a few seconds.

And that effort was worth it.

Is pkgtruth finished?

Not completely.

There are still plenty of things we'd like to improve.

Right now, the project focuses on Python packages. In the future, we'd like to support more package ecosystems, improve release monitoring, and explore better ways to investigate suspicious changes.

We also want to keep improving the experience for developers who are new to software security.

For now, pkgtruth is an open source project with a working web interface, CLI, local AI explanations, and automated tests.

And it has given us a much better understanding of what goes into building real software.

Final thoughts

Hackathons are often about building something quickly.

But I think their biggest value is how much they force you to learn in a short period of time.

You encounter problems you've never faced before, search for solutions, make mistakes, fix them, and gradually see everything come together.

For us, pkgtruth was not just a project about Python package security.

It was also a lesson in debugging, teamwork, software design, and writing code that other people can actually use.

A big thanks to Hacktoberfest, Rishihood University, Newton School of Technology, and our team, Commit-tee.

If you're interested in checking out what we built, here's the repository:

GitHub: https://github.com/k3tan7/pkgtruth

And one final lesson from the experience:

Getting your code to work is one achievement. Making it reliable is another.

DEV Community: Arnav Goyal