We Built pkgtruth at Hacktoberfest to Find What's Really Inside Python Packages

Ketan Agarwal — Sun, 01 Nov 2026 00:47:37 +0000

This is a submission for the MLH x DEV Writing Challenge

When we install a Python library, we usually trust that the package contains the same code we see on GitHub.

But what if it doesn't?

That was the question behind our project at Hacktoberfest Hack Day 2026, hosted at Rishihood University, Sonipat.

Our team, Commit-tee, worked on Problem Statement 4: Published Package vs Source Code Integrity Analyzer.

We ended up building pkgtruth, a tool that helps developers understand what is actually inside a published Python package before they decide to trust it.

The problem we wanted to solve

Almost every developer uses open source libraries. A simple pip install can bring thousands of lines of code into a project.

We might check the GitHub repository, read the documentation, or look at the source code. But the file we actually install is often a built package called a wheel.

And the contents of that wheel may differ from the source code we reviewed.

Some differences are completely normal. Packaging tools add metadata, and documentation or test files may be excluded.

But other differences deserve attention.

What if a published package contains an extra Python file? What if a startup script appears in the wheel but not in the source? What if a new release introduces network access or changes an entry point?

Finding these differences manually can take time, especially when working with unfamiliar packages.

We wanted to make that process easier.

Introducing pkgtruth

Our idea was simple:

Don't just trust the package. Verify the release.

pkgtruth compares a published Python package with selected source code and shows what changed.

We built both a web interface and a command line tool, powered by the same Python analysis engine.

It can help developers:

Compare files using SHA-256 hashes.

Identify added, removed, and modified files.

Inspect wheel metadata and RECORD consistency.

Detect potential execution surfaces such as .pth files.

Examine security indicators, including network and subprocess capabilities.

Compare two releases to understand what changed over time.

Generate reports with supporting evidence.

One decision was especially important to us: pkgtruth never needs to execute the package it is investigating.

It inspects package contents statically instead of installing or running potentially suspicious code.

Making security reports understandable

Finding a difference is only half the problem. Explaining it clearly is just as important.

During development, we realized that a report full of hashes, file paths, and technical labels wouldn't be very useful to someone who was new to software security.

So we worked on making the reports easier to read.

Instead of only displaying raw findings, pkgtruth starts with a short summary explaining what deserves attention and why.

Developers can still inspect the detailed evidence, file differences, and hashes when needed.

We also added three features that became important parts of the project.

Release DNA helps compare package releases and identify newly observed changes.

Proof Capsule creates a portable report containing findings, hashes, and the limitations of the analysis.

Release Gate applies configurable rules and returns an ALLOW, REVIEW, or BLOCK recommendation.

These are recommendations based on the available evidence, not guarantees that a package is safe.

We also experimented with local AI

We wanted to explore how AI could help someone understand security findings without replacing the actual analysis.

For this, we integrated Qwen 3.5 4B through Ollama, running locally.

The analysis engine first identifies the evidence. The language model then helps explain selected findings in simpler terms.

The important distinction is that AI doesn't decide whether a package is safe. It doesn't change the file hashes, security findings, or policy decisions.

Its job is to help people understand the results.

We liked this approach because the underlying checks remain deterministic, while the explanations become easier to follow.

The part that taught us the most

Building the features was only one part of the challenge. Getting everything to work together took a lot of debugging.

We worked on the analysis engine, web interface, CLI, package installation, security checks, and automated tests.

One interesting moment came near the end.

Our local test suite had 216 passing tests, but our GitHub pull request still failed CI.

The problem wasn't the scanner itself. Some example reports were generated using an older version of our output format. After updating the code, we had forgotten to regenerate those examples.

We fixed the mismatch, committed the updated files, and ran CI again.

This time it passed.

It was a useful reminder that passing local tests doesn't always mean a project is ready. Packaging, reproducible outputs, and continuous integration matter too.

One thing we learned about security

A security tool should be clear about what it knows and what it doesn't.

For example, a valid wheel RECORD file shows internal consistency. It doesn't prove that the publisher is trustworthy.

Similarly, detecting a suspicious import doesn't prove that the code was executed or that a package is malicious.

We tried to preserve these distinctions throughout pkgtruth.

Sometimes the correct answer is simply that more investigation is needed.

What's next?

pkgtruth is still a project with plenty of room to grow.

We would like to expand support for more package ecosystems, improve release monitoring, and make investigations even easier for developers.

For now, we're happy that we managed to turn a hackathon problem statement into a working tool with a web interface, CLI, local AI assistance, automated tests, and a complete GitHub workflow.

More importantly, we learned a lot about Python packaging, software supply chain security, and what it takes to build a project beyond a simple prototype.

A big thank you to Hacktoberfest, Rishihood University, Newton School of Technology, and everyone who helped make the event possible.

And of course, to our team Commit-tee for building this together.

GitHub: https://github.com/k3tan7/pkgtruth

Final thought: The code you review and the package you install are not always the same thing. It's worth checking the difference.

DEV Community: Ketan Agarwal

We Built pkgtruth at Hacktoberfest to Find What's Really Inside Python Packages