We Built pkgtruth: Looking Beyond the Code We Trust

Arpit Chhimpa — Sun, 01 Nov 2026 00:39:53 +0000

This is a submission for the MLH x DEV Writing Challenge

pkgtruth: What You Review Isn't Always What You Install

What I Built

During Hacktoberfest 2026 at Newton School of Technology × Rishihood University, my team participated in a Hack Day where we had to turn a problem statement into a working product under a limited time.

Our team, Commit-tee, chose Problem Statement 4:

Published Package vs Source Code Integrity Analyzer.

At first, the problem seemed simple: compare the source code available on GitHub with the package published on a registry such as PyPI.

But then we started asking some important questions:

What if the published package contains files that aren't present in the source repository?
What if someone modifies a function in the published package?
What if the difference is simply generated metadata from the build process?
How can we distinguish a normal packaging difference from something that deserves a security investigation?

That's how we started building pkgtruth — Package Integrity & Provenance Intelligence.

The problem behind pkgtruth

Consider a Python developer who creates a library, pushes the source code to GitHub, reviews it, and publishes a release on PyPI.

Another developer installs it using:

pip install demo-package

The important thing to understand is that users install the published package, not the GitHub repository that the developer reviewed.

These two are related, but they are not automatically identical.

A published package may contain generated files and metadata, which are perfectly legitimate. However, it could also contain unexpected files or modified code that deserves investigation.

Finding differences is easy. Understanding which differences matter is the real challenge.

How pkgtruth approaches the problem

pkgtruth is designed to help developers investigate published Python package releases by comparing them with their corresponding source code.

We structured the product around four main capabilities:

1. Compare

Identify added files, missing files, modified contents, hashes, and metadata differences.

2. Investigate

Examine security-relevant indicators such as executable .pth files, unexpected imports, and potential execution surfaces.

3. Explain

Use AI-assisted explanations to help developers understand why a particular finding deserves attention.

4. Decide

Apply deterministic policies to recommend whether a release should be allowed, reviewed, or blocked.

A small example

Imagine the reviewed source contains:

# demo/message.py

def message():
    return "reviewed source"

But the published wheel contains an additional file:

demo/
├── message.py
└── extra.py

startup.pth

The startup.pth file contains an executable Python directive.

Certain .pth files can contain lines that Python processes during startup. This makes an unexpected executable startup file worth investigating.

pkgtruth aims to identify this difference, show the evidence, and explain its potential impact.

Importantly, the presence of such an indicator does not automatically prove malicious activity.

What makes pkgtruth different?

Beyond basic file comparison, we explored three additional capabilities:

Release DNA: Understand how package contents evolve across releases and highlight unusual changes.
Proof Capsule: Preserve artifact identity, hashes, findings, evidence, and analysis limitations in a reproducible report.
Release Gate: Translate configured security rules into ALLOW, REVIEW, or BLOCK recommendations.

We also wanted AI to have a specific responsibility.

We chose a locally hosted Qwen 3.5 4B model through Ollama for explaining security findings. The deterministic analysis engine handles the actual comparisons and security checks, while AI helps communicate the results in understandable language.

Our principle is:

AI explains the evidence. It doesn't invent the verdict.

Technology stack

Component	Technology
Frontend	HTML, CSS, HTMX
Backend	Python, FastAPI
Analysis	Archive inspection, SHA-256, file comparison, security rules
Package ecosystem	PyPI
AI	Ollama, Qwen 3.5 4B
Testing	pytest
CI	GitHub Actions

One of our most important design decisions was to never install or execute the package being investigated. We inspect its contents as data instead.

We also learned that a clean scan does not guarantee that a package is completely safe. A good security tool must communicate its limitations rather than make unrealistic promises.

Demo

GitHub Repository: pkgtruth — Team Commit-tee

pkgtruth is designed around a local-first web application and a CLI using the same Python analysis engine.

The intended investigation workflow is:

Provide a package artifact and its corresponding source archive, or select a PyPI release.
Safely inspect the archive contents.
Compare files, hashes, and relevant metadata.
Investigate security-related indicators.
Review findings and supporting evidence.
Generate an investigation report.

Demonstration scenarios

Scenario 1 — Normal release

Compare a package and its corresponding source and inspect the differences.

Scenario 2 — Unexpected change

Identify a simulated unexpected executable .pth file inside a published package.

Scenario 3 — Uncertain provenance

Show how pkgtruth handles unresolved source mapping instead of pretending that verification was successful.

[Insert your actual application screenshot here.]

[Insert your actual analysis-results screenshot here.]

[Embed your demo video or add a link here.]

The goal of the demo is to show not just that a difference exists, but why it matters and what evidence supports the finding.

Partner Technologies

One of the interesting parts of pkgtruth was exploring how to use a locally hosted language model in a security-focused application.

We used Ollama to host Qwen 3.5 4B locally.

Our approach separates deterministic analysis from AI-generated explanations.

The analysis engine performs tasks such as:

Inspecting package archives.
Comparing file contents and hashes.
Checking wheel RECORD consistency.
Identifying configured security indicators.
Producing structured findings.

The language model receives structured evidence and helps explain it in plain language.

For example, if the engine detects an executable directive in a .pth file, the model can explain why that file deserves attention and what a developer should investigate.

We deliberately avoid asking the model to independently decide whether a package is malicious.

This separation is important because security decisions should be grounded in reproducible evidence.

Working with a locally hosted model also encouraged us to think about what happens when the model is unavailable. The deterministic analysis should remain useful even without AI explanations.

Note: Ollama and Qwen are the technologies we used for local AI integration. We are not claiming that they are MLH partner technologies.

Hackathon Experience

Participating in Hack Day was more than just building an application. It was an experience of understanding an unfamiliar problem, discussing possible solutions, making technical decisions, and turning an idea into something demonstrable.

Our team, Commit-tee, consisted of:

Ketan Agarwal
Arnav Goyal
Arpit Chhimpa
Asmi Mishra

At the beginning, we had a broad problem statement and many possible directions. We could have built a simple file comparison tool, but we wanted to go further.

We started exploring questions around package integrity, source mapping, build-generated files, security indicators, release history, and evidence preservation.

One of the biggest things we learned was that not every difference is dangerous.

For example, a build process can generate metadata that doesn't exist in the original repository. On the other hand, an unexpected executable startup file may deserve much closer attention.

That distinction influenced how we designed pkgtruth.

Another important learning was about AI. Instead of allowing AI to make security decisions, we gave it the responsibility of explaining findings produced by deterministic checks.

We also realized that a security analyzer must be careful about its own behaviour. That is why we designed pkgtruth to inspect untrusted package contents without installing or executing them.

What I will remember most

The most valuable part of this experience was seeing how a problem that initially sounded like a simple comparison task could lead to a much deeper engineering challenge.

We had to think beyond writing code:

Understand the actual problem.
Decide what belongs in the core product.
Design a safe analysis workflow.
Make technical findings understandable.
Test the implementation.
Communicate the product through a clear demonstration.

It reminded me that building a useful product is not just about adding more features. It is about understanding the problem well enough to decide which features actually matter.

What comes next?

pkgtruth currently focuses on Python packages and PyPI. There is room to expand into additional ecosystems, strengthen provenance verification, improve release monitoring, and harden the application for production use.

We see this as a starting point rather than the end of the project.

Final Thoughts

This project changed how I think about software supply-chain security.

We often review source code and trust the package that eventually reaches our environment. But there is an important journey between those two points, and that journey deserves verification.

Building pkgtruth helped us explore that gap and understand how file comparison, security analysis, evidence, and AI can work together.

Don't just trust the package. Verify the release.

Project: pkgtruth

Team: Commit-tee

Event: Hacktoberfest 2026 · Hack Day

Institution: Newton School of Technology × Rishihood University

Problem Statement: Published Package vs Source Code Integrity Analyzer

Explore pkgtruth on GitHub

DEV Community: Arpit Chhimpa