Magen: Local Penetration Testing

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Magen is a local-only security testing copilot. It captures traffic while you have the proxy running, flags leaked secrets and personal data, and uses a local open-weight model to reason about what it found and to write and run discovery probes. The one thing that defines it: every part runs on your machine. Nothing sensitive leaves the laptop.

The problem it solves is specific. When you security test a real application you are handling live session tokens, .env files, and other people's personal data. You cannot paste any of that into a hosted LLM without creating a new liability. A model running locally removes that constraint, so the tool can reason over the real data instead of a redacted shadow of it.

Who would pay for it: freelance security testers, small teams, and bug bounty hunters who want an AI assistant but cannot ship client data to a third party.

What it does today:

• A scope-locked intercepting proxy (built on mitmproxy) that only records hosts you explicitly opt in.
• Deterministic secret and PII detection with stable redaction, so a report can reference a secret by a token like «jwt:7c5b0229» instead of the value.
• Local anomaly detection across captured flows, using robust statistics and no model at all, to surface the one request that behaves unlike its siblings.
• Local-model triage, plus an agentic "explore" mode that writes standard-library Python probes, runs them in a guarded sandbox, and scores findings from the evidence they produce.
• A loopback-only dashboard to drive all of it.

How I Built It

The open pieces are the project, not an accessory to it:

• Inference: Gemma (gemma3:12b-it-qat, with gemma3:4b as a fast fallback) running locally through Ollama.
• Capture: mitmproxy as the interception engine.
• Everything else is the Python standard library, on purpose, so the tool works offline with almost no dependencies.

The design choice I am happiest with is how the agent handles secrets. When the model writes a probe to test an endpoint, Magen injects the real captured token into the probe's process through an environment variable at run time. The model never sees the token and never writes it into code, yet the probe still authenticates for real. The model reasons, the machine holds the secret, and nothing leaves the device.

Why Does Open Innovation Matter?

This is the whole reason the project exists. Three things a closed API could not give me:

1. I can hand the model real credentials and real responses, because they never leave the machine. That is the core capability, and for a security tool it is not a tuning preference, it is a hard requirement. Cloud reasoning over live secrets is a non-starter, not a trade-off.
2. It costs nothing per run and does not depend on anyone's pricing or uptime.
3. I can swap models freely and, in principle, fine-tune one for this exact task.

Running an open model locally was not a nice-to-have here. It was the only honest way to build something that touches credentials. While a hosted open-weight model could have done work here, I felt it went against the principals of what I was trying to accomplish, and that meant making external requests.

What Worked, What Didn't, and What I Would Change

This project ended up being more of a learning experience than a finished product, and I think that is worth noting.

The deterministic parts worked well: capture, secret and PII detection, redaction, and the statistics-based anomaly detection. Those are reliable, genuinely useful on their own, and they need no model.

The local-model layer is where I fell short. The models that run comfortably on my hardware were not quite strong enough for the agentic security reasoning I was asking for. The small model is fast but shallow. The larger one reasons better but takes close to a minute per step and was still inconsistent; it would reach for libraries that were not available, pick weak endpoints to probe, or misjudge whether a response was actually a finding. That is not the model being bad. It is a mismatch between a general-purpose local model with no task-specific tuning and the fairly precise job I wanted it to do.

Two things would have closed most of that gap, and I had time for neither: fine-tuning a model for this narrow task, and simply iterating longer on the prompts and the evidence format the model has to produce.

The bigger lesson was about scope. I started too broad. Under time pressure I kept adding features: a service manager, a one-click demo, a managed test browser, multiple probe goals. Each one widened the surface area, and each one introduced bugs I then had to chase instead of improving the core. A tighter starting scope, something like "capture traffic, scan it for secrets locally, and produce a redacted report," would have been a smaller and far more solid product, with the local model doing one job it could actually do well. Improper scoping at the very start was probably the single biggest thing working against me.

So I do not read this as a failure. The privacy thesis holds up, the deterministic tooling is good, and I now know exactly where a local-first AI tool earns its keep and where it still needs either a stronger model or real tuning time. That is a useful thing to carry into the next build.

Prize Categories

• Best Use of Gemma: Magen runs Gemma locally through Ollama as its reasoning and discovery engine.