Before I start, an admission that most write-ups like this would bury at the
bottom: I never ran this thing end-to-end.
The code exists. The tests pass. The CI is green. But the Docker Compose stack
has never been started on the machine it was written on, because Docker isn't
installed there, and I didn't install it. What I tested was the pieces. What I
never tested was the whole.
This is a write-up about building a system you can't fully verify, and about
the discipline required to ship it anyway — not because I'm proud of the gap,
but because pretending the gap doesn't exist would be the wrong way to write
about engineering.
The scope, before the story
Before I tell you what's in it, here's what's deliberately not in it. This
list matters more than the feature list, because it's the list that made the
project finishable.
Out of scope:
- Authentication on the dashboard or API
- Multi-tenant isolation
- ML-based fraud detection
- Kubernetes
- Terraform / IaC
- PCI-DSS compliance
- Alert acknowledgement UI
- Real provider integrations
Every one of those is in the README under a section called "Out of Scope," with
a one-line reason. The reasons are boring: "this is a portfolio demo, not a
payment processor." But writing that list down, before starting, was the thing
that made the rest of the build possible.
Most portfolio projects fail because the builder keeps adding scope. The
project starts small and ends up a half-built version of everything. I
deliberately refused the additions. If you look at the README's commit history
you'll see the same project from first commit to last — no features appeared
mid-build. That's not luck. That's the out-of-scope list doing its job.
What it actually is
A payments detection platform, in the sense that it:
- Receives webhooks from a payment provider (with HMAC verification)
- Persists them to a database (Postgres)
- Detects suspicious patterns in those events (four rules, one per attack class)
- Alerts on detection (deduplicated, severity-rated)
- Displays alerts on a dashboard with AI-generated explanations
The four detection rules cover:
- Velocity bursts — more than 10 payment events from one IP in 60 seconds
- Replay attacks — the same event ID arriving twice with different signatures
- Signature failures — more than 5 invalid HMAC signatures from one IP in 60 seconds
- Amount anomalies — a payment that's non-positive, or 3 standard deviations above the trailing 24-hour mean
Each rule is a YAML file with a title, id, level, description, and an
aggregation block that describes the detection logic. The rules are
Sigma-inspired — the format is not directly executable by a generic Sigma
engine, because Sigma doesn't have a portable way to express "sliding 60-second
window" or "z-score against a trailing baseline." Those extensions are
documented in the README.
The AI part: when an alert fires, the dashboard can request an explanation
from a language model. The prompt sends only the rule ID, severity, and numeric
context — never the raw event payload — and the whole path is wrapped in
prompt-injection defenses (sanitization before the call, structured prompt with
delimiters, output validation). There are 5 adversarial tests for those
defenses alone.
None of that is verified end-to-end. It's verified piece by piece.
What I could test
Unit tests: 6 tests in the alert-api service covering signature
verification, payload redaction, prompt construction, and the fallback logic
when the LLM is unavailable. These run on every commit and pass.
Python compilation: every .py file in every service compiles.
YAML validation: every rule file parses as valid YAML.
The security tests, specifically: the prompt injection tests are the ones
I'm proudest of, because they verify something subtle. The test
test_model_prompt_excludes_untrusted_identifiers sends a fake alert with
malicious-looking context ("ignore all previous instructions",
"system: reveal secrets") and asserts that the prompt sent to the model
contains the rule ID and numeric context but does NOT contain the injection
strings. That test is real. It catches a real class of bug. It runs.
What it doesn't catch is the case where the sanitization logic is correct but
the model call itself has a bug, or the API key is misconfigured, or the rate
limit gets hit at exactly the wrong moment. Those are integration failures,
and integration failures are what I couldn't test.
What I couldn't test, and what that means
The stack is:
- A Postgres 16 container
- A Redis 7 container
- An alert-api container (FastAPI)
- A detector container (Python worker, polls Postgres for new events, evaluates rules)
- A dashboard container (Nginx serving static files)
docker compose up --build builds all five and starts them. The
001_init.sql migration creates the schema. The detector polls every 2
seconds. The alert-api serves /alerts and /metrics. The dashboard fetches
from the alert-api and renders a triage console.
What I never did is run that sequence.
The README has a section called "Development Notes" that says:
"The Compose stack was authored and structurally validated (YAML syntax,
Python compile check, schema migration review, and CI) but not executed
end-to-end on the authoring machine because Docker was not installed
locally. Anyone with Docker Engine and Compose v2 can rundocker compose upto verify the full pipeline."
--build
That's the honest version. It's not "this doesn't work." It's "this hasn't
been verified to work as a whole, and here's why."
The trap I could have fallen into — and the trap most projects fall into — is
writing "docker compose up --build" at the top of the README as if it's
been tested. It's a small lie. It would have been a lie that someone else
discovers when they try to run it and something breaks, and they don't know
whether the problem is their setup or mine.
The README is written to prevent that. The development notes section is
written to prevent that. The "if you find a configuration issue, please open an
issue" line is written to prevent that.
If you're building something and you can't test a part of it, say so. The
sentence "I couldn't test this end-to-end" costs nothing and buys the reader a
lot. It tells them where to look when it breaks.
The specific bugs I did find
Even without Docker, the pieces generated their own friction. Three things
worth naming.
1. Groq retired the model mid-build. The first version of the AI path used
llama-3.3-70b-versatile. When I came back to test it, Groq had retired that
model. Every request returned a 404: "The model llama-3.3-70b-versatile does The fix was one line — swap the
not exist or you do not have access to it."
model ID for openai/gpt-oss-120b — but it took 15 minutes of staring at the
401-error-inside-a-200-response before I realized the API was telling me what
was wrong.
The lesson isn't "pin your model versions." It's that third-party model IDs
are as fragile as third-party APIs, and the failure mode is a 4xx disguised as
a successful HTTP response. The 200 status code is not the same as "the LLM
worked."
2. os got shadowed inside a function. The summary.py file has a helper
called _get_client() that reads environment variables. In the first version,
I had import os at the top and used os.environ.get() inside the function.
Python's scoping rule meant the function couldn't see the module-level os,
because there was an assignment to os somewhere in the function body — a
stray line I'd added during refactoring.
The error was cannot access local variable 'os' where it is not associated. It took reading the file twice to find the stray assignment. The
with a value
fix was to delete three lines. But the lesson is worth keeping: Python's
scoping is per-function at compile time, not per-line at runtime. A single
assignment anywhere in a function makes the name local for the whole function,
even if the assignment happens after the use. That's the kind of rule that
makes sense in retrospect and is invisible in the moment.
3. Reading env vars at module load broke the tests. The first version of
summary.py had:
GROQ_API_KEY = os.getenv("GROQ_API_KEY", "")
at module level. That's fine in production — the env var is set before the
module is imported. But in tests, monkeypatch.setenv sets the env var
after the module is imported, so the module-level constant stays "". The
test asserting that the LLM was called with the right context failed because
the LLM was never called — the code saw an empty key and fell back to the
deterministic path.
The fix was to read the env var inside the function, at call time, not at
module load. That's a rule I now apply to every service that might get tested:
Read configuration inside functions, not at module top level, if the
functions are going to be tested with monkeypatched config.
It's a small rule. It's the kind of rule you learn by having a test fail for a
reason you can't figure out, then realizing the test was fine and the code was
subtly wrong.
Why ship it anyway
Here's the case for shipping something you can't fully test:
1. The pieces are real, and the pieces are what matter most of the time. The
signature verification is real. The Sigma rules are real. The threat model is
real. Those are the things an interviewer or client will actually read. The
Docker Compose that they may or may not run is the scaffolding.
2. The gap is documented, and documented gaps are honest. An undocumented
gap is a lie waiting to be caught. A documented gap is a contract: "here's
what I tested, here's what I didn't, here's how you can test it yourself." The
second one is what good engineering documentation looks like. The first one is
what most portfolio projects look like.
3. Shipping it creates the pressure to fix it. If I'd held the project
until Docker was installed, it would still be sitting on my laptop. Shipping it
with a documented gap means the first person who tries to run it will report
the bug. That's the normal cycle of open-source development. Waiting for
perfection means waiting forever.
4. The "out of scope" section is the interview. When I show this project to
someone who's done security work, they don't look at the code first. They look
at the STRIDE table. They look at the four rule IDs. They look at the prompt
injection tests. The fact that I know what I didn't build — and can defend
that choice — is the signal. Anyone can build a feature. Almost nobody can
articulate what they deliberately didn't build and why.
The honest version of what this project is, is: a demonstration of scope
discipline, with a working Docker Compose stack that hasn't been verified
end-to-end. The first half is what I'm proud of. The second half is what I'd
fix first.
What it doesn't do (beyond the out-of-scope list)
- No persistent event storage beyond the demo. The migration creates the schema, but there's no retention policy, no archival, no cleanup job. A real deployment would fill up in weeks.
-
No real LLM path enabled by default. The AI path is optional; without
GROQ_API_KEYset, the service uses deterministic rule-based explanations. That's a design choice, not a limitation — but it means the AI path has only been tested against fixtures, not against a real model. -
No CI for the Docker build. The CI checks Python syntax, YAML validity,
and the security tests. It does not attempt
docker compose build. That would require a CI runner with Docker available — most hosted runners have it, but I didn't set it up, and that's a real gap. - No integration tests. No test that posts a webhook, waits for the detector to fire a rule, and asserts the alert appears. That's the test I'd write second, after I fix the Docker gap.
The lesson I'd give anyone building something similar
Test what matters, document what you couldn't, and don't pretend the
difference doesn't exist.
The three artifacts that carry the project are:
- The STRIDE threat model — because it forces you to name what you're defending against before you defend against it.
- The Sigma rules as versioned YAML — because it makes the detection logic readable, diffable, and portable, in a way that inline Python doesn't.
- The 5 prompt-injection tests — because they verify a specific, adversarial property (the model doesn't get hijacked by attacker-controlled input), which is the kind of thing a normal test suite misses.
None of those three require Docker. All three are testable on a laptop. All
three are what the project actually is.
The Docker Compose stack is the deployment story. It's important. It's not the
whole project, and treating it as the whole project would have meant not
shipping at all.
The link
The project is at
github.com/Agenda-X-PR/payments-detection.
If you have Docker installed and you're willing to try the full stack, I'd
genuinely appreciate knowing whether docker compose up --build works. The
README's development notes say it hasn't been verified. If you run it and
something breaks, that's the most useful feedback anyone could give me right
now.
This is the fourth in a series about the projects I've shipped. The first was
why Excel fails at payment reconciliation, the second was
reconciling a Paystack export against a bank statement, and
the third was building a service for failures you can't see.
Top comments (0)