DEV Community

Osama Malik
Osama Malik

Posted on Originally published at osamamalik.me

BleedStop: How I Built an LLM Cost and Security Guard

BleedStop is a cost and security guard I built for LLM, SMS and payment APIs. An app calls POST /decide before it spends, and BleedStop answers ALLOW, LIMIT, CHALLENGE or DENY from deterministic per tenant rules.

How I built BleedStop

This is the longer note behind the case study. The code is in usmamalik345/bleedstop. The stack on the case study is Node.js, Python, Postgres, Redis and Kubernetes. The numbers below are the ones I measured, or the ones the repository README already publishes. I am not adding a customer, a production cluster, or a savings figure that those pages do not state.

Why the decision sits in front of the spend

Once an agent can call tools and paid APIs, one prompt injection or one runaway loop can leak data or burn a budget. I mapped that to OWASP LLM10, Unbounded Consumption. BleedStop is the control plane in front of the spend. Rules are set per tenant. Budgets are enforced atomically across instances through Redis.

The hot path is deterministic. The repository README says agents only explain, and they are not the thing that denies a call. That is the design I want: a hanging model, or a poisoned prompt, should not be what decides whether money moves.

A call looks like this. The app, or the Node SDK Express middleware, asks POST /decide. BleedStop checks the request, including the phrase list and, when I turn it on, the classifier described in the prompt injection post. It checks the budget. It writes the decision. The caller gets ALLOW, LIMIT, CHALLENGE or DENY.

Budgets, the audit log, and the deploy

Redis holds the budget so several instances share one ceiling. On the case study, a correctness test of 2,000 concurrent calls against a $5 ceiling on 3 instances overspent by $0.000. The same shape behind a load balancer did 6,291 decisions per second, with p99 at 31.9 ms.

Every decision is written to a Postgres audit table. A trigger blocks UPDATE and DELETE, so a decision can be reviewed later and cannot be edited in place.

The Docker Compose stack is 3 API replicas, Redis, Postgres and nginx. The Helm chart adds readiness and liveness probes, a PodDisruptionBudget, an HPA and graceful drain. Around that I have Prometheus metrics, Grafana dashboards, alert rules, GitHub Actions CI, a Node SDK, a Python SDK, multi tenant YAML policy, and OpenTelemetry.

There is also an OpenAI compatible proxy in the repo. POST /v1/chat/completions reserves budget for the prompt plus a capped output. The README says DENY comes back as HTTP 429 and CHALLENGE as HTTP 403, in an OpenAI error shape, and LIMIT downgrades to the small model. The output cap is the smallest of the requested cap, OPENAI_PROXY_MAX_OUTPUT_TOKENS (1024), and the remaining budget. The README says that cap is enforced mid stream even if the upstream ignores max_tokens, and that reservations are committed with actual usage.

What I measured

All of the case study numbers were measured by me on one shared 8 vCPU Linux machine. The load generator, the load balancer and the services share that hardware, so I treat the absolute rates as a lower bound for a real cluster. Raw results are in the repository.

Test Result
Docker Compose stack (3 API replicas, Redis, Postgres, nginx), one 8 vCPU machine 5,433 decisions per second, p99 36.2 ms, 0 failed requests out of 60,000
3 instances behind a load balancer with Redis atomic budgets 6,291 decisions per second, p99 31.9 ms, $0.000 overspend against a $5 ceiling
Kubernetes rolling restart of 6 pods under load 0 failed requests out of 287,403. The README also records p99 33.2 ms for this run.
Postgres stopped for about 7 seconds under load 86,071 requests, 0 failed, 0 ledger rows dropped
Golden set of 12,140 labeled requests Precision 100%, benign false positive rate 0.0%, recall 48.0%
Capped request against a live model API 64 output tokens instead of 2,618 (97.6% fewer)

The README gives more detail on that capped request, and I am using it because it is the same comparison. It was a small run on Groq's free tier, model openai/gpt-oss-20b, a streaming request for 1500 words, with a 64 token cap. Groq honoured the rewritten max_tokens, so BleedStop's own mid stream cut was not what stopped it. The README's reconciled cost at the policy rate fell from $0.0271 to $0.00151 per request. The actual bill was $0, because the tier was free. Over 20 pairs of calls that did not stream, direct p50 was 245.8 ms and the path through BleedStop was 234.2 ms. The README says the proxy overhead was not measurable over internet noise at that size.

The overhead estimate the README does stand behind is the mock upstream: npm run bench:openai, n=1000 calls that do not stream, added latency p50 1.53 ms and p99 1.86 ms. A separate mock flood in that same section, 40 requests for 20,000 words against an upstream that ignores max_tokens, cut 40 of 40 streams. At the README's policy rate of $0.01 per 1,000 tokens, with tokens approximated as characters divided by 4, that is $10.70 unprotected and $0.41 through the proxy. I am not calling that a customer saving. It is a mock.

The README's attack suite row, which I did not put on the case study, says a 400,000 character token flood was 100% blocked, a shared device farm 97%, a retry storm 94%, and normal users 0% blocked. The summary row does not give the sample size, so I am not inventing one. The file it points at is results/ in the repo.

The public demo is not that measurement

The live demo is bleedstop-demo.vercel.app. The README is explicit: it runs on Vercel Hobby functions, in memory, with no Redis and no Postgres. State is per function instance. Budgets, velocity windows and rate limits can reset on a cold start or when another instance handles the next call. The $5 cluster ceiling is not reliably enforced there. The measured numbers above come from the Compose and Kubernetes stack with Redis.

The README documents a demo key, demo-public-key, tenant demo, and this call. It says 500 estimated tokens come back ALLOW, and 400,000 estimated tokens come back DENY, against an unpaid token cap of 32,000.

curl -s -X POST https://bleedstop-demo.vercel.app/decide \
  -H "Content-Type: application/json" -H "X-API-Key: demo-public-key" \
  -d '{"actorId":"demo-user-1","action":"llm.call","paid":false,"estimatedTokens":500,"signals":{"deviceFingerprint":"fp-demo-1"}}'
Enter fullscreen mode Exit fullscreen mode

There is a project site at usmamalik345.github.io/bleedstop. The browser classifier on the demo does not protect the API. I cover that in the defence post.

Bugs that showed up under load

Running the stack found problems unit tests did not. The case study says I found and fixed eight, and kept the evidence in the repository. Four of them are:

  • Pods crash looping after loading 1.09 million rows at boot. Hydration is now skipped when Redis is authoritative, and otherwise capped to the top 50,000 rows by spend.
  • A Postgres writer that hung for good when the database went down. Connect and query timeouts fixed it.
  • Unhandled connection errors that crashed replicas when Postgres was stopped.
  • State kept per process, which split abuse signals across replicas behind a round robin load balancer. It now lives in Redis, and results match a single instance.

I am not naming the other four here, because the case study does not name them and I am not going to reconstruct them from memory.

What I will not claim

  • The golden set and the rules share an author, so the accuracy numbers are not field results. Recall on that set is 48.0%. Precision is 100%, with no benign false positives.
  • The case study lists these as not measured there: NetworkPolicy enforcement, multi node or cloud clusters, Postgres high availability, and real provider latency or pricing as a benchmark. The Groq run above is one small free tier check in the README, not that missing benchmark.
  • The Vercel demo is not the cluster that produced 5,433 decisions per second or the rolling restart result.
  • I do not have a production customer figure for money saved. Where the README prints a dollar amount, it is a policy rate or a local sample, and I have said which.

Source and results · Case study · Prompt injection layers · Hire me

About the author

Osama Malik (also written Usama Malik) is an AI agent engineer at Onlinx Solutions. This note is first hand from BleedStop. Home · Open source · Agent Receptionist · Hire

Originally published at https://osamamalik.me/blog/bleedstop-cost-and-security-guard.html

Top comments (0)