DEV Community: nujovich

Tracepath — I Built an AI Agent Audit Middleware in One Weekend

nujovich — Sun, 12 Jul 2026 20:43:27 +0000

This is a submission for Weekend Challenge: Passion Edition

What I Built

Tracepath is an open-source middleware that makes any AI agent — LangChain, CrewAI, AutoGen, or your own — auditable, traceable, and compliant with regulations like the EU AI Act and FINRA.

It's a full audit stack you can spin up with a single docker compose up. Every tool call your agent makes gets intercepted, signed with Ed25519, checked against OPA policies, stored in an immutable WORM log, and monitored in real-time for anomalies.

"Can you audit what your agents did yesterday?" — Tracepath answers that.

The dashboard in one screenshot

Five tabs: Audit (event trail), Incidents (real-time detection), Policies (versioned OPA rules with diff & rollback), Reports (FINRA & EU AI Act compliance), and Gemini (semantic incident classification).

🤍 The Passion Behind It

I didn't start in AI. I started in security.

When I began working in tech, I specialized in web application security — OWASP Top 10, XSS, CSRF, DoS attacks. I spent years thinking about how to control what applications do, how to prevent them from being abused, how to draw a boundary between "allowed" and "denied."

Then AI happened. And I fell in love with building agents — autonomous systems that can reason, use tools, and make decisions. But the security part of my brain wouldn't shut up:

How do you know what your agent did yesterday?
What if it calls a tool it shouldn't?
What if it burns through your budget?
How do you prove compliance to a regulator?

That's the passion. Marrying the two halves of my career: the security engineer who says "trust nothing, verify everything" and the AI builder who wants agents to be powerful and autonomous.

The EU AI Act was the spark. When I read Article 50 — the requirement for high-risk AI systems to maintain logs and enable human oversight — I realized: this is exactly what I spent years doing for web apps, but nobody's built it for AI agents yet.

So I built it. In one weekend. Because that's what passion does — it makes you forget to sleep.

Demo

Watch the 1-minute walkthrough

Quick tour of all five dashboard tabs: Audit, Incidents, Policies, Reports, and Gemini.

One command to start

git clone https://github.com/nujovich/tracepath.git
cd tracepath/docker
AUDIT_SIGNING_KEY=$(openssl rand -hex 32) docker compose up -d

Then open http://localhost:3000. You'll see:

Tab	What it shows
Audit	Every tool call signed, policy-checked, stored in PostgreSQL + MinIO WORM
Incidents	Real-time detection: denial spikes, budget overruns, suspicious patterns, rate limit breaches
Policies	Git-based OPA versioning with visual diff and one-click rollback
Reports	One-click FINRA and EU AI Act compliance reports
Gemini	Google Gemini 2.5 Flash classifying incidents — "this is a misconfiguration, not an attack"

What happens when you send audit events

curl -X POST http://localhost:9001/audit/step \
  -H "Content-Type: application/json" \
  -d '{
    "session_id":"demo","agent_id":"researcher","agent_type":"researcher",
    "step_number":1,"tool_name":"web_search",
    "tool_input":{"q":"EU AI Act Article 50"},
    "tool_output":{"results":3},"cost_cents":5,
    "timestamp":"2026-07-12T12:00:00Z"
  }'
# → {"status":"recorded","signature":"<ed25519>","policy_decision":{"allowed":true,"denials":[]}}

Every event is:

Signed with Ed25519 (cryptographic non-repudiation)
Checked against OPA WASM policies (allowlist, budget, rate limit) in <1ms
Stored in PostgreSQL (queryable) + MinIO S3 Object Lock (WORM, 365 days)
Streamed via NATS JetStream to the incident detector
Classified by Gemini 2.5 Flash for semantic severity refinement

Code

nujovich / tracepath

Agent Audit Stack — auditable multi-step AI agent middleware. Audit Gateway + Incident Response + Policy Evolution.

Tracepath

Auditable multi-agent middleware for AI governance. Make any agent framework (LangChain, CrewAI, AutoGen, LangGraph) compliant with EU AI Act, FINRA, and SOC2 — in a single docker compose up.

"Can you audit what your agents did yesterday?" Tracepath answers that with immutable Ed25519-signed logs, real-time policy enforcement, and a compliance dashboard.

🏆 Submitted to DEV Weekend Challenge: Passion Edition — Best Use of Google AI category

Quickstart

Prerequisites

Docker + Docker Compose
Python 3.10+ (for SDK)
Rust 1.80+ (for gateway development)

1. Start the stack

git clone https://github.com/nujovich/tracepath.git
cd tracepath/docker
AUDIT_SIGNING_KEY=$(openssl rand -hex 32) docker compose up -d

2. Open the dashboard

http://localhost:3000

Five tabs: Audit, Incidents, Policies, Reports, and Gemini.

3. Record your first

…

View on GitHub

The stack:

Rust (actix-web + OPA WASM + Ed25519) — the audit gateway
Python (NATS JetStream + aiohttp + Gemini) — the incident detector
React (TypeScript + Tailwind + shadcn/ui) — the compliance dashboard
OPA (Rego → WASM) — policy engine
PostgreSQL — queryable audit log
MinIO (S3 Object Lock) — WORM storage
NATS JetStream — event bus

How I Built It

Phase 1: Foundation (Saturday morning)

The core pipeline: intercept → sign → check → store. I built the Rust gateway with actix-web, embedded OPA WASM for policy evaluation, and wired Ed25519 signing. PostgreSQL for the queryable audit log, MinIO with S3 Object Lock for immutable WORM storage.

Phase 2: Incident Response (Saturday afternoon)

Streamed events from the gateway to NATS JetStream, then built a Python incident detector that watches for four anomaly types: denial spikes, budget overruns, suspicious patterns, and rate limit breaches. All surfaced in a React dashboard with real-time polling.

Phase 3: Policy Evolution (Saturday night)

Implemented git-based policy versioning. Every OPA policy change is a git commit. Built a visual diff viewer and one-click rollback in the dashboard. Added a replay engine that lets you replay historical events against a different policy version to answer: "What would have happened if this policy was active then?"

Phase 4: Compliance & Gemini (Sunday)

Generated FINRA and EU AI Act compliance reports as HTML. Integrated Google Gemini 2.5 Flash via OpenRouter as a semantic classifier — it takes threshold-triggered incidents and refines their severity by analyzing the context. A denial spike from a misconfigured policy gets downgraded from CRITICAL to WARNING. A real attack stays CRITICAL.

The Gemini integration caches classifications persistently, so the dashboard shows the reasoning behind every severity decision even between container restarts.

What I discovered about the EU AI Act

Article 50 of the EU AI Act requires high-risk AI systems to:

Automatically record events (logs) during operation
Enable human oversight through monitoring and intervention
Ensure traceability of AI system decisions

Tracepath implements all three. The FINRA report validates data integrity (Ed25519 signatures), record retention (WORM storage), and access controls (API authentication).

Prize Categories

I'm submitting to Best Use of Google AI.

Tracepath uses Gemini 2.5 Flash (via Google AI) as a semantic classifier that refines incident severity. Instead of blindly flagging every threshold breach as CRITICAL, Gemini analyzes the context:

Incident	Original Severity	Gemini Reasoning	Final Severity
6 `image_generate` denials in a session	CRITICAL	"All denials were for image generation, suggesting a misconfiguration rather than a malicious attempt"	WARNING
Budget exceeded by 2x in a single session	WARNING	"The tools used match a legitimate research workflow with no policy bypass attempts"	INFO

This is the difference between a noisy alert system and a useful one — and it's powered by Google AI.

What's Next

This is a weekend project, but it's not a toy. The roadmap:

[ ] Helm chart for Kubernetes deployment
[ ] TypeScript SDK with @audit decorator parity
[ ] PDF reports for FINRA/EU AI Act (currently HTML)
[ ] SOC2 certification readiness package
[ ] AWS Marketplace listing
[ ] Multi-tenant dashboard with organization scoping
[ ] Webhook alerts for incident notifications (Slack, PagerDuty)
[ ] Custom policy UI — write Rego rules directly in the dashboard

→ Star the repo to follow along.

Built with ❤️‍🔥 by Nadia Ujovich — a security engineer turned AI builder who still believes the best agents are auditable ones.

A plugin for Observability + Budget Guardrails built with Hermes Agent

nujovich — Sun, 31 May 2026 16:38:17 +0000

Challenge Entry for the Hermes Agent Challenge

🚀 What Problem Does This Solve?

AI agent deployments often suffer from two critical blind spots:

Cost visibility — you discover a $500 OpenAI bill at the end of the month with no clue which cron jobs or sessions caused it
Budget control — runaway loops or expensive model choices can drain your account before you notice

hermes-telemetry solves both by giving you real-time observability and automatic budget enforcement for Hermes Agent.

🎯 Why This Plugin Matters

Every production AI system needs observability and cost control. This isn't just a nice-to-have — it's essential infrastructure.

Before this plugin, Hermes users had no way to:

Track spending per cron job or messaging platform
Set budget limits that actually pause runaway processes
Compare cost efficiency between different models
Get real-time cost alerts before hitting billing limits

Now they can manage their AI spend like a modern SaaS — with dashboards, alerts, and automatic circuit breakers.

✨ Key Features

Real Usage Data (Not Estimates)

Captures actual token counts and costs returned by providers like OpenRouter, OpenAI, and Anthropic. No guesswork.

Multi-Level Budget Enforcement

Soft warnings at 80% of budget
Hard tool blocks at 100% (prevents new API calls)
Cron job pauses for automated workflows
Scope-specific limits (global, per-cron-job, per-platform)

Rich Analytics via Slash Commands

/stats — session performance, tool usage, cost breakdowns
/stats cron week — cron job cost comparison across time
/stats providers — which providers return real vs estimated data
/budget — current spending vs limits with visual indicators

Zero Model Awareness

Pure observability layer — captures everything through hooks without affecting model behavior or adding latency.

📊 Screenshots

Session Analytics (`/stats`)

Budget Status (`/budget`)

Cron Job Cost Comparison (`/stats cron week`)

Provider Analysis (`/stats providers`)

🧪 Proof of Concept: Real Data

I tested the plugin with three different models to validate pricing accuracy and budget enforcement:

Model	Cost per Test Run	Budget Behavior
`owl-alpha` (free)	$0.00	No limits triggered
`claude-sonnet-4-6`	$0.31	Soft warning at $0.001 limit
`claude-opus-4-7`	$2.23	Hard pause enforced ✅

Budget enforcement works. When I set a $0.001 daily limit and ran a cron job, it correctly paused at $0.18 spending. When I raised the limit to $2.00, jobs resumed normally.

Real provider data. OpenRouter returned actual token counts (Est% = 0%), not estimates. The plugin correctly captured and priced these.

🏗️ Technical Implementation

Hook Pipeline Architecture

on_session_start → pre_api_request → ★ post_api_request → post_tool_call
                                     │
                                     ▼
                               [capture usage]
                                     │
                                     ▼
pre_llm_call (budget check) → pre_tool_call (tool gate) → SQLite storage

Data Layer

SQLite WAL database — efficient, local, no external deps
Custom pricing.yaml — override provider rates for accurate cost calculation
budget.yaml configuration — flexible limits (daily/monthly, global/scoped)
94 comprehensive tests — full coverage of edge cases and enforcement logic

Provider Compatibility

Works with any provider that follows the Hermes Agent provider interface:

✅ OpenRouter (tested with real usage data)
✅ OpenAI (pricing table included)
✅ Anthropic (pricing table included)
✅ Custom providers (via pricing.yaml overrides)

🎯 Production Ready

This isn't a demo — it's production infrastructure. The plugin includes:

Error handling — graceful fallbacks when providers return no usage data
Hot-reload — update budgets via /budget set without restart
Concurrent safety — SQLite WAL mode handles multiple sessions
Memory efficiency — hook pipeline adds negligible overhead
Comprehensive logging — debug telemetry issues with structured logs

🚀 Installation & Usage

1. Install

cd ~/.hermes/plugins
git clone https://github.com/nujovich/hermes-telemetry.git
# Add 'hermes-telemetry' to plugins.enabled in config.yaml
# Restart gateway: hermes gateway restart

2. Configure Budget (Optional)

# Set daily budget
hermes> /budget set global daily 5.00

# Check status  
hermes> /budget

3. Monitor Usage

# Session stats
hermes> /stats

# Cron job breakdown
hermes> /stats cron week

# Provider analysis
hermes> /stats providers

That's it. The plugin immediately starts capturing usage data for all sessions and cron jobs.

🏆 Why this is a win-win

This plugin solves a universal need in AI systems — cost visibility and control. Every Hermes Agent deployment, from personal automation to enterprise cron jobs, benefits from this infrastructure.

It's not just useful, it's essential. Without budget controls, a misconfigured cron job with an expensive model can cost hundreds of dollars overnight. This plugin prevents that.

Real-world tested. I built, deployed, and validated this with actual usage data across multiple providers and models. It's not a concept — it's working infrastructure that saves money and provides operational insight.

Community impact. This sets a standard for observability in the Hermes ecosystem. Other plugin authors can build on these patterns, and users get immediate operational confidence.

📋 Technical Details

Repository: https://github.com/nujovich/hermes-telemetry
Documentation: Complete README with architecture, configuration, and troubleshooting
Tests: 94 passing tests covering all major functionality
License: MIT
Dependencies: PyYAML only (for config files)

👨‍💻 About the Author

I'm Nadia Ujovich.

I understand the operational challenges of running AI systems at scale, and I built this plugin to solve the observability gap I see in every deployment.

This plugin makes Hermes Agent production-ready for cost-conscious deployments. It's the infrastructure piece that every serious AI system needs, but few teams build themselves.

Give your agents the observability they deserve. Try hermes-telemetry today.

Made with ☕ for the Hermes Agent ecosystem

Building an Autonomous MCP Lead Generation System with Hermes Agent

nujovich — Mon, 25 May 2026 21:43:58 +0000

How I built a daily research pipeline with Hermes Agent that finds SMEs with legacy APIs, scores them on MCP readiness, and pre-audits their public API surface — all running autonomously on my own infrastructure.

The Problem: Finding Companies That Need MCP

MCP (Model Context Protocol) is the emerging standard for connecting legacy APIs, databases, and manual workflows to AI agents. But finding companies that actually need MCP integration — and that have the budget and readiness to act — is a manual, time-consuming sales process.

I'm constantly thinking about where to focus my help and services. My problem was simple: **I needed a way to find prospects daily without spending hours on LinkedIn and Reddit.

Enter Hermes Agent.

What I Built

A fully autonomous MCP Lead Generation & Pre-Audit System that runs daily on my own infrastructure:

Every day at 10:00 AM, Hermes Agent researches LinkedIn, Reddit, GitHub, Product Hunt, and tech blogs looking for SMEs that show signs of systems fragmentation
It scores each prospect against a 13-point MCP Readiness Checklist
For high-scoring prospects, it pre-audits their public API documentation to identify MCP integration opportunities
It emails me a structured report with company name, contact info, score, and specific MCP tools that could be built

The result is a daily pipeline of warm, pre-vetted leads — each one with a concrete MCP action plan ready to go.

The MCP Readiness Checklist

I designed a 13-question scoring system based on four dimensions:

1. Systems Fragmentation (3 pts)

Uses 5+ SaaS tools that don't talk to each other
Has proprietary/legacy internal APIs or siloed databases
Uses Zapier/Make for integrations (bonus if complaining about cost)

2. Manual Work & Friction (4 pts)

Someone copies/pastes data between systems regularly
Workflows that "should be automated" but aren't
Data conflicts between systems
A "data person" doing undocumented manual magic

3. Scale & Growth (3 pts)

Each new client adds manual work hours
Wants automation but no devs available
Planning 50%+ growth in the next 12 months

4. Budget & Readiness (3 pts)

Already spending €500+/month on SaaS
Has budget for consulting (€20-40k)
Has a CTO or VP Ops champion

8+ points = High priority. 5-7 = Warm. 0-4 = Skip.

How Hermes Agent Runs the Pipeline

The system is a Hermes Agent cron job — a self-contained prompt that the agent executes on schedule. Here's how it works:

Phase 1: Parallel Research

Hermes Agent launches parallel research tasks across multiple sources:

delegate_task(tasks=[
  {"goal": "Search LinkedIn for operations managers
            at SMEs mentioning 'tool sprawl'..."},
  {"goal": "Scan Reddit r/SaaS, r/smallbusiness
            for integration pain points..."},
  {"goal": "Check ProductHunt and tech blogs
            for companies with obvious API needs..."},
])

Phase 2: Scoring & Analysis

Each prospect is evaluated against the 13-point checklist. The agent reads company websites, checks their API docs, looks for GitHub repos, and builds a complete profile.

Phase 3: Pre-Audit (The Secret Weapon)

For companies with public API documentation or GitHub repos, the agent performs a deeper analysis:

API surface mapping — What endpoints exist? Auth model? Data formats?
MCP tool identification — Which endpoints would make good MCP tools?
Complexity estimation — Easy / Medium / Hard to build MCP wrappers
Security considerations — What needs attention before building

This transforms a cold lead into a warm prospect with a concrete MCP proposal ready to present.

Real Results: Two Pre-Audited Companies

On its first run, the system found and pre-audited two LATAM tech companies:

1. Cobre (Fintech, Colombia) — Score: 9/13

Signal: API-first payments platform with fragmented client systems.

The agent found their API documentation, analyzed ~50 REST endpoints across 18 categories (accounts, payments, FX, checkouts), and identified 5 specific MCP tools that could be built:

cobre_crear_contraparte — Register beneficiaries by country/rail
cobre_ejecutar_pago_local — Execute SPEI/ACH/Bre-B payments
cobre_crear_cotizacion_fx — Get FX quotes for cross-border
cobre_ejecutar_pago_internacional — Execute international payments
cobre_consultar_estado_pago — Track payment status

Key finding: Cobre already has an MCP server via Apidog — but a custom MCP server would be more powerful and fill their SDK gap (no Python/Node.js SDKs exist).

2. Truora (Identity Verification, Colombia) — Score: 7/13

Signal: API-first identity verification with ~120 endpoints across 5 API families.

The agent mapped their entire API surface: Checks API (background checks), Digital Identity (document/facial verification), WhatsApp Engagement (chatbots), Business Rules Engine, and Validator Suite. Suggested 5 MCP tools:

truora_check_background — Create background checks per country
truora_get_check_result — Get verification results with scores
truora_create_identity_process — Start digital identity validation
truora_send_whatsapp_message — Send WhatsApp via templates
truora_list_checks — Query verification history

Key finding: Truora only has iOS SDKs — no Python, Node.js, or Go tooling. An MCP server would fill a critical gap in their developer ecosystem.

The Architecture

The heart of the system is a self-contained Hermes Agent cron job prompt. Here's the structure:

# Create the cron job
hermes cron create \
  --schedule "0 10 * * *" \
  --prompt "research prospects..."

The prompt instructs Hermes Agent to:

Research — Use browser and web tools to find SMEs across LinkedIn, Reddit, GitHub, and blogs
Score — Apply the 13-point MCP Readiness Checklist
Pre-audit — For READY TO AUDIT companies, visit their API docs and GitHub repos
Compile — Format everything into a structured plain-text report
Deliver — Email via himalaya template send

cat << 'MAILDELIM' | himalaya template send
From: me@email.com
To: me@email.com
Subject: 🔌 MCP Prospects — [DATE]

═══════════════════════════════════════
1. AUDIT NOW — Company Name
   🔧 PRE-AUDIT: READY TO AUDIT
   API Docs: https://docs.example.com/
   ...
   🛠️ Suggested MCP tools:
   • tool_name — what it does
═══════════════════════════════════════
MAILDELIM

Why Hermes Agent Was the Right Tool

I chose Hermes Agent for three reasons:

Autonomous research — The agent can browse, read, analyze, and make decisions without me in the loop. It's not a script — it reasons about what it finds.
Tool ecosystem — Hermes Agent's built-in browser, web search, and terminal tools let it navigate real websites, parse API docs, and interact with GitHub — just like a human researcher would.
Self-hosted — This runs on my own infrastructure. No API costs, no data leaks to third parties. The prospect data stays mine.
Cron jobs — The built-in scheduler means the system runs itself. I wake up to a fresh lead report every morning.

What's Next

This system is already running daily. My roadmap includes:

Building the actual MCP servers — Taking the pre-audit output and building the MCP servers for the highest-priority prospects
Adding more research sources — AngelList, Crunchbase, and Y Combinator companies
Automated outreach — Integrating with LinkedIn DM or email to send personalized MCP proposals
Refining the scoring model — Learning from which leads convert and adjusting the checklist weights

Try It Yourself

If you're running Hermes Agent, you can build something similar in a few hours:

Define your scoring criteria
Write a research prompt (the agent handles the execution)
Create a daily cron job
Sit back and watch the leads roll in

The hardest part isn't the technology — it's knowing what signals to look for. That's where your domain expertise comes in.

Built with Hermes Agent — the open-source agentic system that runs on your own infrastructure.

🔗 **Full code and prompts:* github.com/nujovich/hermes-mcp-lead-gen*

Follow me on LinkedIn

hermesagentchallenge #devchallenge #agents #mcp #opensource

DEV Community: nujovich

Tracepath — I Built an AI Agent Audit Middleware in One Weekend

What I Built

The dashboard in one screenshot

🤍 The Passion Behind It

Demo

Watch the 1-minute walkthrough

One command to start

What happens when you send audit events

Code

nujovich / tracepath

Agent Audit Stack — auditable multi-step AI agent middleware. Audit Gateway + Incident Response + Policy Evolution.

Tracepath

Table of Contents

Quickstart

Prerequisites

1. Start the stack

2. Open the dashboard

3. Record your first

How I Built It

Phase 1: Foundation (Saturday morning)

Phase 2: Incident Response (Saturday afternoon)

Phase 3: Policy Evolution (Saturday night)

Phase 4: Compliance & Gemini (Sunday)

What I discovered about the EU AI Act

Prize Categories

What's Next

A plugin for Observability + Budget Guardrails built with Hermes Agent

🚀 What Problem Does This Solve?

🎯 Why This Plugin Matters

✨ Key Features

Real Usage Data (Not Estimates)

Multi-Level Budget Enforcement

Rich Analytics via Slash Commands

Zero Model Awareness

📊 Screenshots

Session Analytics (/stats)

Budget Status (/budget)

Cron Job Cost Comparison (/stats cron week)

Provider Analysis (/stats providers)

🧪 Proof of Concept: Real Data

🏗️ Technical Implementation

Hook Pipeline Architecture

Data Layer

Provider Compatibility

🎯 Production Ready

🚀 Installation & Usage

1. Install

2. Configure Budget (Optional)

3. Monitor Usage

🏆 Why this is a win-win

📋 Technical Details

👨‍💻 About the Author

Building an Autonomous MCP Lead Generation System with Hermes Agent

The Problem: Finding Companies That Need MCP

What I Built

The MCP Readiness Checklist

1. Systems Fragmentation (3 pts)

2. Manual Work & Friction (4 pts)

3. Scale & Growth (3 pts)

4. Budget & Readiness (3 pts)

How Hermes Agent Runs the Pipeline

Phase 1: Parallel Research

Phase 2: Scoring & Analysis

Phase 3: Pre-Audit (The Secret Weapon)

Real Results: Two Pre-Audited Companies

1. Cobre (Fintech, Colombia) — Score: 9/13

2. Truora (Identity Verification, Colombia) — Score: 7/13

The Architecture

Why Hermes Agent Was the Right Tool

What's Next

Try It Yourself

hermesagentchallenge #devchallenge #agents #mcp #opensource

Session Analytics (`/stats`)

Budget Status (`/budget`)

Cron Job Cost Comparison (`/stats cron week`)

Provider Analysis (`/stats providers`)