DEV Community: Tomas Rampas

The security check that never ran — shipping agentic-framework v4.1

Tomas Rampas — Wed, 05 Aug 2026 13:27:08 +0000

This week the framework caught one of its own security checks doing absolutely nothing.

The private-key scanner had been reporting “No private keys detected” on every single run. It looked healthy. It was green in CI. And it was completely broken.

That’s the kind of quiet failure that only shows up when the review chain is willing to dig past the green checkmark. It became the headline of v4.1.1. But the story actually started a few days earlier with a more ordinary kind of drift.

agentic-framework is a Claude Code plugin that turns Claude into an orchestrated team: 21 specialist subagents, 10 slash commands, 9 skills, and 5 hooks — including a peer-review Stop gate that refuses to let a session end when committed work hasn’t been through review. The repo also validates itself. A consistency battery keeps agent frontmatter, the model registry, MCP tool grants, and hook wiring in agreement. When that agreement breaks, the battery is supposed to notice.

v4.1.0 — cleaning up the drift

Most of the week went into fixing configuration that had slowly diverged from what the validators actually enforced.

I canonicalized effort: across all 21 agents, added mcpServers: to 17 of them, wrote explicit tools: allowlists for the seven language experts, and moved eight agents (the six language experts plus security-specialist and spec-compliance-reviewer) from sonnet to opus. The model registry in claude.json was updated in the same commit so the model-parity check stayed green.

To keep this from happening again, I added validator check 15. It now confirms that effort is a known tier, that every mcpServers entry exists in the MCP plugin manifest, that every mcp__<server>__ tool in an allowlist has its server declared, and that any serena tool grant includes serena’s two required bootstrap tools. The rule exists because a real version of this defect had already shipped mid-branch and taken two follow-up commits to clean up. The check came after the pain.

The other half of the release was hardening migrate-legacy, the script that moves a legacy ~/.claude install onto the plugin pipeline. It had two quiet bugs: it could delete tracked hook files during its own run, dirty the checkout, and still exit 0; and it dead-ended for anyone whose tracked CLAUDE.md contained personal edits. I added a live dirty-guard filtered by a protected-paths list (CLAUDE.md included) and by paths the run itself touches, changed the exit code to 2 with the offending paths named, and wrote four test topologies covering tracked hooks, a modified CLAUDE.md that must survive byte-exact, genuine dirt that must abort, and unpushed commits that must also abort.

I ran the migration on my own machine as the final test. The legacy checkout is gone. Everything is now plugin-served. The peer-review Stop gate is live — a real reviewer run wrote the per-session verdict marker the gate actually reads.

PR #40 landed as twelve commits. The adversarial peer review took three rounds. It first found a false positive in the new check, then found that the fix for that false positive had opened a recall hole — a spelling of the tools list that let the original defect slip back through. It only approved after the tokenizer treated every single-line spelling the same way.

v4.1.1 — the check that never ran

The smaller release fixed three things and uncovered one that had been broken the entire time.

The stats table in the team-presentation doc became a fully generated block; every row, including the validator check count, is now read at runtime from the check registry. The hand-maintained assertion-count row that had drifted three times in a single branch was deleted. Check 15 received a real frontmatter parser after review discovered the first draft silently skipped validation on malformed shapes — it now fails loudly on a missing or unterminated fence. And the secret scanner stopped flagging quoted values that start with $ (variable expansions and command substitutions), after a variable named tool_tokens burned a full CI round.

Then came the private-key rule.

The scanner contained this line:

grep -r --exclude-dir=.git "-----BEGIN.*PRIVATE KEY-----" .

grep treats the leading dashes as an option cluster and exits with status 2. The fail-open redirect treated that exit code as “clean.” So the script printed “No private keys detected” on every run — including runs that contained a planted PEM sitting in the tree.
The security-specialist found it. The peer-review-critic then reproduced the failure at root cause before approving the fix. The change was simple: add -e, and exclude the scanner’s own source now that the pattern actually matches its own lines. We verified it red with a planted key and green on a clean directory.
PR #41 was five commits.

Process note

Both pull requests travelled the same path: code-review-gatekeeper, then security-specialist for a focused pass, then peer-review-critic doing multi-round adversarial review that verifies its own findings with probes, then all four CI jobs green (validate + tests on Ubuntu, the lint job, and the macOS and Windows hook legs).
This framework’s own Claude Code agents wrote and reviewed these changes, through the framework’s own gates. No claim beyond that.
Current measured state: 15 validator checks, 88 assertions in the consistency suite, 103 in the migration suite, 34 in plugin-manifests. All exit 0.

Repo: https://github.com/tomas-rampas/agentic-framework
text

Framework that forces Claude Code to follow a proper Spec Build Review process

Tomas Rampas — Tue, 21 Jul 2026 07:18:59 +0000

For some time I was not happy with how Claude Code (and other coding agents) behave. They often start coding too early, skip proper requirements, or declare the work finished when it is still incomplete. So I decided to create something that puts real structure around the agent.

The result is this repository:
https://github.com/tomas-rampas/claude-agentic-framework

It is not a new AI tool. It is a configuration layer that sits on top of Claude Code CLI and tries to make the agent more disciplined.

The core idea

I designed a clear pipeline:

Spec → Build → Review

Everything starts with the /delegate command. If there is no specification yet, the system automatically runs /spec. In the specification phase the agent asks questions one by one, then scores its own understanding. Only when I approve the status, the work can continue to the build stage.

During /build the system first maps every requirement to concrete files. Then it starts implementing. After that comes the Review & Fix Loop. This loop can run maximum three times. If the reviewers are not happy, the agent has to fix the issues and go through the review again. Only when the final status becomes “built”, the process is considered done.

There is also a Quality Bar that shows the progress (from red to green) and a per-todo BDD style loop.

Two important mechanisms

I added a Review Chain with three different roles:

spec-compliance-reviewer — checks every requirement one by one
code-review-gatekeeper — looks at code quality before commit
peer-review-critic — final independent review (this is the last gate)

And a Self-Scoring Loop that the agent uses when it needs to improve its own non-code output. It follows five steps: Rubric → Score → Name Weak points → Rewrite → Rescore. It continues until the improvement becomes very small.

At the bottom of the process there is an Enforced Stop Gate. Two PowerShell hooks make sure the agent cannot just end the session if the peer review is not satisfied.

What is inside the framework

Currently it contains:

Around 20 specialized agents (language experts + roles like product-owner, system-architect, security-specialist, etc.)
Real enforcement hooks written in PowerShell
Commands such as /delegate, /spec, /build, /review-spec
An anti-drift system that tries to keep the agent registry, documentation and files consistent
Integration with several MCP servers

You still need Claude Code CLI. This project only configures and constrains it.

Why I made it

I wanted less “hope the agent does the right thing” and more real process. The agent should not decide by itself when the work is finished. There are hard limits and gates.

Of course the quality still depends a lot on how good the individual agent definitions are. But at least the structure is there.

If you use Claude Code and sometimes feel that the agent drifts or finishes too early, you can try this setup. Installation instructions are in the repository.

I’m still improving it, so feedback is welcome.

Claude Agentic Framework: Turning Claude Code into a Real Engineering Team with Verdict Gates and Self-Scoring Loops

Tomas Rampas — Sun, 19 Jul 2026 09:27:37 +0000

I have been working on this Agentic Framework for some time now, and I want to share where the Claude Agentic Framework stands today. It is a configuration layer that sits on top of Claude Code CLI and turns a single chat session into a disciplined team of 21 specialized agents. The agents do the work. Hooks enforce the quality. A consistency system keeps everything honest. Recent changes made the reviews much stronger and finally gave us a proper self-scoring loop for non-code work.
This is not another prompt collection. It is real enforcement.

From Chatbot to Agent

Most people still treat Claude like a chatbot. You ask a question, you get an answer, then you copy the code, run it, fix the errors yourself, and ask again. The model never sees the result of its own work.
An agent works differently. You give it a goal. It plans, edits files, runs commands, reads the output, and keeps going until the outcome is finished or it hits a real decision point. Context is gathered by the agent itself. Memory lives in CLAUDE.md and memory files. Tools are full — filesystem, shell, git, MCP servers. Feedback loops exist. Guardrails are hard.
The difference shows most clearly in context economics. A single long session fills the window with every file and log until early decisions fall out of memory. In the agentic setup the orchestrator stays lean. Heavy reading happens in disposable child windows that return only short summaries. During the last overhaul of this repository itself, the two review agents alone burned around 357k tokens inside their own windows. The coordinating session received two reports of roughly 1,500 tokens each.

Architecture in Practice

Everything starts from one objective. Routing (driven by CLAUDE.md rules, the /delegate command, and agent descriptions) sends the work to the right specialist. Skills and MCP servers support that specialist. When the specialist finishes, the code-review-gatekeeper looks at quality. Findings get fixed. Then the peer-review-critic runs an independent final review with a deliberately read-only toolset. A stop gate checks that a real review was recorded. Only then can the session end.

The framework recently went through its own pipeline. A multi-agent audit found the gaps, ten commits fixed them, both review agents signed off, three CI jobs passed, and the stop gate enforced the final review live.

Stronger Reviews That Actually Block

The biggest recent change is the verdict-aware peer-review stop gate. Before, the system only checked whether a review had happened. Now the peer-review-critic must emit a clear VERDICT: APPROVED or VERDICT: CHANGES_REQUIRED line. Recorder hooks log the decision. The stop gate refuses to let the session end if the verdict is not APPROVED. There is no quiet bypass.
New and improved commands support this:

/build runs an autonomous loop that builds exactly what the spec says and iterates with the spec-compliance-reviewer until it receives APPROVED (max three iterations).
/review-spec performs a manual conformance check.
/delegate now consumes specifications more cleanly and avoids earlier per-todo deadlocks.

The enforcement finally feels mature. It is no longer “was there a review?” It is “what was the actual verdict and did the system respect it?”

Self-Scoring Loop for Specs, Plans and Documents

Code already has the review gates. Everything else — specifications, architecture notes, project plans, proposals — used to rely on vague “make this better” instructions. The new self-scoring-loop skill changes that.
The loop is simple and strict:

Write a rubric first (5–7 weighted criteria that sum to 100). The rubric stays fixed for the whole loop.
Score the current version 0–100 against each criterion with a short justification.
Name the one or two weakest criteria and explain exactly why points were lost.
Rewrite only the weak parts. Keep what already scored well.
Rescore. Stop if the score improves by less than 3 points, reaches 90+, or hits three iterations.
Return the polished version together with the final scores and the trajectory (for example 68 → 81 → 87).

Guardrails are deliberate. No mid-loop rubric changes. Scoring without a visible rubric is invalid. The skill is advisory and never replaces the peer-review gates for code.
I use this myself before I present any non-code deliverable. The improvement is measurable and the stopping rules prevent endless churn.

Related Discipline in vouchfx

The same mindset shows up in my other project, vouchfx. There the “delegate runs” idea appears in a different form: declarative .e2e.yaml specs are compiled once into memory-safe C# delegates via Roslyn. Those delegates then execute against a full container topology orchestrated by .NET Aspire and Testcontainers. Verdicts are explicit (Pass, Fail, EnvironmentError, Inconclusive) and map cleanly into CI exit codes and reports. The self-scoring idea from the agentic framework and the strict delegate-based execution in vouchfx come from the same place — quality must be measurable and enforced, not hoped for.

Numbers and Getting Started

Today the framework ships with:

21 specialized agents across seven categories and three model tiers
4 registered hooks (one hard blocking gate)
Several operational skills including the self-scoring-loop
5 MCP servers
A consistency validator that derives truth at runtime and fails the build on any drift 3 CI jobs that include a Windows leg

Installation is straightforward:

git clone https://github.com/tomas-rampas/claude-agentic-framework ~/.claude
pwsh -NoProfile -File ~/.claude/scripts/install.ps1
bash ~/.claude/scripts/validate-consistency.sh   # expect RESULT: PASS

You need Claude Code CLI, PowerShell 7+, bash, jq, and the usual Node/uv pieces for the MCP servers.

Closing

I built this because I was tired of agents that sound confident and then quietly skip the hard parts. The combination of real stop gates, verdict recording, and a structured self-scoring loop for everything that has no test suite makes the system feel much closer to a real engineering team.
The framework is open. The presentation that started this write-up lives in the repository under docs/team-presentation.md. The Medium post that covered the review and scoring changes is already public. Feedback, issues, and better rubrics are welcome.
Agents write the code. Hooks make quality non-negotiable. The consistency system keeps the whole thing honest. That is the current state, and I plan to keep tightening it.

VouchFx: Write End-to-End Tests in YAML and Run Them Against Real Distributed Systems

Tomas Rampas — Thu, 16 Jul 2026 12:06:40 +0000

When I started working on distributed systems, one of the most painful parts was always end-to-end testing. You have services in different languages, they talk through REST, Kafka, databases, and webhooks, and you need to verify that a full business flow actually works. Most tools either force you to write everything in code or become very flaky in CI.

That’s why I built VouchFx — a declarative end-to-end testing platform that lets you describe tests in simple YAML, while still running them against real containers with proper orchestration.

How VouchFx works

The idea is quite simple but powerful. You write a test in .e2e.yaml. VouchFx validates it, compiles it to real C# code (only once), and then executes it against containers that are started and managed properly.

Here’s the flow:

This approach gives you several advantages:

Tests are written in readable YAML
You get the full power of C# when you need it (via script.csharp)
Real containers are used (not mocks)
The tool works with any language on the service side
Memory is handled safely and CI exit codes are meaningful

A simple example

Here’s what a basic test can look like:

metadata:
  name: simple-order-flow
  tags: [smoke]

environment:
  services:
    api:
      image: mycompany/orders-api:latest
    db:
      image: postgres:16

steps:
  - id: create-order
    type: http.rest
    method: POST
    url: http://{api}/orders
    body: { "customerId": 42, "amount": 99.5 }
    capture:
      orderId: $.id

  - id: verify-in-db
    type: db-assert.postgres
    connection: { service: db }
    query: SELECT status FROM orders WHERE id = {orderId}
    expected: { status: "CREATED" }

You describe what should happen. VouchFx takes care of starting the containers, waiting for them to be healthy, running the steps with retries, and collecting clear results.

When YAML is not enough

Most of the time, writing tests in declarative YAML is enough and actually very pleasant. However, sometimes a test scenario becomes really complex — for example when you need custom logic, loops, or advanced calculations.

In those cases you can use C# scripts directly. VouchFx supports the script.csharp step type, where you can write C# code inline or even load external .csx files. This gives you the full power of the C# language when you need it, while the framework still handles container orchestration, resilience, variable capturing, and reporting.

So you don’t have to choose between simple YAML and full code — you can mix both approaches in the same test suite.

Why I built it this way

I wanted something that feels natural to write, but still reliable when it runs in CI. Many testing tools either become too complex or produce flaky results because they don’t handle infrastructure properly.

With VouchFx you get:

Clear separation between Pass, Fail, Environment Error, and Inconclusive
Proper exit codes so your pipeline only breaks on real test failures
Good reporting (terminal, HTML, JUnit)
VSCode support with autocomplete and Test Explorer integration

It also works well in polyglot environments. Your services can be in Java, Python, Node.js, Go — VouchFx doesn’t care. It only cares about the seams between them.

Honest comparison with other tools

Here’s a comparison with some popular tools in this space. I tried to be honest about the strengths and weaknesses:

Feature	VouchFx	NBomber	Testcontainers + xUnit	Aspire.Hosting.Testing	Reqnroll (SpecFlow)	Karate
Main style	Declarative YAML	C# code	C# code	C# code	Gherkin + C#	Declarative (DSL)
Polyglot support (any service language)	Strong	Limited	Limited	.NET only	.NET only	Good
Real container orchestration	Built-in (Aspire + Testcontainers)	Manual	Good	Good	Manual	Manual
CI integration	Ready-to-use templates (GitHub + GitLab)	You build it yourself	You build it yourself	You build it yourself	You build it yourself	You build it yourself
Exit code handling	Clear (Fail / Env Error / Inconclusive)	Basic	Basic	Basic	Basic	Basic
Memory safety & performance	Verified (collectible ALC)	Good	Depends on your code	Depends on your code	Good	JVM-based
Extensibility	Provider SDK + Community hub	Plugins	Write your own	Limited	Step definitions	Custom steps (Java/Kotlin)
Best for	Polyglot distributed systems	Load & performance testing	.NET teams using containers	.NET Aspire projects	BDD-style .NET teams	Teams comfortable with JVM

Quick summary of VouchFx advantages:

Strong focus on polyglot systems (your services don’t need to be .NET)
Much better CI reliability thanks to proper exit codes and ready templates
Declarative YAML makes tests easier to read and maintain for complex flows
Built-in support for both simple YAML and powerful C# scripts when needed

Other tools can be better if you are doing heavy load testing (NBomber), pure BDD (Reqnroll), or if you are already deep in the Java ecosystem (Karate).

Getting started

Installation is simple:

dotnet tool install --global vouchfx --prerelease

Then you can run:

vouchfx run

There are also ready-to-use GitHub Actions and GitLab CI templates if you don’t want to set everything up yourself.

You can find the full documentation and examples here:

https://tomas-rampas.github.io/vouchfx/

Would love your feedback

VouchFx is still relatively new. I’ve been using it on my own projects and I think it solves some real pain points, but I would really like to hear from other people who do integration and end-to-end testing regularly.

If you work with distributed systems and have 30–60 minutes, I would appreciate it if you could try it on one of your flows (or just look at the samples) and tell me what you think — what feels good, what’s confusing, or what’s missing.

No pressure at all. Honest feedback is very valuable at this stage.

Source: https://tomas-rampas.github.io/vouchfx/

Samples: https://tomas-rampas.github.io/vouchfx-samples/

Thanks for reading!

Integration testing

Tomas Rampas — Thu, 16 Jul 2026 11:12:31 +0000