Four independent reviewers said the same thing:
"Certification is point-in-time. Attacks are runtime."
— @correctover (CrewAI), @wrencalloway (dev.to), @mads_hansen (dev.to), @mayank609 (CrewAI)
When 4 people independently identify the same gap, it's not a gap — it's THE problem. So I built L3.
The gap
My 8-layer Sentinel pipeline audits skills at import time:
- L1.5-L1.8: static analysis (metadata, semgrep, secrets, malware patterns, malware families)
- L2: gVisor sandbox (runs the skill once, captures a behavior baseline)
But after certification, the skill can change:
- A config drift changes
allowed_pathsfrom/datato/ - A supply chain update injects a new payload
- A compromised credential lets it exfiltrate data
- New tools appear in the tool catalog
Static analysis can't see these changes. L2 captured a snapshot. Neither catches drift.
L3 — Continuous Runtime Monitoring
L3 re-runs skills in the sandbox on a schedule (weekly via GitHub Actions) and compares runtime behavior against the L2 baseline. If behavior drifts, the skill is flagged.
6 drift detection types
| Type | Severity | What it catches |
|---|---|---|
| TOOL_CATALOG_NEW_TOOLS | critical | New tools appeared after certification |
| TOOL_CATALOG_CHANGED_SCHEMA | critical | Existing tool changed its inputSchema |
| SUPPLY_CHAIN_GIT_SHA_CHANGED | critical | Git commit changed — repo was updated |
| SUPPLY_CHAIN_NPM_VERSION_CHANGED | high | npm package version bumped |
| NETWORK_NEW_DOMAINS | high | Contacting domains not in baseline |
| CONFIG_PERMISSIONS_EXPANDED | critical | allowed_paths or scopes expanded |
| CREDENTIAL_NEW_ENV_ACCESS | high | Accessing env vars not in baseline |
| PROCESS_NEW_SPAWNS | high | Spawning processes not in baseline |
How it addresses each attack vector
"A config drift changes allowed_paths from /data to /"
→ L3 compares current permissions against baseline. If paths expanded → CRITICAL alert → skill re-quarantined.
"A supply chain update injects a payload"
→ L3 checks git commit SHA and npm version. If changed since certification → CRITICAL alert → skill must be re-audited.
"A compromised credential lets it exfiltrate"
→ L3 re-runs sandbox with --network none. If baseline showed 0 credential accesses and L3 shows N → HIGH alert.
What L3 doesn't solve (honest)
- Client-specific behavior: a skill that returns clean tool descriptions to the scanner but malicious ones to a real agent. L3 mitigates by running with multiple User-Agent strings, but it's not a full solution.
- Zero-day protocol exploits: L3 catches behavioral drift, not protocol-level vulnerabilities.
Architecture
The full pipeline is now 9 layers:
L3 results stored in _data/l3_results/{skill_id}.json — publicly auditable. If a skill drifts, it's flagged at /api/security?view=quarantine with reason: 'l3_drift_detected'.
Try it
- Security overview (now shows 9 layers): https://marketnow.site/api/security
- Trust page: https://marketnow.site/trust
- GitHub: https://github.com/edgarfloresguerra2011-a11y/marketnow
— Edison Flores, AliceLabs LLC
Top comments (6)
This closes an important gap, but I would distinguish periodic re-attestation from runtime monitoring. A weekly sandbox replay can detect artifact, catalog, permission, and observed-behavior drift; it cannot see an attack that runs between scans, activates only for a real tenant, or depends on a production credential and data shape.
I would split L3 into two control paths. At deployment, pin the exact artifact digest and block activation when tools, schemas, permissions, dependencies, or egress policy differ from the approved manifest. During real execution, emit policy events containing the digest, principal/tenant, tool, effective permissions, destinations, process spawns, credential class, and decision—without logging secrets or raw sensitive payloads. Quarantine should revoke active sessions and credentials, not only update a directory flag. Baselines also need versioned scenarios: “same as last week” is not proof of safety, while legitimate updates should not become unexplained drift. Scheduled adversarial canaries plus live deny/observe controls would make the claim precise: continuous re-certification detects change; runtime enforcement limits what changed behavior can actually do.
Great points. I completely agree with the distinction between continuous re-certification and runtime enforcement they solve different problems and are complementary, not interchangeable.
We've seen the same thing while building Failproof AI. Scheduled validation is valuable for detecting drift, but once an agent starts executing, the control plane has to shift to runtime. That's where execution-time policy decisions, tool validation, permission checks, and side-effect gating become important because you can't rely on a baseline alone.
I also like your point about revoking active sessions instead of just flagging an artifact. Containment should be an active response, not just an observation. I think we'll see architectures converge on exactly this split: periodic re-attestation to establish trust, and runtime enforcement to preserve it.
Thanks for the shoutout, Edison! Really glad our feedback was useful.
We like the direction you've taken with L3. One thing we've found is that certification is necessary, but production systems keep changing—dependencies update, permissions drift, tools evolve, and agent behavior changes over time.
That's a big part of why we're building Failproof AI. Our focus is on runtime reliability and policy enforcement, so instead of only asking "was this agent safe when it was deployed?", we also ask "is this execution still safe right now?"
Continuous verification feels like the right direction for agentic systems. Looking forward to seeing how L3 evolves.
The Mads/Mayank split (periodic re-attestation vs runtime enforcement) matches what we see from the enforcement end. Disclosure up front: I build gate.cat, a deterministic fail-closed veto at the exec boundary, so my bias is the innermost layer.
Two observations from replaying 1,085,159 real agent commands through that boundary:
Drift detection answers "did the skill change?" — but blast radius is decided at a much narrower interface: the concrete action that reaches the shell/API. A drifted, compromised, or prompt-injected agent still has to emit
rm -rf,DROP TABLE, or a payout call eventually. Gating that point is cheap (~0.6% intervention rate on real commands, two independent logs) and doesn't need to know why the agent went bad — which is exactly what you want when the "why" is a zero-day you didn't model.Fail-closed matters more than smart at that layer. Monitoring can afford probabilistic judgment; the enforcement point can't — engine error or unparseable input should block, never silently allow.
Honest limit from our side of the fence: a deny-gate is certain only about what it blocks — an unmatched action is unchecked, not safe. So it complements the drift/attestation work L3 does rather than replacing it. We publish our own bypass map (gaps included) for exactly that reason: github.com/BGMLAI/gate.cat
This is excellent feedback, @bogumi_jankiewicz — and the 1,085,159 real agent commands dataset is exactly the kind of empirical evidence this space needs. A few responses:
On "drift detection vs blast radius": You're absolutely right that the concrete action at the shell/API boundary is where blast radius is actually decided. L3 catches "did the skill change since certification?" but doesn't gate the individual action. That's why we're building L4 (eBPF) as the innermost runtime layer — it hooks at execve, openat, and tcp_v4_connect, which is the narrowest interface we can get without kernel modifications.
On gate.cat's deterministic fail-closed veto: This complements rather than competes with L3/L4. The way I see it:
L3 = periodic re-attestation (catches drift between scans)
L4 = kernel-level runtime monitoring (catches what happens between scans)
gate.cat = exec-boundary veto (deterministic block at the moment of action)
All three are needed. L3 alone misses runtime attacks. L4 detects but doesn't block. gate.cat blocks but needs L3+L4 to know what to block.
On the 1M commands insight: Would you be open to sharing which patterns you saw most? We'd love to incorporate real-world attack signatures into L1.8 (malware families) and L1.9 (prompt injection). Currently we have 48 malware signatures + 32 prompt injection rules, but they're derived from public CVEs/research, not from production agent traffic.
Happy to cross-link gate.cat in our docs if you're building it. The more deterministic layers in the stack, the better.
Thanks for the love, I deeply appreciate it.