Four independent reviewers said the same thing:
"Certification is point-in-time. Attacks are runtime."
— @correctover (CrewAI), @wrencalloway (dev.to), @mads_hansen (dev.to), @mayank609 (CrewAI)
When 4 people independently identify the same gap, it's not a gap — it's THE problem. So I built L3.
The gap
My 8-layer Sentinel pipeline audits skills at import time:
- L1.5-L1.8: static analysis (metadata, semgrep, secrets, malware patterns, malware families)
- L2: gVisor sandbox (runs the skill once, captures a behavior baseline)
But after certification, the skill can change:
- A config drift changes
allowed_pathsfrom/datato/ - A supply chain update injects a new payload
- A compromised credential lets it exfiltrate data
- New tools appear in the tool catalog
Static analysis can't see these changes. L2 captured a snapshot. Neither catches drift.
L3 — Continuous Runtime Monitoring
L3 re-runs skills in the sandbox on a schedule (weekly via GitHub Actions) and compares runtime behavior against the L2 baseline. If behavior drifts, the skill is flagged.
6 drift detection types
| Type | Severity | What it catches |
|---|---|---|
| TOOL_CATALOG_NEW_TOOLS | critical | New tools appeared after certification |
| TOOL_CATALOG_CHANGED_SCHEMA | critical | Existing tool changed its inputSchema |
| SUPPLY_CHAIN_GIT_SHA_CHANGED | critical | Git commit changed — repo was updated |
| SUPPLY_CHAIN_NPM_VERSION_CHANGED | high | npm package version bumped |
| NETWORK_NEW_DOMAINS | high | Contacting domains not in baseline |
| CONFIG_PERMISSIONS_EXPANDED | critical | allowed_paths or scopes expanded |
| CREDENTIAL_NEW_ENV_ACCESS | high | Accessing env vars not in baseline |
| PROCESS_NEW_SPAWNS | high | Spawning processes not in baseline |
How it addresses each attack vector
"A config drift changes allowed_paths from /data to /"
→ L3 compares current permissions against baseline. If paths expanded → CRITICAL alert → skill re-quarantined.
"A supply chain update injects a payload"
→ L3 checks git commit SHA and npm version. If changed since certification → CRITICAL alert → skill must be re-audited.
"A compromised credential lets it exfiltrate"
→ L3 re-runs sandbox with --network none. If baseline showed 0 credential accesses and L3 shows N → HIGH alert.
What L3 doesn't solve (honest)
- Client-specific behavior: a skill that returns clean tool descriptions to the scanner but malicious ones to a real agent. L3 mitigates by running with multiple User-Agent strings, but it's not a full solution.
- Zero-day protocol exploits: L3 catches behavioral drift, not protocol-level vulnerabilities.
Architecture
The full pipeline is now 9 layers:
L3 results stored in _data/l3_results/{skill_id}.json — publicly auditable. If a skill drifts, it's flagged at /api/security?view=quarantine with reason: 'l3_drift_detected'.
Try it
- Security overview (now shows 9 layers): https://marketnow.site/api/security
- Trust page: https://marketnow.site/trust
- GitHub: https://github.com/edgarfloresguerra2011-a11y/marketnow
— Edison Flores, AliceLabs LLC
Top comments (4)
This closes an important gap, but I would distinguish periodic re-attestation from runtime monitoring. A weekly sandbox replay can detect artifact, catalog, permission, and observed-behavior drift; it cannot see an attack that runs between scans, activates only for a real tenant, or depends on a production credential and data shape.
I would split L3 into two control paths. At deployment, pin the exact artifact digest and block activation when tools, schemas, permissions, dependencies, or egress policy differ from the approved manifest. During real execution, emit policy events containing the digest, principal/tenant, tool, effective permissions, destinations, process spawns, credential class, and decision—without logging secrets or raw sensitive payloads. Quarantine should revoke active sessions and credentials, not only update a directory flag. Baselines also need versioned scenarios: “same as last week” is not proof of safety, while legitimate updates should not become unexplained drift. Scheduled adversarial canaries plus live deny/observe controls would make the claim precise: continuous re-certification detects change; runtime enforcement limits what changed behavior can actually do.
Great points. I completely agree with the distinction between continuous re-certification and runtime enforcement they solve different problems and are complementary, not interchangeable.
We've seen the same thing while building Failproof AI. Scheduled validation is valuable for detecting drift, but once an agent starts executing, the control plane has to shift to runtime. That's where execution-time policy decisions, tool validation, permission checks, and side-effect gating become important because you can't rely on a baseline alone.
I also like your point about revoking active sessions instead of just flagging an artifact. Containment should be an active response, not just an observation. I think we'll see architectures converge on exactly this split: periodic re-attestation to establish trust, and runtime enforcement to preserve it.
Thanks for the shoutout, Edison! Really glad our feedback was useful.
We like the direction you've taken with L3. One thing we've found is that certification is necessary, but production systems keep changing—dependencies update, permissions drift, tools evolve, and agent behavior changes over time.
That's a big part of why we're building Failproof AI. Our focus is on runtime reliability and policy enforcement, so instead of only asking "was this agent safe when it was deployed?", we also ask "is this execution still safe right now?"
Continuous verification feels like the right direction for agentic systems. Looking forward to seeing how L3 evolves.
Thanks for the love, I deeply appreciate it.