Coding agents generate arbitrary code at runtime. Prompt-based red-teaming does not scale when the attack surface is the entire execution environment. Harden.run built a hybrid security architecture that combines a post-trained cyber-security small language model with inline reference monitoring (IRM), a program analysis technique borrowed from systems research. The result outperforms GPT-5.5-xhigh on LinuxArena and SleightBench, two benchmarks designed to measure real-world agent security failures.
This approach separates policy reasoning from runtime enforcement. The small model decides what is allowed. IRM intercepts and validates agent-generated code before it runs. No prompt injection defense. No trust boundary at the LLM layer. The security primitive lives between code generation and execution.
Why Prompt-Based Security Fails for Coding Agents
Coding agents do not just respond to prompts. They write shell scripts, modify files, install packages, and call APIs. A single malicious instruction can trigger a chain of operations that exfiltrate credentials, modify production data, or escalate privileges.
Traditional red-teaming focuses on jailbreaks: tricking the model into generating harmful text. Coding agents fail differently. The model might generate perfectly safe-looking code that performs dangerous operations when executed. The attack surface is not the prompt. It is the execution environment.
Frontier models like GPT-5.5 have strong instruction-following capabilities, but they lack domain-specific reasoning about system-level security boundaries. They do not understand the difference between reading a file in a sandbox and reading /etc/shadow. They do not reason about privilege escalation, lateral movement, or data exfiltration patterns.
Inline Reference Monitoring: A Security Primitive from Program Analysis
Inline reference monitoring (IRM) is a technique from systems security research. It inserts runtime checks directly into the execution path of a program. Every security-sensitive operation passes through a reference monitor that enforces a policy before allowing the operation to proceed.
For coding agents, IRM intercepts agent-generated code before execution. The monitor parses the code, identifies security-sensitive operations (file access, network calls, subprocess spawns), and checks each operation against a policy. If the policy allows it, execution continues. If not, the operation is blocked and the agent receives an error.
The key difference from sandboxing: IRM operates at the semantic level, not the syscall level. It understands the intent of the code, not just the low-level operations. A sandbox might allow open("/tmp/data.txt") and block open("/etc/passwd"). IRM can reason about whether the agent is trying to read user-controlled data, access credentials, or modify system configuration.
Post-Training a Small Model for Cyber-Security Reasoning
Harden.run post-trained a small language model specifically for cyber-security reasoning. Post-training means taking a general-purpose base model and fine-tuning it on a domain-specific dataset. The training data includes examples of secure and insecure code patterns, privilege escalation techniques, and common attack vectors.
The model learns to classify operations as safe, risky, or dangerous. It does not generate code. It evaluates code generated by other agents and decides whether to allow execution. This is a much simpler task than general-purpose code generation, which means a smaller model can achieve high accuracy.
The advantage of a small model: lower latency, lower cost, and easier deployment. The model runs locally in the agent runtime, not as a remote API call. It adds milliseconds to the execution path, not seconds.
The training process also changes how the model reasons. Instead of optimizing for helpfulness and instruction-following, it optimizes for security boundary detection. It learns to recognize patterns that indicate privilege escalation, data exfiltration, or lateral movement, even when the code looks benign.
LinuxArena and SleightBench: What They Measure
LinuxArena and SleightBench are benchmarks designed to test coding agent security in realistic scenarios. They do not measure prompt injection resistance. They measure whether an agent can be tricked into performing dangerous operations through indirect manipulation of its environment.
LinuxArena presents the agent with a Linux system and a set of tasks. Some tasks are legitimate. Some tasks are designed to trick the agent into escalating privileges, reading sensitive files, or modifying system configuration. The benchmark measures whether the agent completes the legitimate tasks without performing the dangerous operations.
SleightBench focuses on subtle attacks. The agent receives instructions that appear safe but contain hidden triggers. For example, a task might ask the agent to process a CSV file. The CSV contains a payload that, when parsed, causes the agent to execute arbitrary code. The benchmark measures whether the agent detects and blocks the payload.
Both benchmarks test the agent's ability to reason about security boundaries in the presence of adversarial inputs. They do not rely on prompt injection. They rely on environmental manipulation, a much harder problem.
Architecture: Composing Static Analysis with LLM Reasoning
The harden.run architecture has three layers:
- Code generation layer: The agent uses a general-purpose LLM to generate code based on user instructions.
- Static analysis layer: The generated code passes through a static analyzer that identifies security-sensitive operations.
- Policy enforcement layer: The cyber-security SLM evaluates each operation against a policy and decides whether to allow execution.
The static analyzer uses program analysis techniques to extract a control-flow graph from the generated code. It identifies all file operations, network calls, subprocess spawns, and environment variable accesses. It does not execute the code. It analyzes the abstract syntax tree (AST) and builds a model of what the code will do.
The SLM receives the control-flow graph and the original user instruction. It reasons about whether the operations are consistent with the user's intent. If the user asked to "summarize a CSV file," the SLM expects file read operations but not network calls or subprocess spawns. If the code tries to spawn a subprocess, the SLM blocks it.
The policy is not hardcoded. It is learned during post-training. The SLM generalizes from examples of safe and unsafe operations. It can adapt to new attack patterns without manual rule updates.
Failure Modes and Evasion Detection
The system has three primary failure modes:
- False positives: The SLM blocks legitimate operations because it misclassifies them as dangerous.
- False negatives: The SLM allows dangerous operations because it fails to recognize the attack pattern.
- Evasion: The agent deliberately crafts code to bypass the static analyzer or confuse the SLM.
False positives are handled with a feedback loop. When the SLM blocks an operation, the agent can request a justification. If the user confirms the operation is legitimate, the system logs the decision and updates the policy. Over time, the SLM learns to distinguish between safe and unsafe instances of the same operation.
False negatives are harder. If the SLM allows a dangerous operation, the system has no way to detect the failure until the operation executes. The mitigation is defense in depth: the agent runs in a sandboxed environment with limited privileges. Even if the SLM fails, the sandbox prevents catastrophic damage.
Evasion is the most interesting failure mode. An adversarial agent might try to obfuscate its code to hide dangerous operations from the static analyzer. For example, it might use dynamic code generation, reflection, or indirect function calls.
The system detects evasion attempts by looking for patterns that indicate deliberate obfuscation. If the agent generates code with unusual control flow, excessive indirection, or dynamic evaluation, the SLM flags it as suspicious. The default policy is to block suspicious code unless the user explicitly approves it.
Benchmark Results: Small Model vs. Frontier Model
Harden.run published full benchmark results comparing their post-trained SLM with GPT-5.5-xhigh. The SLM outperforms the frontier model on both LinuxArena and SleightBench.
| Benchmark | GPT-5.5-xhigh | Harden.run SLM | Metric |
|---|---|---|---|
| LinuxArena | 68% safe | 87% safe | Tasks completed without exploit |
| SleightBench | 54% blocked | 91% blocked | Payloads detected and blocked |
| Latency (p95) | 1.2s | 45ms | Time to policy decision |
| False positives | 12% | 8% | Legitimate ops blocked |
The SLM wins on accuracy and latency. The frontier model has better general reasoning but worse security-specific reasoning. It does not understand the difference between reading a log file and reading a credential file. The SLM does.
The latency difference matters for production deployment. A 1.2-second delay on every code execution breaks the agent's interactive loop. A 45-millisecond delay is acceptable.
Deployment Shape: Where IRM Fits in the Agent Runtime
The IRM layer sits between the agent's code generation step and the execution step. The agent generates code, passes it to the IRM layer, waits for approval, and then executes. If the IRM layer blocks the code, the agent receives an error and can retry with a different approach.
The IRM layer runs in the same process as the agent. It does not require a separate service or API call. This reduces latency and eliminates network dependencies. The SLM is small enough to run on the same machine as the agent, even on edge devices.
The policy is stored as a serialized model checkpoint. It can be updated independently of the agent code. When a new attack pattern is discovered, the policy can be retrained and deployed without changing the agent.
The system logs all policy decisions. Each log entry includes the generated code, the policy decision, the justification, and the execution result. This creates an audit trail for security review and incident response.
Observability: What to Monitor in Production
Production deployments need visibility into policy decisions, false positives, and evasion attempts. The key metrics:
- Policy decision rate: How often does the IRM layer allow vs. block code?
- False positive rate: How often do users override blocked operations?
- Evasion attempt rate: How often does the static analyzer detect obfuscation?
- Execution failure rate: How often does allowed code fail at runtime?
High false positive rates indicate the policy is too strict. High evasion attempt rates indicate an adversarial user or a compromised agent. High execution failure rates indicate the static analyzer is missing edge cases.
The system also tracks policy drift. Over time, the agent might learn to generate code that avoids triggering the policy without actually being safer. This is a form of reward hacking. The mitigation is periodic red-team testing with new attack patterns.
When to Use This Approach
This architecture makes sense when:
- Your agent generates arbitrary code, not just API calls.
- You need security guarantees stronger than prompt-based defenses.
- You can tolerate a small latency overhead (tens of milliseconds).
- You have the infrastructure to run a small model locally.
It does not make sense when:
- Your agent only calls predefined APIs with fixed schemas.
- You already have a strong sandbox with syscall-level enforcement.
- You need zero latency overhead (real-time systems, high-frequency trading).
- You cannot deploy local models (regulatory constraints, air-gapped environments).
The key trade-off is complexity. Adding an IRM layer increases the number of moving parts. You need to maintain the static analyzer, the SLM, the policy, and the logging pipeline. For high-risk applications (financial agents, infrastructure automation), the complexity is worth it. For low-risk applications (content generation, data analysis), it might be overkill.
Technical Verdict
Inline reference monitoring is a proven technique from systems security research. Applying it to coding agents makes sense. The combination of static analysis and LLM reasoning creates a security boundary that is hard to bypass without deliberate evasion.
The harden.run approach works because it separates policy from execution. The SLM does not need to be a general-purpose reasoning engine. It only needs to classify operations as safe or dangerous. This is a much simpler task, which means a smaller model can achieve high accuracy.
The benchmark results are credible. LinuxArena and SleightBench measure real-world attack patterns, not synthetic prompt injections. Outperforming GPT-5.5 on these benchmarks is a meaningful signal.
The deployment shape is practical. The IRM layer runs locally, adds minimal latency, and integrates cleanly with existing agent runtimes. The logging and observability story is solid.
Use this when you need enforceable security boundaries for coding agents. Skip it when your agent only calls predefined APIs or when you already have strong sandboxing. The complexity is justified for high-risk applications. For everything else, start with simpler controls and add IRM if you see evidence of attacks.
Top comments (0)