Every platform engineer who runs agentic systems in production has hit the same wall: one agent misbehaves, and your only options are to kill the entire workflow or let it run and hope. Neither works. A red card, borrowed from sports but stripped of the drama, is a scoped, reversible control-plane action. It isolates the offending agent while the business workflow keeps moving. That's the operating model we're building here.
The operating problem
What does a policy violation actually look like when an agent goes off the rails? A customer-support agent pulls PII from a CRM table it shouldn't touch during a live chat. A code-generation agent in a CI/CD pipeline emits a hardcoded AWS secret into a pull request. A procurement planner loops on a vendor API, burning $400 in tokens in 20 minutes. These aren't hypotheticals. They're the daily reality of running multi-agent systems at enterprise scale.
The problem isn't that agents misbehave. The problem is that your enforcement options are binary. You either shut down the whole multi-agent system, which takes out every other agent and the business workflow they support, or you log the violation and hope it doesn't recur. Over-blocking causes outages. Under-blocking causes silent breaches. Both are expensive.
You need a graduated, identity-aware enforcement model. One that can issue a red card to a single agent, suspend its actions, preserve audit context, and enable safe reintegration, without treating every violation as a system-wide failure. That's the architecture we'll lay out.
The architecture that holds up
The architecture that holds up has five control points, and none of them are optional. Agent identity. Real-time instrumentation. A policy engine. Enforcement actions. An immutable audit log. Miss any one, and your red card becomes a guess.
Start with agent identity. Every agent invocation must carry a stable, unique identifier, a version, and a lineage of parent calls and tool invocations. Without that, you can't attribute a violation to a specific agent version or invocation. You're left guessing which agent did what, and enforcement becomes guesswork. This isn't a nice-to-have. It's the foundation. Link this to your existing instrumentation for explainability and audit.
Implement identity using short-lived, scoped credentials issued by a control-plane identity service. For example, each agent instance receives a JWT with claims for agent_id, version, workflow_id, and parent_agent_id. Propagate this token through every tool call via standard headers (e.g., Authorization: Bearer <agent-jwt>) or context propagation in OpenTelemetry baggage. For service-to-service calls, use mTLS with per-agent certificates from SPIFFE/SPIRE. The key is that identity is not just a label; it's a cryptographically verifiable claim that downstream services can validate and log. Without this, any enforcement action is guesswork.
Next, instrument real-time signals. Tool calls, token spend, data access patterns, output validation. You need a stream of events that tells you what each agent is doing right now, not what it did ten minutes ago. That stream feeds the policy engine.
Use a streaming platform like Kafka with a schema registry (Avro or Protobuf) to capture events. Each event must include the agent identity, a correlation ID for the workflow, a timestamp with microsecond precision, and a typed payload. For low-latency enforcement, process events with a stream processor (e.g., Kafka Streams, Flink) that can evaluate policies in under 100ms p99. If you need sub-second enforcement, avoid batch processing; use a push-based model where the agent runtime emits events directly to the policy engine via gRPC or HTTP, with a local buffer for offline resilience. Trade-off: push reduces latency but increases coupling; pull via Kafka decouples but adds at least tens of milliseconds of lag.
The policy engine evaluates those signals against policy violation classes. Scope violations: an agent acting outside its approved domain. Data access violations: touching PII, secrets, or regulated data without authorization. Tool use violations: calling an unapproved API or writing to a production database. Budget violations: token spend or API cost exceeding thresholds. Output quality violations: emitting malformed code, hallucinated facts, or unsafe content. Each class needs its own detection rules and severity mapping.
For deterministic policies (e.g., "agent X cannot access table Y"), use a rules engine like Open Policy Agent (OPA) with Rego policies. OPA can evaluate policies in microseconds and can be embedded as a sidecar or library. For anomaly-based detection (e.g., "unusual data access pattern"), use a streaming ML model that scores events and triggers when the score exceeds a threshold. Trade-off: rules are explainable and auditable but brittle; ML adapts to novel patterns but introduces false positives and requires training data. A hybrid approach, rules for hard constraints and ML for soft anomalies, works best in practice. Ensure the policy engine is stateless and horizontally scalable, with policy versioning and canary rollouts.
Map severity to graduated enforcement levels. A warning for a first-time, low-impact scope drift. A throttle for repeated budget overruns. A suspend, the red card, for a data access violation or a secret leak. A terminate for a confirmed malicious or unrecoverable agent. The key: the red card is scoped. It stops the offending agent, not the workflow. Route the workflow to a fallback agent or a human. Preserve the session, the context, and the business outcome.
Agent Enforcement State Machine
The state machine above shows the agent lifecycle: active, warned, throttled, red-carded, quarantined, reinstated. Each transition is a control-plane action with a clear trigger and a clear reversal path. No state is permanent except terminate, and even that should require a human approval.
When a violation signal fires, the policy engine makes a decision. It issues an enforcement action. It writes an immutable audit record linking the violation evidence, the enforcement decision, and the remediation steps. It routes the workflow to a fallback or a human. That sequence is the heart of the red card.
Red-Card Enforcement Sequence
The sequence diagram shows the handoffs: detection signal, policy engine decision, enforcement action, audit log write, fallback routing. Each step is synchronous and observable. If any step fails, you know exactly where.
Enforcement Level Comparison
The comparison table above lays out the graduated levels: warning, throttle, suspend, terminate. Each level has a trigger, a scope, a duration, and a reversal path. Warning is advisory. Throttle reduces rate or budget. Suspend stops the agent but keeps the workflow alive. Terminate removes the agent permanently. The red card is the suspend level, and it's the workhorse of scoped enforcement.
Design the red-card action as scoped quarantine. Stop the offending agent, not the workflow. Route to a fallback agent or a human. In the customer-support scenario, a PII access violation triggers a suspend on that support agent. The chat session doesn't drop. It routes to a human agent with full context. The offending agent sits in quarantine, its actions frozen, its audit trail intact.
Implement the suspension mechanism at the agent runtime level. When the policy engine issues a red card, it sends a cancellation signal to the agent's execution context, for example, a gRPC context cancellation, a message on a dedicated control channel, or a revocation of the agent's API tokens. In-flight tool calls are handled based on their idempotency and side effects: idempotent read-only calls are allowed to complete and their results discarded; non-idempotent calls with side effects (e.g., a database write) are terminated and, if possible, rolled back via compensating transactions. The agent's state, including its memory, conversation history, and intermediate results, is snapshotted and stored in a quarantine store (e.g., S3 with object lock or an append-only ledger). The workflow engine then routes to a fallback agent or human, passing the snapshot and the audit context. This ensures the business outcome is preserved while the offending agent is isolated.
Capture an immutable audit record. Link the violation evidence, the enforcement decision, and the remediation steps. That record is your compliance artifact. It's also your post-incident review input. Without it, you can't answer "what happened, who did it, what did we do, and did it work?"
Use an append-only log with hash chaining to guarantee immutability. Each audit record includes a hash of the previous record, the violation event (with evidence), the policy decision, the enforcement action, and a correlation ID that ties together all related events. Store this in a system like Kafka with log compaction disabled, or a dedicated audit service backed by a blockchain-like structure. Ensure the audit log is replicated across at least three availability zones and is tamper-evident. For compliance, provide a read-only API for auditors and integrate with your SIEM.
Define reintegration criteria. An agent doesn't stay red-carded forever. You need a path back. Post-incident review, root cause analysis, policy update, and a controlled test in a staging environment. Then reinstatement with a probationary throttle. If the agent violates again, escalate to terminate. This isn't punishment. It's operational hygiene.
Automate the reintegration pipeline. When an agent is red-carded, create a ticket in your incident management system. The post-incident review produces a root cause analysis and a policy update (e.g., a new OPA rule or a retrained ML model). The agent is then tested in a staging environment with a replay of the original violation scenario to verify the fix. If it passes, the agent is reinstated with a probationary throttle (e.g., 50% of normal rate or budget) for a defined period. During probation, any violation automatically escalates to terminate. This closed loop ensures that reintegration is not a manual afterthought but a controlled, auditable process.
Integrate human escalation for ambiguous or high-severity violations. A policy engine can't decide everything. A secret leak in a code-generation agent? That's a red card, no human needed. But a novel data access pattern that might be legitimate? Escalate to a human. The human reviews, decides, and the decision feeds back into the policy engine as a new rule. That loop keeps your enforcement from becoming brittle.
Implement human escalation via a review queue. When the policy engine encounters an ambiguous case (e.g., confidence score between 0.4 and 0.6), it pauses the agent and routes the decision to a human reviewer with full context: the violation event, the agent's history, and the proposed enforcement action. The human's decision is recorded and used to update the policy, either as a new rule or as labeled training data for the ML model. This feedback loop reduces false positives over time and ensures the system adapts to new patterns without manual policy writing.
Where teams usually fail
You've probably seen at least one of these failure modes in your own stack. Which one is costing you the most?
Over-blocking is the most common. One agent fails, and the team red-cards the entire workflow. The business grinds to a halt. A 40% false positive rate on red cards means your team starts ignoring alerts. That's worse than no enforcement at all. The fix: scope the red card to the agent, not the workflow. Use the failover and resilience patterns you already have for infrastructure.
Technically, implement scoped suspension by decoupling the agent's lifecycle from the workflow's lifecycle. Use a workflow engine (e.g., Temporal, Cadence) that treats each agent as a separate activity. When an agent is red-carded, cancel only that activity's context, not the entire workflow. The workflow engine then routes to a fallback activity or a human task. This requires the workflow to be designed with compensation and fallback paths from the start, retrofitting is painful.
Under-blocking is the silent killer. Repeated low-severity violations get logged and ignored. A support agent drifts out of scope five times in a week. Each time, it's a warning. No one acts. Then the sixth time, it exfiltrates a customer list. The fix: aggregate low-severity signals into a severity escalation. Three warnings in 24 hours becomes a throttle. Five becomes a suspend. Don't wait for the high-impact breach.
Implement aggregation in the stream processor. Maintain a sliding window (e.g., 24 hours) of violation events per agent. When the count of low-severity events exceeds a threshold (e.g., 3), automatically issue a throttle. When it exceeds a higher threshold (e.g., 5), issue a suspend. Use exponential backoff for thresholds to avoid flapping. This can be implemented with a simple stateful stream operator or a rules engine with windowing support.
Missing agent identity is the root cause of both. If you can't attribute a violation to a specific agent version or invocation, you can't scope enforcement. You're forced to over-block or under-block. The fix: make agent identity a first-class primitive. Every agent call carries a unique ID, a version, and a lineage. Link this to your data contracts for agentic AI so identity propagates through tool calls.
Enforce identity at the platform level. Require all agents to authenticate via mTLS or OAuth2 client credentials. Issue short-lived tokens with agent-specific scopes. Propagate identity through all tool calls using standard headers or context propagation. Log identity on every event. If an agent can't present a valid identity, deny it access to any tool or data. This makes identity non-negotiable.
Static rules cause false red cards. The same action allowed in dev but not prod. A policy threshold set for one workload but applied to another. The fix: context-aware policies. The policy engine evaluates the agent's environment, the data sensitivity, the user's role, and the business impact. A $50 token spend in a dev sandbox is fine. The same spend in a production procurement workflow is a throttle. Context is everything.
Implement context-aware policies by enriching events with environment metadata. When an agent emits an event, attach context such as environment (dev/staging/prod), data_classification (public/internal/confidential/restricted), user_role, and business_impact. The policy engine then evaluates rules that include these attributes. For example, a budget rule might be: if environment == "prod" and token_spend > $50 then throttle. Use OPA with data from a context service to evaluate these rules dynamically.
No reintegration path creates shadow agents. An agent gets red-carded, and the team just spins up a new one with a different name. The old agent sits in quarantine forever, and the new one has no history, no lineage, no audit trail. The fix: make reintegration a first-class workflow. Post-inc
Top comments (0)