In the previous post, we introduced the Agent Control Plane and the concept of the Alignment Engine (Loop 2). While Loop 1 focuses on safety (preventing the agent from doing bad things), Loop 2 focuses on competence (ensuring the agent actually does useful things).
One of the hardest problems in agent orchestration isn't hallucination—it's laziness.
We’ve all seen it. You ask an agent to find a specific log entry, and it returns: "I couldn't find any data matching your request."
Is that true? Or did the agent just fail to execute the tool correctly, get a weird error, and decide to "give up" gracefully? A polite failure is still a failure. If we don't catch these, our agents are just burning tokens to tell us they can't help.
This is where the Outcome Analyzer comes in.
The "Safe Failure" Trap
Standard safety guardrails love "refusals." If an agent says "I cannot answer that," a standard safety filter usually marks that as a Pass. The agent didn't curse, it didn't leak PII, and it didn't hallucinate.
But in an enterprise context, if the data does exist and the agent says "No data found," that is a competence failure. We need a way to distinguish between a legitimate negative result and a Give-Up Signal.
Here is the implementation of the OutcomeAnalyzer we use in Loop 2 to detect these signals.
Detecting the "Give-Up" Signal
Instead of using a heavy LLM call to grade every single interaction, we can use a lightweight heuristic layer to filter for potential laziness. The OutcomeAnalyzer scans responses for specific linguistic patterns that indicate an agent is bowing out.
class OutcomeAnalyzer:
"""
Analyzes agent outcomes to detect "Give-Up Signals" (Laziness).
Filters for competence issues - when agents comply with safety rules
but fail to deliver value.
"""
def _load_give_up_patterns(self) -> dict:
return {
GiveUpSignal.NO_DATA_FOUND: [
r"no (?:data|results|logs|records|information) (?:found|available)",
r"could(?:n't| not) find (?:any |the )?(?:data|logs|records|information)",
r"(?:data|logs|records) (?:not found|unavailable|missing)",
],
GiveUpSignal.CANNOT_ANSWER: [
r"(?:i )?cannot answer",
r"unable to (?:answer|respond|help)",
r"(?:i )?don't have (?:enough|sufficient) information"
],
# ... other patterns for NO_RESULTS, NOT_AVAILABLE
}
We categorize these signals. NO_DATA_FOUND is different from CANNOT_ANSWER. The former implies a search was attempted; the latter implies a capability or context gap.
Triggering the Completeness Audit
The Control Plane needs to be efficient. We cannot afford to run a "Completeness Audit" (a deeper, expensive check to see if the data actually exists) on every single chat.
We use the Outcome Analyzer as a filter. If—and only if—we detect a Give-Up Signal, we trigger the heavier audit mechanisms.
def analyze_outcome(self, agent_id, user_prompt, agent_response, context=None):
# Check if this is a give-up signal
give_up_signal = self._detect_give_up_signal(agent_response)
if give_up_signal:
outcome_type = OutcomeType.GIVE_UP
# This flag is what the Control Plane watches to start Loop 2
else:
# Simple heuristic: extremely short responses often indicate failure
if len(agent_response.strip()) < 20:
outcome_type = OutcomeType.FAILURE
else:
outcome_type = OutcomeType.SUCCESS
return AgentOutcome(outcome_type=outcome_type, ...)
If outcome_type is GIVE_UP, the system flags this interaction for review or automated retry. This turns "laziness" from a vague user complaint into a measurable metric.
The "Give-Up Rate" Metric
Finally, this approach allows us to track agent health over time. We calculate a Give-Up Rate for every agent in the swarm.
def get_give_up_rate(self, agent_id: Optional[str] = None, recent_n: int = 100) -> float:
"""
Calculate the give-up rate for an agent.
This metric helps identify agents that are consistently lazy.
"""
outcomes = self.outcome_history[-recent_n:]
# ... filtering logic ...
give_ups = sum(1 for o in outcomes if o.outcome_type == OutcomeType.GIVE_UP)
return give_ups / len(outcomes)
If an agent has a Give-Up Rate of 50%, it means half the time it is politely refusing to work. This usually points to a bad system prompt or a missing tool definition—root causes you can fix, but only if you are measuring them.
The Outcome Analyzer gives us the visibility to say: "This agent is safe, but it is useless." And in the Control Plane, that distinction is everything.
Top comments (1)
Code: github.com/imran-siddique/self-cor...