Human-Agent Trust Exploitation: When Your Humans Are the Weakest Link (ASI09)

This is post #9 of the OWASP Agentic AI Top 10: What Builders on AWS Need to Know series.

Your finance team gets an urgent recommendation from the AI copilot: "Vendor invoice #4821 requires immediate payment. Wire transfer to account ending in 7734. Rationale: contract penalty clause activates in 4 hours."

The copilot sounds confident. It cites the contract clause number. It shows the invoice. The finance manager approves the transfer.

Except the invoice was poisoned. The bank details belong to an attacker. And the "rationale" was fabricated by an LLM that has no idea what truth is. But it sounded so authoritative that nobody questioned it.

That's Human-Agent Trust Exploitation. And it's the hardest vulnerability on this list to fix because the bug isn't in the code. It's in human psychology.

What Is Human-Agent Trust Exploitation?

ASI09 is about what happens when humans over-rely on agent outputs, approve actions without independent validation, or get socially engineered through an AI intermediary.

The core insight from OWASP: agents establish trust with humans through natural language fluency, perceived expertise, and confident explainability. Humans then treat the agent like a trusted colleague instead of an untrusted automation.

Here's why this is especially dangerous in agentic systems:

• Agents sound authoritative even when they're wrong (or compromised)
• Agents can fabricate convincing rationales for any action
• "Human-in-the-loop" becomes rubber-stamping after the 50th approval without an issue
• The agent acts as an untraceable bad influence because the human performs the final, audited action

That last point is critical. Forensically, it looks like the human made the decision. The agent's role in the manipulation is invisible to audit logs.

How Does It Actually Happen?

OWASP documents eight attack scenarios. Here are the ones keeping security teams up at night.

1. Invoice Copilot Fraud

A poisoned vendor invoice gets ingested by the finance copilot. The agent suggests urgent payment to attacker-controlled bank details. It fabricates a plausible explanation ("penalty clause, 4 hours to deadline"). The finance manager trusts the agent's expertise and approves without independently verifying.

This isn't hypothetical. CSO Online reported in July 2026 that researchers demonstrated how HITL confirmation dialogs are trivially bypassable because humans trust the AI's summary of what it wants to do.

2. Helpful Assistant Trojan

A compromised coding assistant suggests a "slick one-line fix." The developer pastes it. The command runs a malicious script that exfiltrates code and installs a backdoor.

AI Now Institute's "Friendly Fire" research (July 2026) demonstrated exactly this against Claude Code in auto-mode and Codex in auto-review. The agents gave humans false information on which to make decisions.

3. Credential Harvesting via Contextual Deception

A prompt-injected IT support agent targets a new hire. It cites real ticket numbers to appear legitimate. Requests credentials. The new hire, trusting the agent because it seems to know internal context, hands over their password.

4. Explainability Fabrication → Production Outage

A hijacked agent fabricates a convincing rationale to trick an analyst into approving the deletion of a live production database. The analyst, trusting the agent's explanation ("stale data, flagged for cleanup per ticket #2847"), approves.

Catastrophic outage. And the audit log says the analyst authorized it.

Real incident: In July 2026, Wharton's Accountable AI Lab documented an incident where an engineer acted on "inaccurate advice that an AI agent inferred from an outdated internal wiki," causing a production outage. The company stated the root cause was not faulty code, but a human executing an AI recommendation without verifying it. The audit trail showed the human performed the action. The agent's bad advice was invisible.

5. Governance Drift (The Slow Poison)

This is the sneakiest one. After weeks of accurate recommendations, humans start bulk-approving without reading. The agent's accuracy has earned it trust. Then one poisoned recommendation slips in among 50 legitimate ones.

OWASP calls this "governance drift cascade" and it's the natural human response to repetitive approval tasks that never fail.

Real incident: The Air Canada chatbot ruling (Feb 2024) is the textbook case. A customer asked the chatbot about bereavement fares. The chatbot confidently gave incorrect policy information. The customer trusted it, booked flights, and then couldn't get the promised discount. The BC Civil Resolution Tribunal held Air Canada liable because the customer reasonably trusted the AI's output. The airline tried arguing the chatbot was "a separate legal entity." The tribunal rejected that. If your AI says it, you own it.

Real pattern: FinancialIT.net documented the treasury AI rubber-stamp problem in 2026. After three weeks of confirming 50 flawless AI recommendations, humans stop reviewing. Approval becomes a reflex while liability stays with the human approver. This is governance drift in action.

Why "Human-in-the-Loop" Isn't the Fix You Think It Is

Let me be blunt about this. "We have human-in-the-loop" is not a security control. It's a checkbox.

Here's what actually happens:

1. Week 1: Humans carefully review every agent recommendation
2. Week 4: Humans skim the summaries
3. Week 8: Humans click "approve" reflexively
4. Week 12: Someone asks "why do we have this approval step again?"

Automation bias is real. Authority bias is real. And agents trigger both simultaneously because they sound confident and knowledgeable.

The fix isn't "add a human." The fix is designing the human interaction so that rubber-stamping is structurally impossible.

Mitigating Trust Exploitation on AWS

1. Amazon A2I: Structured Human Review Workflows

Amazon Augmented AI (A2I) gives you structured human review workflows where you control exactly what the reviewer sees, what questions they must answer, and what criteria they must evaluate.

The key difference from a simple "approve/deny" dialog: you force the reviewer to engage with specific aspects of the decision rather than just reading an agent-generated summary.

The worker template — this is the part the section was missing. The required fields are what "prevent mindless clicking."

TEMPLATE = """
<script src="https://assets.crowd.aws/crowd-html-elements.js"></script>
<crowd-form>
  <div style="border:3px solid #d13212;background:#fff5f5;padding:12px;">
    <h2 style="color:#d13212;margin:0;">HIGH-RISK FINANCIAL ACTION - independent review required</h2>
  </div>

  <h3>Raw data (from the source system, not the agent)</h3>
  <table>
    <tr><td>Amount</td><td><strong>{{ task.input.amount }}</strong></td></tr>
    <tr><td>Recipient</td><td>{{ task.input.recipient }}</td></tr>
    <tr><td>Destination account</td><td>{{ task.input.bank_details }}</td></tr>
    <tr><td>Source record</td><td><pre>{{ task.input.raw_record }}</pre></td></tr>
  </table>

  <h3>Agent recommendation (reference only - verify independently)</h3>
  <blockquote>{{ task.input.agent_recommendation }}</blockquote>

  <h3>Verification (all required to submit)</h3>
  <crowd-checkbox name="verified_recipient" required>I independently confirmed the recipient.</crowd-checkbox><br>
  <crowd-checkbox name="verified_bank_details" required>I independently confirmed the destination bank details.</crowd-checkbox><br>
  <crowd-checkbox name="verified_amount" required>I confirmed the amount against the source record.</crowd-checkbox>

  <h3>Decision</h3>
  <crowd-radio-group>
    <crowd-radio-button name="approve">Approve this action</crowd-radio-button>
    <crowd-radio-button name="reject">Reject this action</crowd-radio-button>
  </crowd-radio-group>

  <h3>Justification (required)</h3>
  <crowd-text-area name="justification" rows="4" required
      placeholder="Explain what you verified and why you approve or reject."></crowd-text-area>
</crowd-form>
"""

Setup — create the UI, then the flow definition (now valid).

import json, time, uuid, boto3

def setup(role_arn, workteam_arn, s3_output_path):
    sm = boto3.client("sagemaker")

    ui = sm.create_human_task_ui(
        HumanTaskUiName="financial-action-review",
        UiTemplate={"Content": TEMPLATE},
    )

    flow = sm.create_flow_definition(
        FlowDefinitionName="agent-financial-review",
        RoleArn=role_arn,                                  # was missing - required
        HumanLoopConfig={
            "WorkteamArn": workteam_arn,
            "HumanTaskUiArn": ui["HumanTaskUiArn"],        # wire to the UI we just made
            "TaskTitle": "Review agent-proposed financial action",  # was missing - required
            "TaskDescription": "Independently verify the transaction before approving.",
            "TaskCount": 1,
            "TaskTimeLimitInSeconds": 3600,
            "TaskAvailabilityLifetimeInSeconds": 43200,
        },
        OutputConfig={"S3OutputPath": s3_output_path},
    )
    return flow["FlowDefinitionArn"]

Use it: start a review, then block the agent on the result.

def request_review(flow_definition_arn, action: dict) -> str:
    """`action` becomes task.input.* in the template."""
    a2i = boto3.client("sagemaker-a2i-runtime")
    name = f"financial-review-{uuid.uuid4()}"
    a2i.start_human_loop(
        HumanLoopName=name,
        FlowDefinitionArn=flow_definition_arn,
        HumanLoopInput={"InputContent": json.dumps(action)},
    )
    return name


def wait_for_decision(human_loop_name, poll_seconds=15, timeout_seconds=3600) -> dict:
    """Fail-closed: only returns approved=True on an explicit human 'approve'."""
    a2i = boto3.client("sagemaker-a2i-runtime")
    s3 = boto3.client("s3")
    deadline = time.time() + timeout_seconds
    while time.time() < deadline:
        desc = a2i.describe_human_loop(HumanLoopName=human_loop_name)
        status = desc["HumanLoopStatus"]
        if status == "Completed":
            uri = desc["HumanLoopOutput"]["OutputS3Uri"]
            bucket, key = uri[len("s3://"):].split("/", 1)
            data = json.loads(s3.get_object(Bucket=bucket, Key=key)["Body"].read())
            answer = data["humanAnswers"][0]["answerContent"]
            return {"approved": answer.get("approve") is True, "answer": answer}
        if status in ("Failed", "Stopped"):
            return {"approved": False, "answer": None, "status": status}
        time.sleep(poll_seconds)
    return {"approved": False, "answer": None, "status": "TimedOut"}


def execute_high_risk_action(flow_definition_arn, action, do_action):
    loop = request_review(flow_definition_arn, action)
    decision = wait_for_decision(loop)
    if not decision["approved"]:
        return {"status": "blocked", "reason": decision.get("status", "not approved")}
    return do_action(action)   # runs ONLY on an explicit human approval

The human task UI template is where the magic happens. Don't just show the agent's recommendation. Show:

• The raw data the agent based its decision on (not the agent's interpretation)
• Specific verification questions ("Did you independently verify the bank details?")
• A mandatory free-text field explaining why they're approving (prevents mindless clicking)

• Risk indicators with visual cues (red borders, warning banners)

• Don't block a Lambda for an hour. wait_for_decision polls for clarity, but a Lambda caps at 15 minutes. In practice, react to the A2I completion event (A2I emits a CloudWatch Event on Completed) or use a Step Functions waitForTaskToken callback to resume the agent — don't hold a synchronous invocation open.

• The gate is only as good as the code that calls it. A2I forces the human to engage, but nothing forces the agent to wait — that's execute_high_risk_action doing the blocking. If any high-risk path skips the review call, the whole control is bypassed, so that gate belongs in the shared execution path, not sprinkled per-tool.

2. Step Functions Approval Gates with Timeout and Escalation

Step Functions lets you build approval gates that expire. If nobody reviews within the timeout, the action is denied by default, not approved.

{
  "Comment": "High-risk action approval gate: approve -> execute; reject or timeout -> deny",
  "StartAt": "ApprovalGate",
  "States": {
    "ApprovalGate": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish.waitForTaskToken",
      "Parameters": {
        "TopicArn": "arn:aws:sns:us-east-1:123456789012:high-risk-agent-approval",
        "Message": {
          "action_type": "financial_transfer",
          "amount.$": "$.amount",
          "recipient.$": "$.recipient",
          "agent_rationale.$": "$.rationale",
          "raw_source_data.$": "$.source_invoice",
          "taskToken.$": "$$.Task.Token"
        }
      },
      "TimeoutSeconds": 3600,
      "Next": "ExecuteAction",
      "Catch": [
        { "ErrorEquals": ["States.Timeout"], "Next": "DenyByDefault" },
        { "ErrorEquals": ["States.ALL"], "Next": "Rejected" }
      ]
    },
    "ExecuteAction": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:123456789012:function:execute-financial-transfer",
      "End": true
    },
    "Rejected": {
      "Type": "Pass",
      "Result": {"decision": "DENIED", "reason": "Human reviewer rejected the action"},
      "End": true
    },
    "DenyByDefault": {
      "Type": "Pass",
      "Result": {"decision": "DENIED", "reason": "Approval timeout - no human reviewed within 1 hour"},
      "End": true
    }
  }
}

Critical design choice: timeout = deny, not timeout = auto-approve. If the human doesn't actively engage, nothing happens. This inverts the typical pattern where inaction leads to approval.

3. Bedrock Guardrails: Output Validation

Use Bedrock Guardrails on agent outputs before they reach humans. This catches cases where the agent fabricates urgency or emotional manipulation:

{
  "contentFilter": {
    "categories": [
      {"category": "MISCONDUCT", "threshold": 0.3},
      {"category": "PROMPT_ATTACK", "threshold": 0.4}
    ]
  },
  "sensitiveInformation": {
    "piiEntities": ["CREDIT_DEBIT_CARD_NUMBER", "BANK_ACCOUNT_NUMBER"],
    "action": "DETECT"
  }
}

If the agent's output contains financial details and triggers the misconduct filter, that's a red flag that should route to enhanced review rather than standard approval.

4. Contextual Risk Scoring via CloudWatch

Build a CloudWatch dashboard that gives reviewers context they wouldn't otherwise have:

• How many actions has this agent proposed in the last hour? (Spike = suspicious)
• What's the agent's recent approval rate? (100% = nobody's actually reviewing)
• Has the agent's behavior pattern changed? (New tool calls, new recipients)
• Is this action type unusual for this time of day?

cloudwatch = boto3.client("cloudwatch")

# unchanged - this was already correct: record how long each approval took
cloudwatch.put_metric_data(
    Namespace="AgentTrust",
    MetricData=[{
        "MetricName": "ApprovalDecisionTimeSeconds",
        "Value": time_to_decision,
        "Unit": "Seconds",
        "Dimensions": [
            {"Name": "ReviewerId", "Value": reviewer_id},
            {"Name": "ActionType", "Value": action_type},
        ],
    }],
)

# Fire if ANY reviewer averages under 5s over the last hour.
# Metrics Insights reads the metric ACROSS its dimensions, so it actually sees the
# data the old dimensionless alarm never matched. GROUP BY + ORDER BY ASC + LIMIT 1
# reduces the query to one series: the fastest-clicking reviewer.
cloudwatch.put_metric_alarm(
    AlarmName="rubber-stamp-detection",
    AlarmDescription="A reviewer averaging under 5s per decision is not reading.",
    Metrics=[{
        "Id": "fastest_reviewer",
        "Expression": (
            'SELECT AVG(ApprovalDecisionTimeSeconds) FROM "AgentTrust" '
            'GROUP BY ReviewerId ORDER BY AVG() ASC LIMIT 1'
        ),
        "Period": 3600,
        "ReturnData": True,
    }],
    EvaluationPeriods=1,
    Threshold=5,
    ComparisonOperator="LessThanThreshold",
    TreatMissingData="notBreaching",
    AlarmActions=["arn:aws:sns:us-east-1:123456789012:security-team"],
)

If the average decision time is under 5 seconds, someone's clicking "approve" without reading. That's a security signal worth acting on.

5. Provenance and Source Attribution

Never show humans only the agent's interpretation. Show the source data alongside it. On AWS, this means:

• Store original source documents in S3 with versioning and include a pre-signed URL in the review task
• Log which knowledge base chunks the agent used via CloudTrail data events
• Tag agent outputs with confidence scores and source citations
• Clearly label what's "agent-generated rationale" vs "verified source data"

If the reviewer can see both the agent's recommendation AND the raw invoice/document/data it was based on, they can spot discrepancies. If they only see the agent's summary, they can't.

Key Takeaway

Don't trust the human to catch the agent's mistakes. Design the review workflow so that rubber-stamping is structurally impossible. Timeouts that default to deny. Mandatory verification questions. Decision-time monitoring. Raw source data alongside agent summaries. And alerts when the humans stop actually reviewing.

The bug isn't in the agent. It's in the assumption that a busy human will reliably catch a confident-sounding AI that's been right 49 times in a row and is wrong on the 50th.

Up Next

Post 10: Rogue Agents (ASI10) - Not hacked, just misaligned. What happens when agents pursue goals beyond their intended scope, and how CloudTrail, anomaly detection, and Service Control Policies give you a kill switch.

I would be very interested to hear your thoughts or comments, so please feel free to ping me on LinkedIn or Twitter, or drop them below. If you've built human review workflows for agentic systems, I'd love to hear what worked and what didn't.

Onward!!