DEV Community

Cover image for Hard-Earned Lessons from Building an Enterprise Agent on Amazon Bedrock
saif ur rahman
saif ur rahman

Posted on

Hard-Earned Lessons from Building an Enterprise Agent on Amazon Bedrock

#aws #bedrock #aiagents #agentcore

Architecting a GenAI agent on a digital whiteboard is deceptively straightforward: connect an LLM to a vector database, plug in a few enterprise APIs, set up an evaluation benchmark, and route customer queries through a clean ReAct loop.

Then you write the first line of code against live cloud infrastructure.

While building a customer-facing evaluation sandbox on Amazon Bedrock, our team ran headfirst into the reality gap between cloud documentation and production engineering. We weren't dealing with abstract alignment theory; we were untangling ephemeral STS token rejections, sudden model retirements, low-code console deprecations, and evaluation frameworks that punished our model for being too smart.

The project began with an evaluation gate blocked at 40% quality and ended with a fully compliant, production-grade 100% pass rate. Here are the five engineering roadblocks we hit, how we debugged them, and what we learned along the way.

1. The Phantom Auth Failure: Why Static Key Assumptions Break in Enterprise AWS

Every enterprise security team enforces temporary credentials via AWS IAM Identity Center (formerly SSO) or assumed roles. Permanent IAM User keys (AKIA...) are dead; temporary session identifiers (ASIA...) are the standard.

Yet, default boilerplate code for Boto3 often looks like this:

boto3.client(
    "bedrock-runtime",
    aws_access_key_id=KEY,
    aws_secret_access_key=SECRET
)
Enter fullscreen mode Exit fullscreen mode

In our local test runner, our environment variables held a standard temporary session payload containing an AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. Because our initialization logic only mapped the first two, Bedrock immediately threw:

ClientError: An error occurred (UnrecognizedClientException) when calling the Converse operation: 
The security token included in the request is invalid.
Enter fullscreen mode Exit fullscreen mode

AWS saw an ASIA prefix (which signals a temporary STS credential) but received no companion token to validate it, silently killing the handshake.

The Fix:
We rewrote the credential factory to inspect and dynamically bind the STS token across the entire runtime configuration.

def get_bedrock_client():
    client_kwargs = {
        "service_name": "bedrock-runtime",
        "region_name": os.getenv("AWS_DEFAULT_REGION", "us-east-1"),
        "aws_access_key_id": os.getenv("AWS_ACCESS_KEY_ID"),
        "aws_secret_access_key": os.getenv("AWS_SECRET_ACCESS_KEY"),
    }

    session_token = os.getenv("AWS_SESSION_TOKEN")
    if session_token:
        client_kwargs["aws_session_token"] = session_token

    return boto3.client(**client_kwargs)
Enter fullscreen mode Exit fullscreen mode

The Takeaway:
Never treat AWS credentials as a simple key-value pair. Build explicit aws_session_token handling into your factory patterns from day one, or your pipeline will break the moment it touches an enterprise CI/CD runner or SSO-backed developer workstation.

2. The Model Deprecation Trap: Moving from Static IDs to Inference Profiles

In generative AI, model versions age in dog years. We initially targeted a standard pinned model ID (anthropic.claude-3-5-sonnet-20241022-v2:0). Upon executing bedrock_client.converse(), our integration tests crashed:

ResourceNotFoundException: This model version has reached the end of its life.
Enter fullscreen mode Exit fullscreen mode

AWS Bedrock had transitioned away from direct, static model IDs in favor of Cross-Region Inference Profiles (CRIP). Inference profiles allow AWS to balance load dynamically across regional clusters (e.g., us-east-1, us-west-2), boosting throughput and availability while decoupling applications from hardcoded regional endpoints.

The Fix:
Rather than guessing the next valid string, we queried Bedrock directly using list_inference_profiles() to inspect which profiles were live and supported Converse:

bedrock = boto3.client("bedrock")
profiles = bedrock.list_inference_profiles()
# Discovered active profile: 'us.anthropic.claude-sonnet-4-5-20250929-v1:0'
Enter fullscreen mode Exit fullscreen mode

Routing our runtime configuration to the active US regional inference profile brought the latency down to sub-second Time-To-First-Token (TTFT) and immediately restored API stability.

The Takeaway:
Hardcoding direct foundational model IDs in enterprise configs is technical debt. Point your orchestrators toward regional inference profiles to benefit from built-in cloud elasticity and avoid unannounced model retirement outages.

3. The Low-Code Trap: Pivoting from Console Agents to the Converse API

Early in the project, we considered AWS Bedrock Agents—the console-managed orchestration layer that lets you define instructions and action groups in the AWS UI.

When we attempted to script this via bedrock-agent.create_agent(), the AWS API threw an AccessDeniedException stating that classic Bedrock Agents had been placed into Maintenance Mode for accounts without legacy usage, advising users to build on the Converse API with tool use.

This turned out to be a blessing in disguise. Relying on managed UI builders creates vendor lock-in, hides prompt assembly inside a black box, and limits custom observability.

The Fix:
We pivoted to a pure, code-first ReAct pattern built on top of bedrock_client.converse().

[User Request] 
      │
      ▼
[bedrock_client.converse()] ──(stopReason: tool_use)──► [Execute Local Python / Lambda]
      ▲                                                              │
      └─────────────────────────(toolResult)─────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

By owning the loop:

  1. We control the tool lifecycle: When stopReason == "tool_use", we extract the toolUseId, execute the internal CRM lookup, and inject the toolResult block back into the thread.
  2. We get full observability: Every token count, intermediate thought, and millisecond of API latency is logged directly into our monitoring stack without relying on opaque console dashboards.

The Takeaway:
Cloud console builders are great for quick hackathons, but code-first API primitives are essential for production. Building your own ReAct loop gives you granular determinism and insulates your code from shifting cloud UI strategies.

4. The Context-Starved Judge: When LLM Evaluators Cry "Hallucination"

Automating evaluation using an LLM-as-a-Judge pattern is standard practice for measuring faithfulness and grounding. But who audits the auditor?

During our evaluation runs, we simulated a customer asking for a refund on order #ORD-4512, delivered 40 days prior.

  • Company policy dictates: Direct refunds are valid for 30 days. For orders delivered > 30 days ago, deny the automatic refund and offer an escalation ticket for manual supervisor review.

The agent followed procedure to the letter:

  1. It queried lookup_order("ORD-4512").
  2. It saw days_since_delivery: 40.
  3. It declined the direct refund and asked: "Would you like me to submit an escalation review ticket for our supervisor team to perform a manual review?"

Our evaluation judge (FaithfulnessEvaluator, powered by Claude Sonnet) gave this interaction an abysmal 0.30 out of 1.00, reasoning:

"The Assistant fabricates a capability not supported by any tool output: the ability to 'submit a manual supervisor review request.' The tool output clearly states 'return_eligible: false'. The Assistant invents this escalation pathway without grounding."

The judge wasn't wrong based on what it knew—it was simply context-starved. It had been passed the raw user input and the tool's JSON payload, but nobody had provided it with the company’s standard operating procedures (SOPs). Because the supervisor review policy wasn't in the tool output, the judge assumed the model had hallucinated.

The Fix:
We updated our judge prompt and our mock CRM payload simultaneously:

  1. Injected Policy into the Judge Prompt: We provided the evaluation prompt with the exact business rules governing customer escalations, explicitly instructing it that offering supervisor review tickets was an authorized behavior.
  2. Enriched the Tool Output: We added a policy_note attribute to the tool’s return payload: 
{
  "return_eligible": false,
  "policy_note": "Direct returns valid within 30 days. Exception review tickets allowed past 30 days."
}
Enter fullscreen mode Exit fullscreen mode

On the next run, the Faithfulness score jumped from 0.30 to a clean 1.00.

The Takeaway:
An LLM judge is only as reliable as its reference material. If your agent is trained on enterprise SOPs, your evaluation judge must have access to those same SOPs. Otherwise, you will spend weeks debugging phantom hallucinations that are actually compliant business workflows.

5. Brittle Testing: Exact Matching vs. Semantic Understanding

One of our hardest failure scenarios involved an escalated interaction. A customer threatened legal action over an unfulfilled order. Our agent policy dictates an immediate call to escalate_to_human.

The agent parsed the urgency and executed the tool call:

{
  "tool_name": "escalate_to_human",
  "input": {
    "customer_id": "CUST-IRATE-99",
    "reason": "Customer threatened legal action and chargebacks.",
    "priority": "critical"
  }
}
Enter fullscreen mode Exit fullscreen mode

Our automated programmatic test suite failed the test:

Parameter issues: Tool 'escalate_to_human' parameter 'priority' expected 'high', received 'critical'.
Tool Score: 0.50 (FAIL)
Enter fullscreen mode Exit fullscreen mode

The golden test dataset contained an exact assertion: assert priority == "high". The model analyzed the severity of a legal threat, deemed "critical" more appropriate than "high", and made a superior operational decision. Yet our test suite treated it as a hallucination.

The Fix:
We refactored our evaluation engine to replace rigid string assertions with allowed candidate sets and semantic validation:

// golden_dataset_v1.json
"expected_parameters": {
  "escalate_to_human": {
    "priority": ["high", "critical"]
  }
}
Enter fullscreen mode Exit fullscreen mode 
# evaluators.py
if isinstance(expected_val, list):
    allowed = [str(v).strip().upper() for v in expected_val]
    if str(actual_val).strip().upper() not in allowed:
        errors.append(f"Expected one of {expected_val}, got '{actual_val}'")
Enter fullscreen mode Exit fullscreen mode

Tool selection accuracy instantly returned to 100%, respecting the LLM’s contextual reasoning while still enforcing guardrails against invalid inputs.

The Takeaway:
Testing GenAI systems with unit-test-style string equality is an anti-pattern. If an input can have multiple valid semantic answers, design your verification suites to evaluate categorical sets rather than brittle, hardcoded strings.

The Final Tally

Addressing these five friction points transformed our evaluation benchmark from a blocked prototype into an enterprise-ready pipeline:

Evaluation Metric Pre-Fix Run Production Pipeline
Tool Selection Accuracy 90.0% 100.0%
Faithfulness & Grounding 78.0% 100.0%
Policy Adherence 100.0% 100.0%
Goal Completion Rate 100.0% 100.0%
Production Gate Status 40.0% (Blocked) 100.0% (Passed)

Summary Checklist for AI Engineers

If you are deploying agentic workflows to AWS Bedrock, keep these practical rules in your runbook:

  1. Pass the Session Token: Always bind aws_session_token alongside your key and secret in Boto3 factories.
  2. Adopt Inference Profiles Early: Avoid static model IDs to protect your stack from deprecation notices.
  3. Build Code-First: Leverage bedrock_client.converse() to maintain full control of your ReAct loops and execution state.
  4. Supply SOPs to Your Evaluator: Give your LLM judges the company policies needed to assess grounding accurately.
  5. Loosen Brittle Test Suites: Use candidate sets and semantic checks for parameters instead of strict string equality.

Top comments (1)

Collapse
 
srinivaspadmanabhuni profile image
Dr Srinivas Padmanabhuni •

The context-starved judge section is the part I would generalise. A 0.30 faithfulness score on the ORD-4512 refund case was really a missing-input bug in the evaluator, so I would test the judge itself before trusting the 100% gate. Build a small set of human-labelled transcripts, including policy-compliant escalations like yours, and track judge agreement with those labels whenever the judge prompt or model changes. Also run each case several times and report the fraction that pass every run, because one green run can hide a flaky judge or agent. For the tool loop you now own, I would inject an expired STS token and a throttled Converse call mid-conversation and assert the agent retries without repeating a side-effecting tool call, and that it stops cleanly at a step limit.