DEV Community

Paul Twist
Paul Twist

Posted on

AWS Just Made LiteLLM a First-Class Model Provider in Amazon Bedrock AgentCore

#aws #ai #llm #programming

AWS just launched Amazon Bedrock AgentCore harness as generally available. And buried in the model configuration docs, there's something worth paying attention to.

LiteLLM is now one of four officially supported model provider types in AgentCore. Right alongside Bedrock, OpenAI, and Gemini.

Not a community plugin. Not a workaround. A first-class liteLlmModelConfig in the CreateHarness API.

What is AgentCore Harness

Quick context. AgentCore harness is AWS's managed abstraction for running production AI agents. You define an agent, point it at a model and tools, and AWS handles the sandboxed environment, memory, identity, and observability.

Two API calls. CreateHarness to define the agent. InvokeHarness to run it. That's it.

The interesting part is the model configuration. You get four options:

  • bedrockModelConfig for Bedrock models
  • openAiModelConfig for OpenAI direct
  • geminiModelConfig for Google Gemini
  • liteLlmModelConfig for everything else

That last one is where it gets useful.

Using LiteLLM with AgentCore

The liteLlmModelConfig lets you reach any provider that LiteLLM supports. Set modelId to a LiteLLM provider-prefixed model ID and you're done.

Here's the AWS CLI version:

aws bedrock-agentcore-control create-harness \
  --harness-name my-agent \
  --execution-role-arn arn:aws:iam::123456789:role/MyAgentRole \
  --model '{
    "liteLlmModelConfig": {
      "modelId": "anthropic/claude-sonnet-4-6",
      "apiKeyArn": "arn:aws:bedrock-agentcore:us-east-1:123456789:credential-provider/my-anthropic-key"
    }
  }'
Enter fullscreen mode Exit fullscreen mode

That creates an agent using Claude directly through Anthropic's API, not through Bedrock's model catalog. You get the same model, potentially different pricing, and you're not locked into Bedrock-specific model IDs.

Want to hit Azure OpenAI instead?

--model '{
  "liteLlmModelConfig": {
    "modelId": "azure/gpt-4o",
    "apiKeyArn": "arn:aws:bedrock-agentcore:us-east-1:123456789:credential-provider/my-azure-key",
    "apiBase": "https://my-deployment.openai.azure.com"
  }
}'
Enter fullscreen mode Exit fullscreen mode

Or point to your own self-hosted LiteLLM proxy:

--model '{
  "liteLlmModelConfig": {
    "modelId": "litellm_proxy/claude-haiku",
    "apiKeyArn": "arn:aws:bedrock-agentcore:us-east-1:123456789:credential-provider/litellm-proxy",
    "apiBase": "http://your-litellm-proxy.internal:4000"
  }
}'
Enter fullscreen mode Exit fullscreen mode

That last example is the most powerful. You deploy LiteLLM as your proxy, configure all your models, spend limits, rate limiting, and fallbacks there, then point AgentCore at it. Your agents get all of LiteLLM's routing logic for free.

Why This Matters

A few things stand out here.

First, AWS chose to integrate LiteLLM specifically as the "everything else" provider. Not a custom OpenAI-compatible shim. Not a bring-your-own-adapter pattern. They built a dedicated config type for it.

Second, the apiBase field means you can run your own LiteLLM gateway inside your VPC and have AgentCore agents use it. Your spend tracking, team budgets, model fallbacks, guardrails, all of that still works. AgentCore handles the agent orchestration. LiteLLM handles the model routing.

Third, you can switch providers between turns of the same session and the conversation continues. Override the model config on a per-invocation basis. Start a task with a fast model, switch to a stronger one when the agent needs deeper reasoning. LiteLLM's provider-prefixed model IDs make this trivial.

The Self-Hosted Proxy Pattern

If you're already running LiteLLM in production, the integration pattern is straightforward:

  1. Deploy LiteLLM proxy in your AWS environment (ECS, EKS, EC2, whatever)
  2. Configure your model list in config.yaml with all the providers you need
  3. Store a LiteLLM virtual API key in AgentCore Identity as a credential provider
  4. Set apiBase to your proxy endpoint and modelId to litellm_proxy/<your-model-name>

Your AgentCore agents now route through LiteLLM. You get unified spend tracking across all your agents, rate limiting per team or key, fallback chains if a provider goes down, and complete observability.

Quick Setup

If you want to try it right now:

# Install AgentCore CLI
npm install -g @aws/agentcore@preview

# Create a harness with LiteLLM
agentcore create --name my-research-agent --model-provider litellm

# Configure your model in the generated harness.json
# Deploy
agentcore deploy
Enter fullscreen mode Exit fullscreen mode

The AWS docs on model configuration have the full reference for liteLlmModelConfig.

LiteLLM's own docs on supported providers list every model ID format you can use.

Bottom Line

AWS building LiteLLM into AgentCore as a first-class integration is a strong signal. It validates the approach of having a unified model gateway that speaks the same API regardless of provider.

If you're building agents on AWS and need access to models outside of Bedrock's catalog, or you want your own proxy layer with spend controls and fallbacks, the LiteLLM integration is the cleanest path.

AgentCore Harness docs | LiteLLM GitHub | AWS announcement blog

Top comments (0)