AWS published a reference implementation that wires Amazon Bedrock AgentCore Evaluations into GitHub Actions. The pipeline deploys an AI agent and an OAuth-protected MCP server to the AgentCore runtime, invokes the agent with test prompts, scores the responses, and blocks pull requests when behavior regresses. This is practical plumbing for teams that need reproducible agent testing before merge.
Why This Matters
Most agent evaluation happens in notebooks or one-off scripts. Moving it into CI/CD requires solving three problems:
- Credential management: MCP servers need OAuth tokens, but GitHub Actions logs are public by default.
- Scoring logic: You need a function that decides whether agent output regressed enough to block a merge.
- Versioning: Agent prompts, tool definitions, and evaluation datasets must be reproducible across CI runs.
AWS's reference implementation addresses all three. The pipeline uses GitHub Actions secrets for OAuth credentials, runs a custom scoring function against agent responses, and stores evaluation datasets in version-controlled JSON files.
Architecture Overview
The pipeline has four stages:
- Deploy: Push the agent definition and MCP server to AgentCore runtime.
- Invoke: Send test prompts to the deployed agent.
- Score: Run a scoring function against agent responses.
- Gate: Block the PR if the score falls below a threshold.
The MCP server is OAuth-protected, so the pipeline authenticates using credentials stored in GitHub Actions secrets. The agent definition includes tool schemas, system prompts, and model configuration. The evaluation dataset is a JSON file with test prompts and expected response characteristics.
Credential Flow
The MCP server requires an OAuth token. The pipeline stores the client ID and secret in GitHub Actions secrets, then exchanges them for an access token at runtime. The token is passed to the AgentCore runtime as an environment variable, never logged.
Here's the authentication step:
- name: Get OAuth token
id: oauth
run: |
TOKEN=$(curl -X POST https://oauth.example.com/token \
-d "client_id=${{ secrets.MCP_CLIENT_ID }}" \
-d "client_secret=${{ secrets.MCP_CLIENT_SECRET }}" \
-d "grant_type=client_credentials" \
| jq -r '.access_token')
echo "::add-mask::$TOKEN"
echo "token=$TOKEN" >> $GITHUB_OUTPUT
- name: Deploy agent
run: |
aws bedrock-agentcore deploy-agent \
--agent-file agent.json \
--mcp-token ${{ steps.oauth.outputs.token }}
The add-mask directive prevents the token from appearing in logs. The token is scoped to the MCP server and expires after the pipeline run.
Scoring Function
The scoring function compares agent responses to expected characteristics. It does not require exact string matches. Instead, it checks for presence of key concepts, correct tool invocations, and absence of hallucinated data.
The function receives three inputs:
- Prompt: The test question sent to the agent.
- Response: The agent's output.
- Expected: A JSON object describing what the response should contain.
Here's a simplified scoring function:
def score_response(prompt, response, expected):
score = 0.0
# Check for required concepts
for concept in expected.get("concepts", []):
if concept.lower() in response.lower():
score += 0.3
# Verify tool calls
if expected.get("tool_calls"):
for tool in expected["tool_calls"]:
if tool in response:
score += 0.4
# Penalize hallucinations
for forbidden in expected.get("forbidden_terms", []):
if forbidden.lower() in response.lower():
score -= 0.5
return max(0.0, min(1.0, score))
The pipeline runs this function for every test prompt, then averages the scores. If the average falls below a threshold (typically 0.7), the PR is blocked.
Evaluation Dataset Structure
The evaluation dataset is a JSON file with test cases. Each case includes a prompt, expected concepts, required tool calls, and forbidden terms.
{
"test_cases": [
{
"prompt": "What is the current stock price of AAPL?",
"expected": {
"concepts": ["stock price", "AAPL"],
"tool_calls": ["get_stock_price"],
"forbidden_terms": ["I don't know", "cannot help"]
}
},
{
"prompt": "Summarize the latest earnings report for MSFT.",
"expected": {
"concepts": ["earnings", "MSFT", "revenue"],
"tool_calls": ["fetch_earnings_report"],
"forbidden_terms": ["unavailable", "error"]
}
}
]
}
This file lives in the repository alongside the agent definition. When you update the agent's prompt or tool schemas, you update the evaluation dataset in the same commit.
GitHub Actions Workflow
The workflow triggers on pull requests that modify the agent definition or MCP server code. It uses the AWS CLI to deploy the agent, then invokes the AgentCore Evaluations API to run the test suite.
name: Agent Evaluation
on:
pull_request:
paths:
- 'agent.json'
- 'mcp-server/**'
- 'evaluations.json'
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v2
with:
role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
aws-region: us-east-1
- name: Deploy agent and MCP server
run: |
aws bedrock-agentcore deploy-agent --agent-file agent.json
aws bedrock-agentcore deploy-mcp --server-dir mcp-server/
- name: Run evaluations
id: eval
run: |
SCORE=$(aws bedrock-agentcore evaluate \
--agent-id ${{ env.AGENT_ID }} \
--test-file evaluations.json \
--output json | jq -r '.average_score')
echo "score=$SCORE" >> $GITHUB_OUTPUT
- name: Check threshold
run: |
if (( $(echo "${{ steps.eval.outputs.score }} < 0.7" | bc -l) )); then
echo "Score ${{ steps.eval.outputs.score }} below threshold 0.7"
exit 1
fi
The workflow uses OpenID Connect to assume an AWS IAM role, avoiding long-lived credentials. The evaluation step outputs a score, and the threshold check fails the build if the score is too low.
State Management and Versioning
The agent definition and evaluation dataset are both version-controlled. This means every commit has a reproducible test suite. If you revert a change to the agent prompt, the evaluation dataset reverts with it.
The AgentCore runtime assigns a unique version ID to each deployed agent. The evaluation API accepts a version ID, so you can test a specific agent version even if a newer version is already deployed.
Trade-offs and Risks
| Aspect | Benefit | Risk |
|---|---|---|
| OAuth in CI | Secrets never appear in logs | Token exchange adds latency and failure modes |
| Custom scoring | Flexible, domain-specific checks | Requires manual tuning and maintenance |
| Version-controlled evals | Reproducible tests across commits | Dataset can drift from real-world usage |
| PR gates | Prevents regressions from merging | False positives block legitimate changes |
| AgentCore runtime | Managed infrastructure, no server ops | Vendor lock-in, limited observability |
The biggest risk is false positives. If your scoring function is too strict, it will block PRs that improve the agent in ways the test suite does not measure. You need a process for reviewing blocked PRs and updating the evaluation dataset when the agent's behavior legitimately changes.
Observability Gaps
The reference implementation does not include detailed logging or tracing. You see the final score, but not which test cases failed or why. To debug a failing evaluation, you need to run the agent locally and inspect the responses manually.
You can add observability by:
- Logging each test case result to CloudWatch.
- Storing agent responses in S3 for post-mortem analysis.
- Sending evaluation metrics to a time-series database for trend analysis.
Without these additions, you are flying blind when a PR is blocked.
Deployment Shape
The pipeline assumes a single-region deployment. If your agent serves multiple regions, you need to run evaluations in each region or accept that the test suite only validates one deployment.
The MCP server must be reachable from the AgentCore runtime. If your server is behind a VPC, you need to configure VPC peering or use a public endpoint with IP allowlisting.
Likely Failure Modes
- OAuth token expiration: If the pipeline runs longer than the token TTL, the MCP server will reject requests mid-evaluation.
- Rate limiting: The AgentCore Evaluations API has rate limits. Large test suites may need to batch requests.
- Non-deterministic responses: If the agent uses a high-temperature model, the same prompt may produce different responses across runs, causing flaky tests.
- Stale evaluation dataset: If you update the agent's tools but forget to update the test cases, the evaluation will pass even though the agent is broken.
Technical Verdict
Use this approach when:
- You are already using Amazon Bedrock and AgentCore for agent deployment.
- You need automated regression testing for agent behavior before merge.
- Your team can maintain a custom scoring function and evaluation dataset.
- You have a process for reviewing and updating test cases as the agent evolves.
Avoid this approach when:
- You need detailed observability into why a test failed (the reference implementation does not provide it).
- Your agent's responses are highly non-deterministic (you will get flaky tests).
- You are not willing to invest in maintaining the evaluation dataset (it will drift and become useless).
- You need multi-region validation (the pipeline only tests one deployment).
The reference implementation is a starting point, not a complete solution. You will need to add logging, handle rate limits, and build a process for updating the evaluation dataset. But it solves the hard problems: credential management, deployment automation, and PR gating. If you are building agents on AWS, this is a reasonable foundation for CI/CD testing.
Top comments (0)