DEV Community

Cover image for Building an AI Gateway on ECS Fargate with per-application budgets
Alexander Smirnoff
Alexander Smirnoff

Posted on Originally published at builder.aws.com

Building an AI Gateway on ECS Fargate with per-application budgets

App A has spent its one-cent model budget. Its next request to Amazon Bedrock should be denied. App B uses the same model through the same endpoint, and its request should still succeed. A successful chat completion tells you very little about whether those two rules hold.

I built this gateway to test both rules on a real deployment. The applications use separate access, and the gateway tracks spend against each key. The tricky part is timing. A completion can put a key over its budget, and recorded spend can lag behind the requests that incurred it. The test needs to wait for that spend to appear, recognize a budget-specific denial, and then show that the second key still works.

Direct Bedrock access is a reasonable starting point when applications already manage their own AWS identities and usage policies. For this build, I wanted an OpenAI-compatible API and one place to assign model access and budgets to each application. AI Gateway runs on Amazon ECS with AWS Fargate, stores key and spend records in PostgreSQL, and calls Amazon Nova Micro through Bedrock Converse.

In the recorded run, App A was denied after AI Gateway reported $0.010573 in spend against its $0.01 budget. App B still completed a request. This walkthrough shows how to deploy the same gateway, reproduce the budget test, exercise retries independently of AWS, and remove the demo resources.

What happens to a request

Both demo clients connect to 127.0.0.1:4000 through an AWS Systems Manager port-forwarding session. They send the model alias nova-micro and their own bearer keys to /v1/chat/completions. AI Gateway maps the alias to bedrock/converse/amazon.nova-micro-v1:0. The ECS task role is scoped to the Nova Micro foundation model for Bedrock invocation, so the clients never need the task's AWS credentials.

Model alias and proxy settings

model_list:
  - model_name: nova-micro
    litellm_params:
      model: bedrock/converse/amazon.nova-micro-v1:0
      aws_region_name: os.environ/BEDROCK_REGION

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

litellm_settings:
  turn_off_message_logging: true
Enter fullscreen mode Exit fullscreen mode

Bedrock task policy

task.addToTaskRolePolicy(new iam.PolicyStatement({
  actions: ['bedrock:InvokeModel', 'bedrock:InvokeModelWithResponseStream'],
  resources: [this.formatArn({
    service: 'bedrock',
    account: '',
    resource: 'foundation-model',
    resourceName: 'amazon.nova-micro-v1:0',
    arnFormat: ArnFormat.SLASH_RESOURCE_NAME,
  })],
}));
Enter fullscreen mode Exit fullscreen mode

AI Gateway Architecture

 

Fargate gives the long-running proxy and Admin UI a place to run. The task uses a public subnet and a public IPv4 address for outbound HTTPS, with no inbound security-group rule and no NAT gateway. The proxy binds to loopback. Aurora runs in isolated subnets with encryption at rest, required TLS, and PostgreSQL access from the task security group. The image pins LiteLLM version by digest and includes the RDS CA bundle. ECS injects the database credentials and master-key suffix from Secrets Manager. Session access and the master key are operator privileges. Each application gets only its virtual key.

VPC and database · Task and service configuration · Pinned image · TLS and loopback startup

1. Deploy the gateway

Use a disposable AWS environment in us-east-1. You'll need Node.js 24 or later, pnpm 12.18.0, Python 3, Make, Git, AWS CLI v2, Docker with linux/amd64 support, and the Session Manager plugin. Your AWS profile needs CDK deployment permissions and, for verification, access to CloudFormation outputs, ECS task discovery, SSM sessions, and the demo's master-key secret. Check that your account can invoke Nova Micro in the region.

Repository prerequisites

git clone https://github.com/dobeerman/aws-cb-race-3.git
cd aws-cb-race-3

export AWS_PROFILE=your-demo-profile
aws sts get-caller-identity

make install
make check
make synth
make bootstrap   # Run once if this account and region aren't bootstrapped.
make diff
make deploy
Enter fullscreen mode Exit fullscreen mode

Review the diff before deploying. The stack outputs include the ECS cluster and service names, the master-key secret ARN, and the CloudWatch log group name.

Install, check, synthesize, and deploy targets · Stack outputs

2. Open the tunnel and make a real Bedrock call

In terminal A, open the tunnel and leave it running.

export AWS_PROFILE=your-demo-profile
make tunnel
Enter fullscreen mode Exit fullscreen mode

The script finds the running task and uses the gateway container's runtime ID to start an SSM port-forwarding session. If ECS replaces the task, reopen the tunnel.

Tunnel target · Task discovery and port forwarding

In terminal B, check readiness and run the smoke test.

export AWS_PROFILE=your-demo-profile
curl --noproxy '*' -i http://127.0.0.1:4000/health/readiness
make smoke
Enter fullscreen mode Exit fullscreen mode

There's an easy failure to diagnose here. If the smoke test says it can't reach /health/readiness, check the tunnel first. Deployment leaves that step to the operator. An HTTP 200 from readiness shows that the proxy is reachable. The smoke test goes further. It checks that anonymous model listing is rejected, creates a temporary nova-micro key with a $0.01 budget, reads back its policy, requires a nonempty Bedrock completion, and deletes the key.

Readiness and tunnel troubleshooting · Smoke test

3. Exhaust App A's budget and send a request from App B

With the tunnel still running, run the live budget test.

make test-budgets
Enter fullscreen mode Exit fullscreen mode

The script creates two keys for nova-micro, each with a 30-minute lifetime. It sends synthetic inputs through App A's key and uses returned token counts to estimate when to stop generating load. Then it polls AI Gateway's recorded spend and sends a small probe that must return a budget-specific error. An HTTP 429 by itself would be ambiguous because rate limiting can produce the same status. Finally, the script checks that App B still has zero spend before its first request, requires a successful completion, and waits for App B's own charge to appear. It attempts to delete both keys even if a check fails.

Budget target · Budget-error detection · Request limits and spend assertions · Key creation and cleanup

Application Configured budget Recorded spend shown by the run Observed outcome
App A $0.01 $0.010573 Budget-specific denial after exhaustion
App B $0.10 $0.000001 Completion succeeded

App A's recorded spend was 5.73% above its configured threshold at the point reported by the test. That is one observed run, with values taken from the gateway's spend records. The useful distinction is between denying further requests after recorded spend crosses a threshold and reserving the maximum possible cost before each request. This deployment demonstrated the former. The workload has a client-side request limit and an eight-token output limit per request. Those limits and the virtual-key budget cannot cap the AWS bill.

Workload limits and cost assumptions

Budgets

4. Test retries where you control the failures

It's difficult to reproduce precisely two throttles followed by a successful Bedrock response. The reliability suite uses the same pinned AI Gateway image and a local mock provider inside a Docker container with external networking disabled. It asserts the order of upstream requests, the client HTTP status, and the response marker. You can run it without AWS credentials, a database, or a tunnel.

Scenario definitions · Scenario assertions · Router configuration · Docker invocation

make test-reliability
Enter fullscreen mode Exit fullscreen mode

All four cases passed in the recorded run.

Case Primary responses Backup responses Client result
Baseline 200 — 200 from primary
Retry 429, 429, 200 — 200 from primary
Fallback 503, 503, 503 200 200 from backup
Exhaustion 503, 503, 503 503, 503, 503 503

The backup route exists in the test configuration. The deployed gateway exposes one Bedrock model alias. The suite demonstrates how the pinned proxy handles the configured failures, with live Bedrock behavior covered separately by the smoke and budget tests.

Reliability target and scope · Deployed model list

Verification

To repeat the complete check, keep the tunnel open and run make verify from terminal B. The recorded clean run passed stack destruction, redeployment, and verification, including three infrastructure tests, five unit tests, the smoke test, both budget assertions, and all four local reliability scenarios.

Verification target · Clean redeployment record

What this design costs and where it stops

The database keeps key and spend state outside the Fargate task. That state comes with a standing cost. The repository estimates about $69 for 730 hours in us-east-1, assuming one 0.5 vCPU / 2 GiB Fargate task, one public IPv4 address, and Aurora at its configured 0.5 ACU floor. This estimate excludes storage, I/O, logs, secrets, image storage, and inference. App A and App B can stop making requests while those infrastructure charges continue.

Repository cost assumptions · Fargate pricing · Aurora pricing · Public IPv4 pricing

One task and one writer keep the demo small. The service can briefly go offline during an update. The proxy uses the generated database user for both runtime access and schema initialization. SSM is an operator access path, so a shared application service would need its own ingress and authorization design. The two-key test establishes independent spend accounting and enforcement within one deployment. Customer data isolation, recovery, and high availability would each need their own design and verification.

Database configuration · Single-task deployment settings · Production gaps

5. Clean up

Stop the tunnel in terminal A. In terminal B, confirm your AWS identity and destroy the demo stack.

aws sts get-caller-identity
make destroy
Enter fullscreen mode Exit fullscreen mode

Confirm that SpeedRunAiGateway reaches DELETE_COMPLETE in CloudFormation. The demo deletes the database and its key, budget, and spend history without a final snapshot. The shared CDKToolkit bootstrap stack and its ECR assets may remain. Check for other consumers before removing those shared resources.

Destroy target · Cleanup behavior

Conclusion

The result I wanted was an ordinary failure that affected only one application. App A crossed its budget and received a denial. App B went through the same proxy to the same model and succeeded. The useful test was the request I sent after the failure.

The same API boundary gives the design room to grow. A provider such as OpenAI or Anthropic could be added behind the AI Gateway, with its credentials and routing managed at the gateway. If the endpoint, bearer-key authentication, model alias, and API behavior stay compatible, the applications can keep their existing code and access pattern. The live deployment verified Amazon Bedrock. Any additional provider route would need its own checks for request compatibility, spend accounting, and failure behavior.

Project repository

Top comments (0)