Private Integration Between API Gateway and a VPC Model Endpoint

#ai #machinelearning #tutorial #llm

A model server on ECS or EC2 with no public IP, behind an internal load balancer, reachable from a public API Gateway route. The wiring is three API calls, and two of the arguments mean something other than what they look like.

What a VPC link actually is

API Gateway runs outside your VPC. A private integration bridges that gap by having API Gateway create and manage elastic network interfaces inside your account, in subnets you name. Traffic to your model endpoint leaves API Gateway through those ENIs, so it is in-VPC traffic by the time it reaches the load balancer — there is no internet path, no public listener, and no NAT gateway involved.

There are two generations. VPC links V1 target a Network Load Balancer via PrivateLink. VPC links V2, created through the apigatewayv2 API, take subnets and security groups directly and can target an Application Load Balancer as well as an NLB. AWS now documents REST API private integrations against VPC link V2, so the V1 NLB-only constraint is no longer the whole story — if you are building new, build on V2, and if you have a V1 link the integration can be repointed at a V2 one.

The provisioning is asynchronous. A new link sits in PENDING while the ENIs are created and moves to AVAILABLE, which AWS notes can take a few minutes. Any automation that creates a link and immediately creates an integration against it needs to wait on that state rather than assume it.

Creating the link

Put the model service behind an internal load balancer. For an ECS service in awsvpc mode the target group type must be ip. Health checks matter more here than usual: a container that takes ninety seconds to load weights needs a health check grace period longer than that, or the balancer will kill it mid-load forever.

Create the link. It takes subnets and security groups and nothing else:

aws apigatewayv2 create-vpc-link \
  --name model-link \
  --subnet-ids subnet-0aaa1111 subnet-0bbb2222 \
  --security-group-ids sg-0ccc3333

Poll until it is ready: aws apigatewayv2 get-vpc-link --vpc-link-id abcd123 --query 'VpcLinkStatus'. Do not proceed on PENDING.
Allow the traffic. The security group on the load balancer must accept the listener port from the security group attached to the VPC link. This is a normal SG-to-SG rule and it is the most common reason a correctly configured integration returns 504.

One constraint on the AWS CLI call above that is worth knowing before you write the Terraform: all the resources involved must belong to the same AWS account. A VPC link cannot reach a load balancer in another account; cross-account needs a PrivateLink endpoint service in front.

Wiring the integration

The integration is an HTTP_PROXY type with connectionType set to VPC_LINK and connectionId set to the link id. The destination load balancer is named separately, by ARN:

aws apigateway put-integration \
  --rest-api-id abcdef123 \
  --resource-id aaa000 \
  --http-method POST \
  --type HTTP_PROXY \
  --integration-http-method POST \
  --connection-type VPC_LINK \
  --connection-id abcd123 \
  --integration-target 'arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/app/model-alb/1234567891011' \
  --uri 'https://model.internal.example.com:443/v1/completions'

You can put a stage variable in connectionId instead of the literal id, which is how one API definition serves several environments. AWS documents the value as "${stageVariables.vpcLinkV2Id}"; in a shell, double-quote it and escape the dollar sign so your shell does not expand it before the CLI sees it.

The uri field is a Host header

This is the field that costs people an afternoon. In a private integration, uri does not say where the request goes — integration-target does. AWS states plainly that the uri parameter in a private integration points to an HTTP or HTTPS endpoint in the VPC but is used instead to set up the integration request’s Host header. Routing is by ARN; the URI is metadata.

It has one further job when the backend is HTTPS. AWS documents that for an HTTPS endpoint the uri is used to verify the stated domain name against the certificate installed on the VPC endpoint. So the hostname in uri must match a name on the backend’s certificate, or the handshake fails — and it fails as a 504 with nothing informative in the execution log. If the backend serves a private CA certificate for model.internal.example.com, that name goes in the URI, whether or not it resolves in public DNS.

By default private integration traffic uses plain HTTP. Specifying an https:// URI with an explicit port is what selects TLS to the backend. Inside a VPC, whether that is worth the certificate management is a real decision rather than an obvious one — but if the endpoint carries prompts and completions, treat it the way you would treat any other request body containing user text.

Immutability, inactivity and security groups

VPC links V2 are immutable. AWS documents that after creation you cannot change the subnets or the security groups. Adding an Availability Zone, or moving to a new security group, means creating a second link, repointing the integrations, redeploying the stage and deleting the old one. Plan for that in the IaC rather than discovering it during an AZ expansion.
Sixty days of silence deletes the ENIs. AWS documents that if no traffic passes over a VPC link for 60 days it becomes INACTIVE and API Gateway deletes all of the link’s network interfaces, causing dependent requests to fail. Traffic resuming causes reprovisioning, which takes a few minutes. A staging environment that is quiet over a long holiday will therefore fail on the first request back and recover on its own shortly after — which reads exactly like a flaky deploy. Watch the link status, or send a synthetic request weekly.
The security group is on the link, not on the API. It governs the ENIs API Gateway created in your subnets. Egress from that group to the load balancer, and ingress on the load balancer’s group from it, are two separate rules and both are required.
Region availability is not universal. AWS publishes an explicit table of the Regions and Availability Zones where VPC links V2 are supported. Check it before designing around them in a newer Region.

Private integration solves the network path; it does not solve authorisation. The API Gateway route is still public, so it still needs an authorizer, and it is still subject to the throttles described in throttling limits on AWS API Gateway. If what you actually want is for the calling application to reach a managed model service without traversing the internet at all, that is a different mechanism — see the Bedrock VPC endpoint.