Streaming a model response looks like it should cost nothing to move — a few kilobytes of text. The bytes on the wire are roughly forty times that, most of them JSON envelope, and they are metered twice: once coming into your VPC through a NAT gateway, and once going out to the browser. Here is how to get a defensible number for your own traffic.
What is actually metered
Three separate charges are in play and only one of them is the one people think of.
- Data transfer in from the internet is free. The model provider’s response arriving at AWS is inbound and is not charged as data transfer.
- NAT gateway data processing is charged in both directions. If the response reaches your workload through a NAT gateway — which it does whenever the caller sits in a private subnet — every byte is processed and billed. Amazon’s Amazon VPC pricing page lists $0.045 per GB processed in US East (N. Virginia) alongside $0.045 per NAT gateway-hour. There is no free allowance.
- Data transfer out to the internet is charged, after an allowance. This is your service re-streaming the answer to the end user. AWS states that customers receive 100 GB of data transfer out to the internet free each month, aggregated across all AWS services and Regions, on the Amazon EC2 On-Demand pricing page.
The 100 GB allowance is account-wide and shared with everything else you run, so for any established account it is already spent by something. Treat it as zero when you are sizing a new workload on top of an existing bill.
The framing is most of the bytes
A streaming chat completion is delivered as server-sent events. Each token — or each small group of tokens — is a separate event carrying a complete JSON object that repeats the response id, the object type, the creation timestamp, the model name and the choice index every single time. A representative chunk in the OpenAI-compatible shape that most gateways and many providers emit:
data: {"id":"chatcmpl-B7xQ2","object":"chat.completion.chunk","created":1754870400,"model":"gpt-4.1-mini","choices":[{"index":0,"delta":{"content":" the"},"finish_reason":null}]}
Counted as UTF-8, including the data: prefix and the blank line that terminates the event, that is about 180 bytes. The part of it the reader wanted — the word the — is four. The envelope is roughly 176 bytes of overhead per token, and it does not shrink when the model gets better or the answer gets shorter.
Chunk sizes vary by provider and by SDK version. Anthropic’s event stream uses named event types and a different delta shape; Bedrock’s ConverseStream frames events in its own binary event-stream encoding rather than SSE. Measure a real response from the provider you use rather than adopting the 180-byte figure — the method below survives a different number, the arithmetic does not.
A worked figure per million requests
Stating the assumptions explicitly, because all of them are yours to change:
- Output length: 500 tokens per response.
- Chunking: one token per SSE event.
- Chunk size: 180 bytes, from the sample above.
- Region: US East (N. Virginia).
- Path: workload in a private subnet, responses inbound through a NAT gateway, re-streamed to end users over the internet.
500 events at 180 bytes is 90,000 bytes, or 90 KB per response — against about 2 KB of actual answer text. At a million requests that is 90 GB on each leg. Applying the NAT rate AWS publishes:
inbound through NAT 90 GB x $0.045/GB = $4.05
outbound to internet 90 GB x $0.09/GB = $8.10 (first tier, see note)
-------
$12.15 per 1,000,000 requests
The $0.09/GB figure is the widely quoted first internet egress tier for US East (N. Virginia), but AWS renders its tier table dynamically and it is not something to take on trust from a third party. Confirm the rate for your Region in the AWS Pricing Calculator or on your own Cost and Usage Report line for the DataTransfer-Out-Bytes usage type before putting a number in a plan. The NAT rate and the 100 GB allowance above are quoted directly from AWS pages and linked.
Two things fall out of that arithmetic and both are more useful than the total. First, at a million streamed requests a month, egress is a rounding error next to what the tokens themselves cost — this is not a line item worth engineering around at that volume. Second, the split is the surprise: the NAT gateway processing charge on the leg that most people assume is free is a third of the bill, and unlike the internet leg it has no free allowance and does not tier down.
The charge that is usually larger
Scale the same figures to ten million requests and the NAT gateway side reaches $40.50 while the internet side reaches roughly $72 — still small. What changes the picture is when the response never leaves your account at all: an internal batch job, an evaluation harness, a retrieval pipeline that calls a model and writes to a database. There is no internet egress on that path, so the NAT gateway processing charge becomes the entire data cost, and it is charged at a flat rate on traffic that produced nothing a customer saw.
That is the case where the fix is architectural rather than arithmetical. If the model is an AWS service, an interface VPC endpoint takes the traffic off the NAT gateway entirely and onto PrivateLink’s own per-GB rate — see PrivateLink to a model endpoint. If it is a third-party API reached over the public internet, the NAT gateway is the price of not giving the workload a public IP, and the lever you have left is compression. SSE framing is close to identical from one chunk to the next, so it compresses far better than prose does; whether your provider and your own edge negotiate compression on text/event-stream is worth checking with a curl -H 'Accept-Encoding: gzip' and a byte count.
Checking your own number
Do not carry this page’s figure into a budget. Replace three inputs with measurements you can take in an afternoon.
- Capture one real streamed response with
curl -N ... | wc -cand divide by the completion token count the provider reports. That gives your true bytes-per-token, including whatever chunking your provider does. - Take your p50 output token count from your own logs, not from an assumption about answer length. Output length distributions in production are usually far more skewed than people expect, and the mean is dragged by a long tail — streaming cost covers why that tail matters for more than bytes.
- Read the actual charged quantity out of Cost Explorer grouped by usage type, filtered to the
NatGateway-BytesandDataTransfer-Out-Bytesusage types. This is the only number AWS agrees with, and comparing it to your derived figure tells you whether your model of the traffic is right.
If the measured figure is much larger than the derived one, the usual cause is not the model traffic at all: it is container image pulls, package installs, or telemetry export sharing the same NAT gateway. That is worth knowing before you optimise a streaming path that was never the problem.
Top comments (0)