Oluwasegun Adedigba for AWS Community Builders

Posted on Nov 11

Non-functional Application Requirements: Scalability

#aws #architecture #ai #scalability

In 2023, the Prime Video team that monitors stream quality published something few teams admit to. Their first design, a distributed pipeline of serverless components orchestrated by AWS Step Functions, was "proven to only support around 5% of the expected load." It hit an account limit on Step Functions state transitions and paid for heavy S3 traffic on intermediate video frames. They moved the logic into one process on ECS on EC2 and reported a 90% cut in operating costs (InfoQ, 2023).

Nobody on that team picked bad services. Each one scaled fine on its own. What broke was the cost per unit of work, which grew faster than the load. Every scaling option buys you something and charges you for it, so let's put numbers on both.

TL;DR

Scalability means cost per unit of work stays flat as load grows. Elasticity is how fast capacity follows it.

Price one unit of load per option. In us-east-1, Lambda beats always-on containers only while they'd sit below about 35–40% busy.

Your database is usually the real ceiling. Find its limit before adding compute in front of it.

For AI, the scaling unit is tokens per minute and your Bedrock quota is the ceiling. Plan quota like capacity.

A queue with a concurrency cap is often cheaper than more capacity.

What: scalability, elasticity and performance

Scalability is the ability to take on more work by adding resources while the cost of each unit of work stays roughly flat. Twice the users for twice the money scales. A wall you can't spend past doesn't.

Elasticity is how quickly and automatically capacity follows the load. Performance, from Part 5, is how fast one request is at a given load. A fast system can fail to scale, and an elastic one can scale into a bill nobody planned.

Vertical scaling means a bigger machine. It needs no code changes and stops at the largest instance size, with a restart on the way up. Horizontal scaling means more machines behind a load balancer. It has no hard top, but only when any copy can serve any request, so your app has to be stateless, with sessions and caches in a shared store.

flowchart LR
  subgraph V["Vertical: a bigger box"]
    U1["Users"] --> B1["One large instance<br/>app plus session state"]
    B1 --> C1["Ceiling: largest size,<br/>restart to resize"]
  end
  subgraph H["Horizontal: more boxes"]
    U2["Users"] --> LB["Load balancer"]
    LB --> I1["Stateless copy 1"]
    LB --> I2["Stateless copy 2"]
    LB --> I3["Stateless copy N"]
    I1 --> S["Shared state<br/>DynamoDB or ElastiCache"]
    I2 --> S
    I3 --> S
    S --> C2["Ceiling moves to<br/>the shared store"]
  end

Horizontal scaling removes the ceiling from compute and moves it to the shared state store.

To test it, track cost per unit of work (per 1,000 requests or per million tokens), maximum throughput at your latency SLO, scale-out time from a load step to serving capacity, and headroom, the capacity you pay for to cover that gap.

Why: scaling fails in two expensive directions

Under-scaling costs you the event you were built for. On November 15, 2022, the Eras Tour presale sent Ticketmaster 3.5 billion system requests, four times its previous peak, and about 15% of site interactions had issues (Ticketmaster, 2022). Ticketmaster blamed bots and fans without codes. Either way, the bill came due on launch day.

Over-scaling costs you every month, quietly. In Flexera's 2026 State of the Cloud report, respondents estimated that 29% of their cloud spend is wasted, the first rise in five years, with AI named as a growing source of it (Flexera, March 2026). Headroom you add for a launch and never take back is an easy way to land in that number.

And the ceiling is rarely AWS itself. During Prime Day 2025, DynamoDB peaked at 151 million requests per second and SQS at 166 million messages per second (AWS News Blog, 2025). Your design decides how far you get, and at what price.

When: where scalability enters the lifecycle

Design. Name your scaling unit and the component with the lowest ceiling. Price one unit at today's load and at ten times it.
Build. Keep compute stateless, make writes idempotent, and pool database connections from the first commit.
Test. Load test past your peak until something breaks, with a sudden step, since a smooth ramp hides warm-up gaps.
Operate. Put cost per unit next to latency on the dashboard. Alarm on queue age and quota use, not only CPU.

The database is the real ceiling

Here's the pattern I'd expect in most designs. Compute scales out cleanly, and every new copy opens a connection to the same database. Lambda can add 1,000 execution environments every 10 seconds per function. An RDS for MySQL db.m7g.large allows about 630 connections by default (RDS quotas). One burst can use up the pool in seconds.

flowchart TB
  B["Traffic burst"] --> L["Lambda adds up to 1,000<br/>environments per 10 s"]
  L --> Q{"Where does<br/>each one connect?"}
  Q -->|"Directly"| DB["RDS db.m7g.large<br/>about 630 connections"]
  DB --> X["Too many connections<br/>errors under load"]
  Q -->|"Through RDS Proxy"| P["Shared connection pool"]
  P --> DB2["Same database,<br/>steady connection count"]
  Q -->|"Key-value access"| D["DynamoDB<br/>3,000 RCU and 1,000 WCU<br/>per partition"]
  D --> K{"Partition key<br/>spread evenly?"}
  K -->|"No"| HOT["Hot partition throttles<br/>while the table looks idle"]
  K -->|"Yes, or write sharding"| OK["Throughput grows<br/>with partitions"]

Scaling compute only helps when the database behind it can accept the extra connections and writes.

Your options, from cheapest change to most expensive:

RDS Proxy pools connections for $0.015 per database vCPU-hour in us-east-1, so a 2-vCPU instance adds 2 × $0.015 × 730 = $21.90 a month. Statements over 16 KB and some session state pin a client to one connection.
Read replicas take reads. Aurora allows up to 15, with lag "usually much less than 100 milliseconds." They do nothing for writes.
DynamoDB gives each partition 3,000 read and 1,000 write units per second, so a key like plant#date that every write hits is your ceiling however big the table gets. Write sharding adds a suffix (say 1–200) to spread writes, at the cost of reading every shard.
Sharding a relational database is the last resort, because cross-shard joins become your problem. Aurora PostgreSQL Limitless Database (generally available since October 2024) does the routing, but prove it on your queries first.

Where: what each option buys you

Each option trades money for work you no longer do.

EC2 Auto Scaling adds no fee beyond the instances. AWS says to "set the target value as high as possible with a reasonable buffer," because headroom is money. Predictive scaling needs 24 hours of history (14 days is better), forecasts 48 hours ahead and only scales out. Harbor Works, the fictional plant from the Luso companion app, has three production lines whose dashboards fill at every shift change, the textbook predictive case. Choose EC2 for steady load with Savings Plans (up to 72% off), unless you don't want to own AMIs and patching.

ECS on Fargate is the middle ground, with no hosts, per-second billing and Fargate Spot at up to 70% off for interruptible work. It fits a containerized API with daytime load and a small team, not GPU work.

EKS pays off when many workloads share one platform. The HPA scales pods, KEDA scales them from sources like SQS depth, and Karpenter picks and packs nodes. A NodePool's limits is a spending ceiling, and consolidationPolicy: WhenEmptyOrUnderutilized repacks pods onto fewer nodes. Choose EKS when Kubernetes is already your platform.

Lambda stays the default for spiky and low-traffic work. Its limit is concurrency, not money, which is why its most useful scaling setting is a cap.

Decide: price one unit of load

Let's compare options on one unit, one vCPU and 2 GB of memory kept busy for one hour, using us-east-1 on-demand prices from the AWS Price List API, pulled October 29, 2026.

Option	Price per busy unit-hour	How it scales	AWS handles	You still own
Lambda, Arm, 2 GB	$0.096 + $0.0072 for 36,000 requests at 100 ms = $0.103, only while running	1,000 environments per 10 s, 1,000 per Region by default	Servers, OS, AZs, scaling	Concurrency caps, connections, idempotency
Lambda Managed Instances	EC2 price + 15% fee	Doubles in about 5 min, on CPU	Instances, patching	Thread safety, instance choice
ECS on Fargate, Arm	$0.03238 + 2 × $0.00356 = $0.0395 per running hour	Target tracking	Hosts, OS	Task size, policy, AZ placement
EC2 Auto Scaling, c7g.large	$0.0725 ÷ 2 = $0.0363, or $0.0239 with a 1-year RI	Target tracking plus predictive	Hardware	OS, AMIs, warm-up, headroom
EKS with Karpenter	EC2 price + $73/month per cluster on standard support	HPA or KEDA, plus Karpenter	Control plane	Nodes, add-ons, upgrades

Containers and instances look cheaper, but you pay for them while they're idle. At a 60% CPU target, a busy unit-hour really costs $0.0363 ÷ 0.6 = $0.060 on EC2 and $0.0395 ÷ 0.6 = $0.066 on Fargate. Lambda bills only while it runs.

That gives you a break-even line. Fargate wins once its capacity would be busy more than $0.0395 ÷ $0.103 ≈ 38% of the time, EC2 above about 35%. Below that, Lambda wins, and at zero traffic it costs nothing.

Three caveats. A 2 GB function gets slightly more than one vCPU (1,769 MB equals one). Lambda bills wall-clock time, including waits on a database, while a container serves other requests during the wait, so I/O-heavy work tips toward containers sooner. And none of this prices your team's time.

So here's the rule I'd use. Start on Lambda when traffic is spiky or unknown. Move a service to Fargate or Lambda Managed Instances once your dashboard shows it busy for most of the day. Reach for EKS when the platform already exists for other reasons.

What more scale costs elsewhere

Every gain in scale gets paid for somewhere else in this series.

NFR	Helps	Hurts
Performance	Headroom and Priority tier cut queueing	Queue levelling adds latency by design
Availability	Stateless copies across AZs survive failures	Each autoscaled dependency adds a quota that can run out
Security	Smaller, identical units are easier to patch	Every new node pool, proxy and queue is more attack surface
Compliance	In-Region and geographic inference profiles keep processing in one geography	Global profiles move prompts across Regions to find capacity
Maintainability	Managed scaling means less code to own	EKS, Karpenter and self-hosted vLLM add upgrade work
Cost	Scale-to-zero, Flex and batch cut idle spend	Headroom, Reserved tiers and GPU nodes bill whether busy or not
Sustainability	Consolidation and Arm instances mean fewer idle watts	Over-provisioned GPUs burn energy for no work

Queue-based load levelling

A queue lets you size for the average instead of the peak. Producers write as fast as they like and consumers work at a rate your downstream systems survive, for $0.40 per million SQS standard requests after the free million.

On Lambda, the control is the event source mapping's MaximumConcurrency, from 2 to 1,000. Without it, Lambda adds up to 300 concurrent invocations a minute, up to 1,250 per mapping (Lambda docs). That's great for throughput and bad for a database with 630 connections. Provisioned mode goes the other way, reserving pollers at $0.00925 per poller-hour, and can't be combined with the cap.

flowchart LR
  L1["Line 1 shift end"] --> Q["SQS queue<br/>absorbs the burst"]
  L2["Line 2 shift end"] --> Q
  L3["Line 3 shift end"] --> Q
  Q --> E["Event source mapping<br/>MaximumConcurrency 3"]
  E --> W["Lambda workers<br/>one model call each"]
  W --> BR["Amazon Bedrock<br/>TPM quota"]
  W -->|"Throttled: hand back"| Q
  Q -->|"Fails 5 times"| DLQ["Dead-letter queue"]
  Q -.->|"Oldest message age"| A["CloudWatch alarm<br/>backlog over 15 min"]

The queue turns a burst from three lines finishing their shifts at once into a steady stream that stays under the quota.

You're trading latency for a flat bill, so the queue's age becomes the SLO. Alarm on ApproximateAgeOfOldestMessage, not queue length.

Hands-on: size a queue consumer from your quota

At shift end, Harbor Works asks a model to summarize each line's telemetry for the next supervisor, and all three lines finish at once. Let's size the consumers from the Bedrock quota.

Each call reserves its input tokens plus maxTokens against your TPM quota when it starts, and each worker makes 60 ÷ seconds_per_call calls a minute. So the safe worker count is quota × seconds_per_call ÷ (60 × tokens_per_call). With an example quota of 200,000 TPM, 3,400 tokens per call and 4-second calls, that's 200,000 × 4 ÷ (60 × 3,400) ≈ 3.9, so three workers.

Checked against the Terraform AWS provider docs for aws_lambda_event_source_mapping and aws_autoscaling_policy, and the Lambda SQS configuration guide, on Oct 29, 2026. terraform validate passes with Terraform 1.13.4 and AWS provider 6.71.0. Not deployed — test in a sandbox account first.

terraform {
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 6.0" }
  }
}

provider "aws" {
  region = "us-east-1"
}

variable "worker_zip" { type = string }      # zip containing worker.py (below)
variable "worker_role_arn" { type = string } # needs SQS read/delete + bedrock:InvokeModel

locals {
  # Size the fleet from the quota, not from the queue depth.
  tpm_quota        = 200000 # example value: read yours from Service Quotas
  tokens_per_call  = 3400   # input tokens + maxTokens, reserved when a call starts
  seconds_per_call = 4      # measured p50 duration of one worker invocation
  max_workers      = max(2, floor(local.tpm_quota * local.seconds_per_call / (60 * local.tokens_per_call)))
  fn_timeout       = 30
}

resource "aws_sqs_queue" "dlq" {
  name                      = "harbor-summaries-dlq"
  message_retention_seconds = 1209600 # 14 days to inspect bad messages
}

resource "aws_sqs_queue" "jobs" {
  name                       = "harbor-summaries"
  visibility_timeout_seconds = 6 * local.fn_timeout # AWS guidance: at least 6x the function timeout
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.dlq.arn
    maxReceiveCount     = 5 # throttled jobs come back, broken ones stop here
  })
}

resource "aws_lambda_function" "worker" {
  function_name = "harbor-summary-worker"
  role          = var.worker_role_arn
  filename      = var.worker_zip
  handler       = "worker.handler"
  runtime       = "python3.13"
  architectures = ["arm64"]
  memory_size   = 512
  timeout       = local.fn_timeout
  # Must be >= the event source's maximum concurrency, or Lambda throttles messages.
  reserved_concurrent_executions = max(5, local.max_workers)
  environment {
    variables = {
      MODEL_ID     = "global.amazon.nova-2-lite-v1:0"
      SERVICE_TIER = "flex" # async work: 50% off Standard, slower is fine
      MAX_TOKENS   = "400"
    }
  }
}

resource "aws_lambda_event_source_mapping" "jobs" {
  event_source_arn        = aws_sqs_queue.jobs.arn
  function_name           = aws_lambda_function.worker.arn
  batch_size              = 1 # one model call per invoke keeps the sizing math honest
  function_response_types = ["ReportBatchItemFailures"]
  scaling_config {
    maximum_concurrency = local.max_workers # the ceiling that protects the TPM quota
  }
}

# The queue's age is the SLO you are trading for a flat model bill.
resource "aws_cloudwatch_metric_alarm" "backlog_age" {
  alarm_name          = "harbor-summaries-too-old"
  namespace           = "AWS/SQS"
  metric_name         = "ApproximateAgeOfOldestMessage"
  dimensions          = { QueueName = aws_sqs_queue.jobs.name }
  statistic           = "Maximum"
  period              = 60
  evaluation_periods  = 5
  threshold           = 900 # 15 minutes: time to raise the quota or the tier
  comparison_operator = "GreaterThanThreshold"
}

The worker makes one attempt per message and hands throttled messages back to the queue, so the queue does the retrying instead of the SDK.

Checked against the Bedrock Converse API and the Nova 2 Lite model card on Oct 29, 2026. Unit-tested with botocore Stubber on boto3 1.43.121 (a success, a throttle handed back, a validation error raised). Package a current boto3 with the function, because serviceTier needs a recent SDK. Not deployed — test in a sandbox account first.

import json
import os

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

# One attempt only: the queue is the retry mechanism, not the SDK.
bedrock = boto3.client("bedrock-runtime", config=Config(retries={"max_attempts": 1, "mode": "standard"}))

MODEL_ID = os.environ["MODEL_ID"]
SERVICE_TIER = os.environ.get("SERVICE_TIER", "default")
MAX_TOKENS = int(os.environ.get("MAX_TOKENS", "400"))
RETRYABLE = {"ThrottlingException", "ServiceUnavailableException", "ModelNotReadyException"}


def summarize(job: dict) -> str:
    resp = bedrock.converse(
        modelId=MODEL_ID,
        system=[{"text": "Summarize this shift's machine telemetry for a line supervisor in five bullets."}],
        messages=[{"role": "user", "content": [{"text": job["telemetry_digest"]}]}],
        inferenceConfig={"maxTokens": MAX_TOKENS},  # reserved against TPM at call start
        serviceTier={"type": SERVICE_TIER},
    )
    return resp["output"]["message"]["content"][0]["text"]


def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            job = json.loads(record["body"])
            summary = summarize(job)
            print(json.dumps({"line": job["line_id"], "chars": len(summary)}))  # write it wherever you store results
        except ClientError as err:
            if err.response["Error"]["Code"] in RETRYABLE:
                # Hand the message back. It reappears after the visibility timeout.
                failures.append({"itemIdentifier": record["messageId"]})
            else:
                raise
    return {"batchItemFailures": failures}

The Harbor Works API tier runs on EC2, so it gets target tracking for minute-to-minute load and a predictive policy for shift changes. With both active, the group runs at whichever capacity is higher.

Checked against the aws_autoscaling_policy docs and the predictive scaling guide on Oct 29, 2026. Validated with the configuration above. Not deployed — test in a sandbox account first.

variable "asg_name" { type = string }

resource "aws_autoscaling_policy" "cpu_target" {
  name                      = "harbor-api-cpu-60"
  autoscaling_group_name    = var.asg_name
  policy_type               = "TargetTrackingScaling"
  estimated_instance_warmup = 120
  target_tracking_configuration {
    predefined_metric_specification {
      predefined_metric_type = "ASGAverageCPUUtilization"
    }
    target_value = 60 # higher = cheaper, lower = more headroom for spikes
  }
}

resource "aws_autoscaling_policy" "shift_forecast" {
  name                   = "harbor-api-shift-forecast"
  autoscaling_group_name = var.asg_name
  policy_type            = "PredictiveScaling"
  predictive_scaling_configuration {
    mode                   = "ForecastOnly" # compare forecast to reality before ForecastAndScale
    scheduling_buffer_time = 600            # launch 10 minutes before the forecast hour
    metric_specification {
      target_value = 60
      predefined_metric_pair_specification {
        predefined_metric_type = "ASGCPUUtilization"
      }
    }
  }
}

Leave it in ForecastOnly for two weeks, compare the forecast with real load, then switch to ForecastAndScale.

Scalability in the age of AI

AI as a workload: tokens per minute are the unit

Once a model sits in your request path, vCPUs stop being the number that matters. Bedrock limits you per model and per Region in tokens per minute (TPM) and, for most models, requests per minute, with separate quotas for cross-Region and in-Region use. Increases aren't automatic, so request quota the way you'd plan instances for a launch.

The quota is spent in a way that surprises people. A call starts by reserving its input tokens plus your full max_tokens, then settles on input plus output tokens times a burndown rate: 5x for Anthropic models up to version 4.7, higher for some newer ones, 1x for most others (Bedrock docs). You're billed only for real tokens, so a careless max_tokens of 32,000 throttles you long before the bill looks big. Watch the EstimatedTPMQuotaUsage metric, added in March 2026.

Agents multiply load

One user action is no longer one call. Anthropic measured its agents using about 4x the tokens of a chat interaction, and its multi-agent systems about 15x (Anthropic, 2025).

flowchart TB
  U["Supervisor asks: why did line 2 slow down?"] --> A["Agent"]
  A --> M1["Model call 1: plan"]
  M1 --> T1["Tool: query telemetry"]
  M1 --> T2["Tool: read maintenance log"]
  T1 --> M2["Model call 2: read results"]
  T2 --> M2
  M2 --> T3["Tool: compare with line 1"]
  T3 --> M3["Model calls 3 to N:<br/>context grows each turn"]
  M3 --> R["One answer"]
  M3 -.->|"Each call spends"| QT["TPM quota and<br/>tool API capacity"]

One question to an agent becomes N model calls and M tool calls, and every one of them counts against a quota.

Let's put numbers on it with Claude Sonnet 4.6 at $3.30 per million input and $16.50 per million output tokens (US geo profile, us-east-1). One call with 6,000 input and 400 output tokens costs $0.0198 + $0.0066 = $0.0264. Six such calls cost $0.158 per question and, with the 5x burndown, use 36,000 + 2,400 × 5 = 48,000 quota tokens. On a 200,000 TPM quota, that's about four questions a minute for the whole account, so cap agent turns in code.

Bedrock capacity options, weighed by cost

Bedrock prices Priority at a 75% premium over Standard, Flex at a 50% discount and batch at 50% below on-demand for selected models (Bedrock pricing). The us-east-1 price list agrees, with Nova 2 Lite input at $0.33 per million on Standard, $0.165 on Flex and $0.5775 on Priority.

Priority buys the front of the line for customer-facing peaks and shares the Standard quota. Support varies, and Sonnet 4.6 offers neither Priority nor Flex.
Flex is the cheap default for anything nobody is waiting on, like the shift summaries above.
Batch halves the price of evals and backfills (Sonnet 4.6 input drops from $3.30 to $1.65). It runs from S3 files, and its Converse format doesn't support tool calls.
Reserved sells a fixed TPM rate for 1 or 3 months, targets 99.5% uptime and overflows to Standard. Here's the catch. For Sonnet 4.6 (US geo, 1 month), 1,000 input TPM costs $0.198 an hour, exactly 1,000 × 60 × $3.30 per million. You break even only if you use every reserved token every minute, or 90% of them on the 10%-cheaper 3-month term. The 100,000 input plus 10,000 output TPM minimum is about $21,681 a month. Reserved buys certainty, not a discount.
Provisioned Throughput bills model units hourly and is required for custom models. Nova 2 Lite is $55 an hour on a 1-month term, $40,150 a month.

Self-host on EKS, or stay on Bedrock?

Ministral 3 8B makes a fair test, because the same Apache 2.0 open-weight model runs on Bedrock at $0.15 per million tokens (input or output) and on vLLM on your own GPU. Its model card says it fits in 12 GB of VRAM in FP8.

Self-hosting needs two nodes for availability. Two g6.xlarge (one 24 GB L4 GPU each) at $0.8048 an hour plus the EKS fee cost 2 × $587.50 + $73 = $1,248 a month, the price of 8.3 billion Bedrock tokens, or 3,170 tokens per second around the clock.

Now add people. Say drivers, upgrades and on-call take a quarter of an engineer, and call that $4,000 a month (use your own number). Break-even jumps to $5,248 ÷ $0.15 ≈ 35 billion tokens a month, about 13,300 tokens per second sustained. Against Flex or batch at $0.07, it's 75 billion. A 1-year RI cuts the hardware to about $838, which barely moves the total.

flowchart TB
  S["Monthly tokens on one open-weight model"] --> D{"Above break-even?<br/>nodes plus ops divided<br/>by Bedrock price"}
  D -->|"No"| BR["Stay on Bedrock<br/>Flex or batch for async"]
  D -->|"Yes"| U{"Load steady<br/>around the clock?"}
  U -->|"No, bursty"| BR
  U -->|"Yes"| T{"Can your nodes<br/>sustain it at your SLO?"}
  T -->|"Not measured"| BM["Benchmark vLLM<br/>on your prompts first"]
  T -->|"Yes"| SH["Self-host on EKS<br/>Karpenter, GPU or Inferentia"]
  T -->|"Needs more nodes"| D

Self-hosting pays off only when your token volume is high, steady and measured on your own hardware.

That last box matters. There's no published AWS benchmark for this model on L4 or inf2.xlarge ($0.7582 an hour) that I can point you to, so measure it with vLLM's benchmark tooling on your own prompts. Self-host for data control or truly steady volume, not to save money at moderate scale.

Vector indexes scale on a different curve

Retrieval scales on storage and query rate, not tokens. S3 Vectors holds up to 2 billion vectors per index at $0.06 per GB-month plus $2.50 per million queries and a small processing fee. AWS's own example, 10 million vectors and a million queries a month, costs $11.38, with frequent queries around 100 ms. OpenSearch Serverless is faster and richer. A first classic collection bills at least 2 OCUs at $0.24 per OCU-hour, about $350 a month before any data. The newer NextGen collections have no minimum and scale to zero after 10 idle minutes.

One limit catches people out. An S3 Vectors index takes at most 2,500 inserts and deletes per second, so re-embedding 10 million vectors into one index takes at least 67 minutes. Spread big re-embeds across indexes, and keep the hot set users query all day in OpenSearch.

AI as an engineering aid

AI assistants are good at the tedious parts. They draft a Terraform policy, write a k6 script or explain a throttling graph, and AWS DevOps Agent (generally available since March 2026) works on resolving and preventing incidents.

They're weak on exactly the numbers this article depends on. Expect last year's prices, invented default quotas, and CPU target tracking for a workload that waits on I/O. So pull prices from the Price List API, read quotas from Service Quotas, and treat an AI-written scaling policy as untested until a load test has pushed past its target.

Who: owners and the shared responsibility split

Activity	Product owner	Architect	Developers	Platform / SRE
Scaling unit, unit-cost target, growth forecast	A	R	C	C
Ceiling analysis and compute choice	C	R/A	C	C
Stateless code, pooling, idempotency, agent turn caps	I	C	R/A	C
Scaling policies, quotas, load tests, cost dashboards	I	C	C	R/A

R = responsible, A = accountable, C = consulted, I = informed.

For compute, the "AWS handles" and "You still own" columns in the Decide table are the split. For managed AI, AWS serves the model, honors your tier and enforces quotas. You pick the model and tier, request quota early, set max_tokens and cap agent fan-out.

Monday checklist

[ ] Pick your scaling unit and put cost per unit of work on your main dashboard.
[ ] Find your lowest ceiling: database connections, a hot partition key, or a model quota.
[ ] Put RDS Proxy in front of any relational database that Lambda or autoscaled containers talk to.
[ ] Work out where Lambda and always-on capacity break even for your busiest service, using this year's prices.
[ ] Add MaximumConcurrency to every SQS event source that calls something with a hard limit.
[ ] Set max_tokens close to real output sizes and alarm on EstimatedTPMQuotaUsage.
[ ] Move evals, backfills and summaries to Flex or batch.
[ ] Count model and tool calls per user action for each agent, and cap the turns in code.

Conclusion

Every option here scales. They differ on the price of each extra unit. Lambda charges more per busy hour and nothing for idle time, and containers flip that. Reserved model capacity buys certainty at the full on-demand rate. Self-hosting buys control that pays off only at volumes most teams never reach.

So start with the ceiling, because no amount of compute fixes a database at its connection limit or an agent that spends a whole minute of quota on four questions. Then price the unit and pick the option whose cost curve matches your load. Prime Video didn't need faster services. It needed a design whose cost grew with the work instead of ahead of it.

Scale is easy to buy. Find the ceiling and price the unit before you do.

Series: Non-functional Application Requirements

Non-functional Application Requirements: An Introduction
Non-functional Application Requirements: Security
Non-functional Application Requirements: Compliance
Accessibility
Performance
Availability
Scalability
Maintainability (coming soon)
Cost (coming soon)
Sustainability (coming soon)

Cover image: AWS re:Invent 2023 – Keynote with Dr. Werner Vogels

Next up, Maintainability: the pitfalls that make a system harder to change every month, and what CodeBuild, CodeDeploy and CloudWatch can and can't do about them.

DEV Community