Autoscaling an ECS Service by Queue Depth

#ai #machinelearning #tutorial #llm

A worker that spends 90% of each job waiting on a model API will sit at 5% CPU with a queue ten thousand messages deep. Target tracking on CPU utilisation will look at that and scale in.

Why CPU is the wrong signal

Target tracking works by holding one metric near a value, the way a thermostat holds a temperature. For that to produce correct behaviour the metric has to rise when the service is falling behind. CPU utilisation does that for a CPU-bound service and does the opposite of that for an I/O-bound one: adding tasks to a backlog of network-waiting work barely moves average CPU, so the policy concludes it has already done enough, and the queue keeps growing at full speed while every dashboard stays green.

Queue length alone is not the fix either, because it has no units the policy can act on. Ten thousand messages behind one task is a disaster; ten thousand behind five hundred tasks is a normal Tuesday. The quantity that carries the meaning is the ratio, and AWS documents it under the name backlog per task.

Backlog per task

The definition is exactly what it sounds like: the number of messages available for retrieval divided by the number of tasks in the RUNNING state. Application Auto Scaling supports CloudWatch metric math inside a customized metric specification, so the division happens inside the policy and you never publish a metric of your own. AWS specifies the two inputs precisely:

ApproximateNumberOfMessagesVisible in the AWS/SQS namespace, dimensioned on QueueName, with the Sum statistic over one minute.
RunningTaskCount in the ECS/ContainerInsights namespace, dimensioned on ClusterName and ServiceName, with the Average statistic over one minute.

The second one has a prerequisite that costs people an afternoon: RunningTaskCount comes from Container Insights, which is not on by default. If it is not enabled on the cluster the metric does not exist, the expression returns no data, and the alarms the policy creates sit in INSUFFICIENT_DATA doing nothing at all — which looks identical to a policy that has decided not to act. Enable it with aws ecs update-cluster-settings --cluster app --settings name=containerInsights,value=enabled and confirm the metric appears with aws cloudwatch list-metrics before writing the policy.

The policy

Register the service as a scalable target first, then attach the policy. The scalable dimension for an ECS service is ecs:service:DesiredCount and the resource ID is the service/cluster/service triple:

aws application-autoscaling register-scalable-target \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/app/embed-worker \
  --min-capacity 1 --max-capacity 40

The metric math goes in a JSON file. Both raw metrics set ReturnData to false; only the expression returns data, because a target tracking specification must resolve to a single time series:

{
  "CustomizedMetricSpecification": {
    "Metrics": [
      {
        "Id": "m1",
        "Label": "Messages waiting to be processed",
        "MetricStat": {
          "Metric": {
            "MetricName": "ApproximateNumberOfMessagesVisible",
            "Namespace": "AWS/SQS",
            "Dimensions": [
              { "Name": "QueueName", "Value": "embed-jobs" }
            ]
          },
          "Stat": "Sum"
        },
        "ReturnData": false
      },
      {
        "Id": "m2",
        "Label": "Running tasks",
        "MetricStat": {
          "Metric": {
            "MetricName": "RunningTaskCount",
            "Namespace": "ECS/ContainerInsights",
            "Dimensions": [
              { "Name": "ClusterName", "Value": "app" },
              { "Name": "ServiceName", "Value": "embed-worker" }
            ]
          },
          "Stat": "Average"
        },
        "ReturnData": false
      },
      {
        "Id": "e1",
        "Label": "Backlog per task",
        "Expression": "m1 / m2",
        "ReturnData": true
      }
    ]
  },
  "TargetValue": 20,
  "ScaleInCooldown": 300,
  "ScaleOutCooldown": 60
}

aws application-autoscaling put-scaling-policy \
  --policy-name embed-backlog-per-task \
  --service-namespace ecs \
  --scalable-dimension ecs:service:DesiredCount \
  --resource-id service/app/embed-worker \
  --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration file://config.json

A successful call returns the policy ARN and the ARNs of two CloudWatch alarms — an AlarmHigh and an AlarmLow — that Application Auto Scaling created and now owns. Do not edit them by hand; they are regenerated from the policy. AWS also documents a 50 KB ceiling on the PutScalingPolicy payload when metric math is used, which only becomes relevant if you start fanning the expression across many queues.

Deriving the target value

The target is not a taste question; it falls out of a latency budget and a measured per-message service time. Two assumptions, both of which you should replace with your own numbers:

Assumption 1 — the acceptable age of the oldest message is 120 seconds.
Assumption 2 — one task completes one message every 6 seconds, sustained, including its upstream model call.

A task drains its own share of the backlog in backlog_per_task × seconds_per_message. Setting that equal to the budget gives target = 120 / 6 = 20 messages per task, which is the number in the policy above. The shape of the formula is the useful part: the target is inversely proportional to how long a message takes, so if a model change doubles per-message time the target must halve or the queue age silently doubles. That is the maintenance obligation this policy creates, and it is why the constant belongs in the same file as the derivation.

Measure the denominator rather than estimating it. Every job’s service time is already recorded if you emit it, and the p50 is the number you want here — the p99 will make the service scale for a tail it should be handling with retries and timeouts instead.

Zero tasks, cooldowns and scale-in

The expression m1 / m2 divides by the running task count, so a service at zero tasks produces no usable value and the policy cannot recover on its own — the metric it needs is a function of the thing it is trying to change. Two ways out, and they are not equivalent. Keeping --min-capacity 1 is the simple one: one warm task costs a little and keeps the denominator non-zero forever. If you genuinely need to reach zero, pair the target tracking policy with a step scaling policy on the raw queue depth alarm, so something outside the ratio can put the first task back.

The two cooldowns are deliberately asymmetric in the example. Scale-out at 60 seconds because a growing backlog is a problem that compounds; scale-in at 300 because tearing down a worker that is mid-job costs you the job’s progress and, on a model call, the tokens you already paid for. Pair the long scale-in cooldown with ECS_CONTAINER_STOP_TIMEOUT on the task and a SIGTERM handler that stops taking new messages and finishes the current one. Without that handler the autoscaler is silently converting a cost saving into duplicate work, because SQS will redeliver every message whose visibility timeout expires unacknowledged.

One more asymmetry worth setting deliberately: the SQS visibility timeout should exceed your worst-case per-message processing time by a clear margin. A model call that occasionally takes 90 seconds behind a 30-second visibility timeout produces a queue that appears to be growing under load when what it is actually doing is processing the same messages three times each.

DEV Community