The Default Redshift Serverless Setting That Was Quietly Costing Us $2,700 a Month

#aws #redshift #database #cloud

A client of mine spun up a new Amazon Redshift Serverless workgroup, pointed their nightly ELT job at it, and moved on. Three weeks later the AWS bill landed and the Redshift line item was almost four times what they'd modeled. Nobody had touched a knob. The workgroup was still running the base capacity the console picked for them when they clicked "Create": 128 RPUs, on all day, whether anyone was querying or not.

That's the trap with Redshift Serverless. It removes the "how big should this cluster be" question at setup time, but it doesn't remove the need to answer it, it just moves that question to your first bill.

What an RPU actually buys you

A Redshift Processing Unit (RPU) is a bundle of compute and memory, 16 GB of memory per RPU. Base capacity is what a workgroup runs at when idle-but-warm, and Redshift Serverless scales up from there automatically under load. You're billed per second, with a 60-second minimum, at $0.375 per RPU-hour in us-east-1 (check the pricing page for your region, it varies). Storage is separate: Redshift Managed Storage runs $0.024/GB-month in that same region.

Base capacity isn't a free-form number. The allowed steps are:

4 to 8 RPUs, in steps of 4
8 to 512 RPUs, in steps of 8
512 to 1024 RPUs, in steps of 32 (1024 RPU is only available in a handful of regions; most cap at 512)

And your storage ceiling is tied to that number: 4 RPU tops out around 32 TB of managed storage, 8 to 16 RPU gets you up to 128 TB, and anything past that needs 32+ RPU regardless of how light your actual compute needs are.

Finding out what you're actually using

Before you touch the base capacity, measure. Redshift Serverless gives you two system views for exactly this:

-- Total elapsed query time over the window you care about
SELECT
    date_trunc('hour', start_time) AS hour,
    count(*) AS query_count,
    sum(datediff(millisecond, start_time, end_time)) AS total_elapsed_ms
FROM sys_query_history
WHERE start_time >= dateadd(day, -7, current_date)
GROUP BY 1
ORDER BY 1;

-- Actual RPU-hours consumed, which is what you're billed on
SELECT
    date_trunc('day', start_time) AS day,
    sum(charged_seconds) / 3600.0 AS rpu_hours,
    sum(charged_seconds) / 3600.0 * 0.375 AS approx_cost_usd
FROM sys_serverless_usage
WHERE start_time >= dateadd(day, -7, current_date)
GROUP BY 1
ORDER BY 1;

Run that for a representative week before you change anything. If your workgroup is sitting at 128 RPUs but sys_serverless_usage shows you rarely scale past what an 8 or 16 RPU base would cover, you've found your answer.

Right-sizing it

Once you know your number, drop the base capacity with the CLI:

aws redshift-serverless update-workgroup \
  --workgroup-name my-analytics-wg \
  --base-capacity 32

One important restriction: update-workgroup only accepts one changeable parameter per call. You can't set --base-capacity and --port in the same request.

If your workload is spiky rather than steady, consider AI-driven scaling instead of a fixed base. It's now the default for new workgroups, and it dynamically adjusts capacity within a base range of 8 to 512 RPU using a price-performance slider:

aws redshift-serverless update-workgroup \
  --workgroup-name my-analytics-wg \
  --price-performance-target Level=25,Status=ENABLED

Level 1 optimizes hardest for cost, 100 optimizes hardest for speed, 50 is the balanced default. AWS's own TPC-DS benchmarking found the Balanced setting delivered roughly 2x the performance of Optimized-for-Cost for about 3% more spend, which is a genuinely good trade if your workload has any concurrency to it at all.

Where this requires care

A few things bit me or people I work with, worth knowing before you change anything:

The default is not conservative. The console pre-fills 128 RPU for a new workgroup. That's fine for a real production data warehouse and wildly oversized for a dev/test workgroup or a new pipeline you're still validating.
Storage limits move with base capacity. If you drop to 8 RPU to save money and your dataset creeps past 128 TB, you'll hit a hard wall and have to bump the base back up, not just add storage.
Scaling up is automatic, scaling back down is not guaranteed to be instant or even automatic in every case, particularly at the lowest tiers. Watch sys_serverless_usage after a load spike, don't assume the meter resets itself the moment the query finishes.
AI-driven scaling has a recommended floor. AWS's own guidance says it isn't a great fit under roughly 32 base RPU (though the platform-level minimum to enable it has been lowered over time), so for genuinely small workloads a fixed base capacity is often simpler and cheaper to reason about.
Set a usage limit regardless of which mode you pick. RPU-hour limits with an alert or hard cutoff action are the actual safety net, right-sizing your base is preventive, a usage limit is what stops a runaway query from becoming a surprise invoice.

My take

Serverless billing rewards the same discipline provisioned clusters always demanded, it just hides that fact behind a friendlier setup wizard. The teams getting burned aren't picking the wrong number, they're not picking a number at all and letting the default stand. Pull a week of sys_serverless_usage, pick a base capacity that matches your actual p50 workload, lean on AI-driven scaling for the spikes, and set a hard usage limit as a backstop. That's maybe twenty minutes of work against a bill that otherwise compounds silently for months.

DEV Community