DEV Community: Erez

The ECS Task That Billed You All Night (And How to Think About Fixing It)

Erez — Thu, 28 May 2026 12:04:29 +0000

Your ECS service spun up four tasks at 9 AM for a load test. By 11 AM the test was done, the results were in Slack, and everybody moved on. It's now 2 AM. Those four tasks are still running.

Nothing is serving traffic. Nobody is waiting on them. The service health check is green because the tasks haven't crashed — they're just sitting there, healthy and idle and billing you $0.04048 per vCPU-hour and $0.004445 per GB-hour.

This is not a rare edge case. It's the default behavior of most dev and staging environments, because the failure mode is invisible. The instance isn't broken. The service isn't alerting. The cost just... accumulates, quietly, until someone pulls the AWS bill at the end of the month and wonders where the extra $400 went.

This article is about why that happens structurally — what ECS, EC2, and ASG are actually doing while your engineers sleep — and what a better model looks like.

Why the problem is harder than it looks

The naive fix is a schedule: shut things down at 7 PM, bring them back at 8 AM. This is what most teams reach for first, and it solves about 60% of the problem while introducing a different one.

Schedules are an approximation of human behavior, not a model of it. Your engineers are not fungi; they don't follow a predictable diurnal rhythm. Someone is debugging a prod issue at 10 PM. Someone else is finishing a migration at 6 AM before a release. A team in a different timezone starts at noon your time.

The moment you codify a schedule, you start getting exceptions. The 7 PM shutdown becomes 8 PM because someone complained. Then 9 PM. Then you add a "keep-alive" flag for certain instances. Then the schedule starts drifting per-team, per-environment, per-resource, and you've replaced a billing problem with a configuration management problem.

The other failure mode is subtler: the schedule doesn't know what the resource is doing at shutdown time. Your EC2 dev box gets terminated mid-compile. Your staging RDS gets paused while a migration is running. Your ECS task for a long-running job gets killed at T-2 minutes. These aren't theoretical — they happen, and they erode trust in the automation fast enough that teams start disabling it.

What actually drives idle time

To fix the problem properly, you have to model it correctly. Idle time in dev/staging environments comes from a few distinct sources:

Work-session gaps. Engineers don't work in continuous eight-hour blocks. There are lunch breaks, meetings, context switches, and the natural rhythm of focused work followed by review and coordination. A dev EC2 box might be actively used for three or four focused hours in a given workday, with the rest of the time sitting warm but unused.

End-of-day orphans. The most common case: work ends, the engineer closes their laptop, and whatever AWS resources they were using keep running. No shutdown signal was sent. Nothing crashed. The resources just... persist.

Test and load artifacts. ECS tasks, RDS read replicas, and EC2 instances provisioned for a specific test run and not cleaned up afterward. The test finished, the PR merged, and nobody thought to tear down the environment.

Over-provisioned ASGs. Scale-out events during business hours don't automatically reverse when load drops. If your desired capacity isn't decremented or your scale-in policy is conservative (common, to avoid flapping), the extra instances sit at minimum capacity billing you for capacity you're not using.

None of these are pathological. They're all rational behavior by engineers doing their jobs — and they add up fast at AWS prices.

The ASG case is worth examining in detail

Auto Scaling Groups get special attention because they're where the invisible-cost problem is most structurally embedded.

Here's a typical sequence:

09:00  ASG desired capacity: 2 (baseline)
13:00  Load test begins, scale-out triggered
13:15  ASG desired capacity: 6 (scale-out complete)
14:47  Load test ends
15:30  p99 latency back to baseline
17:00  Engineers sign off
23:00  ASG desired capacity: 6 (still)

Why didn't it scale back in? Several reasons:

Scale-in policies typically have a cooldown (300 seconds by default) and require sustained low-CPU, not just low-CPU right now. A brief spike at 5:50 PM reset the cooldown.
The target tracking policy is based on average CPU across the group. Four idle instances average out against two slightly-busy ones and the metric never triggers.
Nobody set a scheduled scale-in for end-of-business because "we'll handle that in the next sprint."

The result is that you're paying for four extra instances from 2:47 PM until someone manually adjusts desired capacity the next morning — roughly 16 hours. At m5.large on-demand pricing in us-east-1, that's about $0.096/hour per instance, so 4 instances × 16 hours × $0.096 = roughly $6.14 for a single day. Multiply by the number of load-test days per month, and by the number of ASGs in your non-production account, and you're talking meaningful money.

The fix that almost works: set a scheduled action to reset desired capacity to 2 at 8 PM. This is better than nothing. It still misses the load test that finishes at 7:30 PM (instances run until 8), and it breaks the engineer who is legitimately using those instances at 8:05 PM.

What "activity-driven" actually means

The phrase "activity-driven" is worth defining precisely, because it gets used loosely.

Activity-driven resource management means the pause/resume decision is based on detected real user presence and work-tool focus — what the engineer's machine is actively doing — rather than a clock or a server-side metric.

This is different from:

Schedule-based: "Pause at 7 PM, resume at 8 AM" — no signal from the human.
Metric-based: "Pause when CPU < 5% for 30 minutes" — signal from the server, not the user. An idle EC2 box running a cron job looks identical to one nobody has touched.
Manual: "Click the stop button when you're done" — relies on engineers remembering, which they don't.

An activity-driven system detects that the engineer's machine has been idle or context-switched away from work tools for a meaningful interval, and uses that as the signal to pause the associated cloud resources. When they return — open their IDE, reconnect a terminal, push a commit — the resources resume automatically before they need them.

The key property is that the signal comes from the human side of the equation, not the infrastructure side. This matters because:

An RDS instance can be completely idle (zero connections, zero queries) while the engineer is mid-thought and about to run a migration. Metric-based systems see idle and pause; the engineer gets a connection timeout.
An EC2 box can show 90% CPU because it's running a background compile while the engineer is in a three-hour meeting. Schedule-based systems keep it running; activity-driven systems can pause it when it's clear the engineer won't need output from that compile for hours.

The RDS wrinkle

RDS makes this harder because the resource has no meaningful idle signal at the instance level. An EC2 instance at 0% CPU is almost certainly idle. An RDS instance at 0% CPU might have an engineer staring at a half-written query in their SQL client, about to hit execute.

The standard AWS tooling here is Instance Scheduler, which lets you define stop/start windows per tag. It's schedule-based, and it has the same failure mode described above — except with RDS the stakes are higher, because RDS startup time is non-trivial. A Multi-AZ RDS instance can take several minutes to become available after a stop. If someone's stop window is wrong and the database restarts mid-migration, you're looking at a real incident, not just a mild inconvenience.

The billing structure also differs from EC2. When an RDS instance is stopped, you stop paying for instance hours — but you keep paying for storage (gp2/gp3 volume), backup storage, and any snapshot overhead. The Multi-AZ standby instance you provisioned for a staging environment that gets promoted to prod-like parity? That's billed at the same rate as the primary. Stopping the primary stops both, but storage keeps running.

For most non-production RDS use cases, the right architecture is a stopped instance that restarts on demand, not a running instance that someone remembers to stop. The challenge is the "on demand" part — the startup latency makes manual workflows painful, which is why most teams don't do it.

What good looks like in practice

A useful mental model: treat non-production cloud resources the way you treat your local dev environment. Your laptop doesn't run your IDE at full capacity all night. It sleeps. It wakes up fast when you need it. The resource is available when you're working and not burning compute when you're not.

Applied to AWS:

An EC2 dev box stops within a few minutes of the engineer closing their last work session, and starts automatically when they open their IDE or terminal the next morning.
An ECS staging service scales to zero when no engineers are actively working, and scales up before they connect — not after they notice a 502.
An RDS instance used only for migration testing stops at end-of-day and restarts when someone connects.
An ASG resets to baseline capacity when work-session signals across the team have been idle for a threshold interval, not on a fixed schedule.

None of this requires perfect engineering. It requires that the pause/resume signal be grounded in actual human activity rather than a wall-clock approximation of it.

The multi-account visibility problem

One thing that makes this worse at scale: non-production AWS resources often live in separate accounts from production, and visibility into those accounts is fragmented.

Your staging account, dev sandbox, and QA environment might all be separate AWS accounts — the right security posture — but the cost data from all three flows through AWS Cost Explorer at the payer account level, tagged inconsistently, and nobody has a clear picture of what's actually running and idle at any given moment.

Teams that are serious about this problem centralize the visibility first. Not just cost reports, but live resource state: what's running, who provisioned it, when it was last used, and what it's costing per hour. Without that view, you can't make good decisions about what to pause.

The tooling for this has improved significantly, but the gap between "I can see the cost" and "I can act on it automatically" is still wide for most teams. That gap is where a large portion of the waste lives.

A note on what not to do

Two approaches that seem reasonable and usually backfire:

Aggressive auto-termination. Setting TTLs on dev resources and terminating them after X hours of inactivity is blunt and creates anxiety. Engineers start hoarding long-lived resources and tagging things to prevent deletion, which is the opposite of what you want. Pause, not terminate, is the right default for dev resources; the state is preserved and startup is fast.

Tagging-and-ask workflows. Sending Slack messages like "your EC2 instance has been idle for 2 hours, should we stop it?" adds friction back to the engineer at exactly the moment they've context-switched away. Response rates on these are low, and the instances that matter most (the ones where someone is mid-task) are the ones most likely to get stopped by an inattentive yes-click.

The goal is zero-friction automation that the engineer can trust, not a different version of manual.

If this matches a problem you're dealing with in your non-production environments, Trigops is built around exactly this model — activity-driven pause and resume for EC2, RDS, ECS, and ASG, across multiple accounts. You can connect a cloud account and see what's actually running at trigops.com.

Why Your EC2 Schedule Is a Lie (And What Activity-Driven Pause Actually Means)

Erez — Thu, 28 May 2026 09:56:21 +0000

Your team set up a cron job. Probably six months ago, maybe longer. It stops EC2 dev instances at 10 PM and starts them again at 7 AM. Someone spent an afternoon wiring it up — a Lambda, a CloudWatch Events rule, maybe an AWS Instance Scheduler stack — and it felt like a win.

It is a win. Compared to nothing, a fixed schedule is real savings. But here is what that schedule actually models: everyone starts at exactly 7 AM, works continuously until exactly 10 PM, and nobody takes a lunch break, attends a three-hour meeting, or leaves early on a Friday.

Nobody works like that. And AWS bills by the second, so every gap between the schedule and reality is money leaving the account for compute that no human is using.

What the Schedule Actually Covers

Let's be concrete. Say your team is eight engineers in two time zones. The schedule runs Monday through Friday, 7 AM–10 PM in your primary zone. That's 15 hours of uptime per weekday, 0 on weekends.

Weekday uptime:  15 hours
Weekend uptime:   0 hours
Total per week:  75 hours (15 × 5)
Total per week, unmanaged: 168 hours

Schedule coverage: 75 / 168 = ~44.6%

Fair. That schedule eliminated more than half the potential uptime. Now look at what it doesn't cover.

A senior engineer in your western office starts at 9 AM local time — two hours after the instance is already running. She has a two-hour all-hands on Tuesday. She's out sick Thursday. Her instance is running and billing from 7 AM regardless.

Another engineer wraps up by 5 PM most days. The instance runs until 10 PM.

Multiply this across eight engineers. You're running instances for stretches that map to no human work at all — not nights or weekends, but the quieter idle time baked into every real workday: mornings before anyone logs in, lunch hours, meetings, end-of-day gaps before the cron fires.

The schedule didn't solve idle time. It solved predictable idle time. The unpredictable kind — the kind that follows actual human behavior — it can't touch.

Why Fixed Schedules Can't Go Further

The instinct when you notice this is to tighten the schedule. Push the start time later, pull the stop time earlier, add a weekend rule. But this runs into a hard wall fast.

Schedules are global. A tighter window that works for your latest-starting engineer leaves your earliest engineer waiting for their instance. You can try per-engineer schedules, but now you're maintaining eight sets of CloudWatch rules and someone's schedule is always wrong after a timezone change or a hire.

More importantly: schedules are static models of a dynamic thing. Human presence during a workday is not a block. It's a pattern of focus, interruption, and absence that changes every day. A meeting runs long. Someone gets pulled onto an incident. The schedule doesn't know any of this.

The other approach — server-side metrics — doesn't solve it either. Low CPU on an EC2 instance means the machine isn't doing much, but it doesn't tell you whether an engineer is actively working in their IDE with the instance open in a terminal. Background processes, keep-alive connections, and idle shells all keep metrics above zero without any human present. You end up with false positives that train your team not to trust the automation, so they disable it.

Activity-Driven: What It Actually Means

Activity-driven pause/resume — the approach Trigops takes — means the signal for stopping or starting a resource comes from observed human presence and work-tool focus on the engineer's local machine, not from a clock or from server-side metrics.

The practical shape of this: a lightweight desktop agent on the engineer's machine watches whether they're actively using tools associated with that instance (terminal sessions, IDE connections, AWS console activity). When the engineer stops working — closes the session, switches to Slack and stays there, locks their screen — the agent detects the transition and signals the associated EC2 instance to pause. When they return to active work, the instance resumes.

This is not a schedule. There is no 10 PM rule. The instance pauses when this engineer, today, right now is not using it — and resumes when they are.

The result is that idle time tracks the actual gaps in each engineer's day, not a shared approximation of when the whole team might be working.

The Architecture Difference

Here is how the two approaches compare at the decision point — the moment of deciding whether to stop a resource:

# Fixed schedule (CloudWatch Events / Instance Scheduler)

Input:  current time
Rule:   if time == "22:00", stop
Result: all instances in scope stop, regardless of who is using them


# Activity-driven (Trigops)

Input:  desktop agent signal — engineer X is idle / active
Rule:   if engineer X is idle, pause engineer X's associated resources
Result: that engineer's EC2 (or ECS service, RDS, ASG) pauses;
        engineer Y, who is still working, is unaffected

The schedule operates at the account or tag level. Activity-driven operates at the engineer level and propagates to their resources.

This matters for multi-account and multi-region teams. If you have a platform team running dev infrastructure across several AWS accounts, a schedule either covers all of them identically or you're managing a proliferation of rules. Activity-driven control stays coherent at the per-engineer, per-resource level regardless of how many accounts or regions the resources live in.

What Resources This Applies To

The resources where this makes the most practical difference are the ones tied to individual developer workflows.

EC2 dev boxes are the clearest case. One engineer, one instance, a direct mapping between human presence and resource need. When the engineer isn't there, the instance doesn't need to run.

ECS services in dev/staging are trickier with schedules because they often have dependencies and minimum task counts that make blanket stop/start risky. Activity-driven control can scope the pause to a specific service without touching the rest of the cluster, and it only fires when nobody is actively using that service — not at an arbitrary time.

RDS instances are expensive to run continuously and slow to stop/start, which is exactly why teams avoid scheduling them: the risk of stopping a shared staging database mid-use is real. Activity-driven control, tied to whether anyone has an active connection through their workstation, provides the signal that a schedule can't.

Auto Scaling Groups are the most complex case. The minimum capacity floor that someone set before a demo is now a permanent floor that nobody reviews. Activity-driven approaches can inform when the floor can safely drop — not by overriding the ASG min directly, but by detecting when no engineer is actively using the environment and signaling a scale-down.

The Real Cost of "Good Enough"

A fixed schedule is not wrong. For a solo project or a team with very predictable hours, it might be genuinely sufficient. The argument here is not that schedules are bad — it's that they have a ceiling, and that ceiling is lower than most teams assume.

The ceiling becomes visible when you look at instance-hours. A schedule that runs 7 AM–10 PM captures the nights and weekends. The instance-hours lost to within-day idle time — meetings, gaps, asynchronous work patterns, time zones — sit inside the schedule window and are invisible to it.

Those hours are not catastrophic individually. Across a team over a month, they add up to a billing line that the schedule was supposed to eliminate.

Replacing the Cron

The operational argument for dropping the fixed schedule in favor of activity-driven control is not just cost. It's maintenance.

Cron jobs and CloudWatch rules have owners, and owners leave. The schedule that's running your EC2 stop/start today was probably set up by someone who has since moved to a different role or left the company. The logic lives in a Lambda function in a region someone had to remember to check. The schedule is six months stale and nobody is confident enough to change it.

Activity-driven automation has no schedule to go stale. The pause/resume decision is made fresh on every session, based on what the engineer is actually doing. There is nothing to update when a new hire joins or when a team shifts to async-first.

You stop managing a schedule. The resources manage themselves around the people using them.

If your team is running EC2 dev boxes, staging ECS services, or shared RDS instances on a fixed stop/start schedule — or no schedule at all — Trigops does the activity-driven layer. It connects to your AWS accounts, maps resources to the engineers who use them, and pauses them when those engineers are idle. No cron required.

trigops.com — connect a cloud account and see which of your resources are running when nobody is home.