Everybody's cloud cost journey has the same first chapter: someone writes a Lambda that stops the dev instances at night and starts them in the morning. It works. It saves real money. And then the environment grows, and one morning the script that ran fine for a year quietly causes an outage. The shutdown script that works on one instance breaks at three hundred, and it breaks in four specific ways. Here is each one, because knowing them is the difference between saving money and writing a postmortem.
The script that works on one instance
# stop_dev.py, EventBridge at 20:00
import boto3
ec2 = boto3.client("ec2")
ids = [i["InstanceId"] for r in ec2.describe_instances(
Filters=[{"Name":"tag:env","Values":["dev"]}])["Reservations"]
for i in r["Instances"]]
ec2.stop_instances(InstanceIds=ids)
At small scale this is fine. At scale, here is what goes wrong.
Break 1: dependency order
Your app instance depends on a database. Stop them in a random order and starting back up, the app comes alive before the database is ready and lands in a crash loop. On one box you get away with it. Across an environment with app tiers, databases, and caches, ordering is not optional: databases up before apps, apps up before the things that call them. A flat list of instance IDs has no concept of "start this after that." Real scheduling needs dependency-aware sequencing (storage, then compute, then application), with delays between tiers.
Break 2: timezones
The script fires at 20:00. Whose 20:00? As you add teams in different regions, a single UTC cron either shuts down someone's environment in the middle of their afternoon or leaves it running all night. At scale, schedules have to be timezone-aware per environment or per team, not one global time that is wrong for most of the world.
Break 3: no overrides, so people disable it
The night QA needs staging up late for a release, the script kills it at 20:00 anyway. This happens twice, and then someone disables the schedule "just for now." It stays off for a year, and all the savings evaporate. At scale you need a first-class override: a "keep this on until midnight" or "hold this environment" button that anyone can use and that expires on its own. Without a safe override, people protect themselves by turning the whole thing off, and an override that requires editing a cron expression is not a real override.
Related trap: an override with no expiry. Someone forces an environment on during an incident and forgets to remove the hold. The savings quietly disappear because half the fleet is now permanently exempt. Overrides must be time-bounded.
Break 4: no proof it actually fired
The script has no memory. Did last night's shutdown actually run? Did every instance stop, or did three fail because of a state transition and keep billing? On one instance you would notice. On three hundred, a 5% silent failure rate means fifteen instances running 24/7 that you think are scheduled, and nothing tells you. At scale you need reconciliation: compare what was scheduled to stop against what actually stopped, and alert on the gap.
What "scheduling at scale" actually requires
Put those four together and the requirements are clear. Real scheduling is:
- Tag-driven discovery, so new resources inherit the schedule automatically instead of being hand-added to a list that rots.
- Dependency-aware sequencing across resource types, with delays.
- Timezone-aware schedules per environment.
- First-class, time-bounded overrides anyone can use.
- Reconciliation and alerting so you know it actually fired.
- Coverage beyond EC2: RDS (which you stop differently, and which auto-restarts after 7 days), Auto Scaling groups, and more.
AWS Instance Scheduler exists and covers some of this, but it is a solution you deploy and operate. This is also exactly the problem ZopNight was built to handle across resource types and clouds, precisely because the DIY script era taught everyone these four failure modes the hard way. Whatever you use, the point is that "stop the dev instances at night" is a harder problem than the first Lambda makes it look.
The take
The one-instance shutdown script is a great way to start and a terrible way to scale. It breaks on dependency order, timezones, overrides, and silent failures, and each break is invisible until it causes an incident or quietly stops saving money. If your environment has outgrown the single Lambda, you do not need a bigger Lambda, you need scheduling that understands dependencies, timezones, overrides, and proof-of-firing.
Which of the four broke first for you? For most people it is the override problem, one badly-timed shutdown during a release, and the whole schedule gets switched off in self-defense.
Top comments (0)