Every cloud team writes the same Lambda eventually. Cron fires at 8pm, StopInstances on everything tagged env=dev, StartInstances at 8am. It works on the first ten machines, the bill dips, everyone moves on. Eighteen months later there are three hundred instances, four teams, three timezones, and the savings have quietly gone to zero while the script still reports success.
Nothing dramatic broke. Seven small things did. This post is the taxonomy of how shutdown automation fails at scale, an honest look at AWS Instance Scheduler, and the requirements checklist for scheduling that survives its own success.
The seven ways the script breaks
1. Dependency order. The 8am start brings the app servers up before the database accepts connections. Monday's first deploy of the day fails, someone blames the schedule, and the fix is the worst possible one: that team's instances get quietly removed from scheduling forever. Real infrastructures need ordered startup (storage, then compute, then applications) with delays between steps, because a database that is "running" is not yet a database that is accepting connections.
2. Timezones. The cron expression is in UTC. The Bangalore team's "8pm stop" is the Virginia team's mid-afternoon outage. You end up with either one compromise schedule that saves less for everyone, or per-team cron math that breaks twice a year when daylight saving shifts and the comment above the cron line goes stale.
3. Overrides that never expire. An incident happens on a Thursday night, someone needs the environment up, and the honest quick fix is removing the tag or disabling the rule. The incident ends; the exception doesn't. Indefinite overrides are the single most common way scheduling savings quietly disappear, because every exception is invisible the moment the incident that justified it is forgotten. Overrides need to be time-bounded by policy, with a maximum duration someone senior chose, and an audit trail of who held what open and why.
4. No proof of firing. The Lambda's IAM role got an explicit deny in a security sweep in March. It has thrown AccessDenied every night since. Nobody noticed, because nobody measures whether schedules actually fired; the dashboard measures projected savings, which are a fiction the moment execution stops. (This failure mode deserves its own post, and has one.)
5. Drift. New instances launch without the tag. An autoscaling group replaces its members and the instance IDs your config pinned are gone. Someone resizes an instance and the script's hardcoded assumptions break. At three hundred instances, the population changes daily, and a script keyed to a point-in-time inventory decays at the same rate.
6. Partial failures at API limits. Three hundred StopInstances calls in one burst meets RequestLimitExceeded. Two-thirds of the fleet stops; a third doesn't; the script exits zero because the exception handler was written for the ten-instance era. You now have the worst of both worlds: inconsistent state and a green checkmark.
7. The things that aren't EC2. RDS has its own stop API with its own rules (an RDS instance stopped for seven days starts itself again, which surprises everyone once). Kubernetes namespaces can't be stopped with either. Aurora, Databricks, and autoscaling groups each have their own semantics. The "shutdown script" becomes five scripts with five failure modes.
The honest take on AWS Instance Scheduler
AWS's answer is Instance Scheduler, a CloudFormation solution: DynamoDB holds periods and schedules, a Lambda evaluates them, tags opt instances in. It's a real step up from a hand-rolled script: cross-account, EC2 plus RDS, maintained.
Know what it is and isn't. The solution itself is free; you pay for the Lambda and DynamoDB it runs on (typically a few dollars a month) and for your time operating CloudFormation. Configuration lives in DynamoDB entries, which is workable at ten schedules and painful at a hundred. And the failure modes above mostly remain yours: it has no dependency ordering between resources, no bounded-override workflow, no proof-of-firing reconciliation, and drift management is still tag hygiene. It schedules instances; it doesn't manage a scheduling practice.
The checklist for scheduling that survives scale
Whether you build or buy, this is the list the seven failures imply:
- Ordered sequencing with per-resource delays: storage, then compute, then applications, and a way to express "wait two minutes after the database before the app tier".
- Timezone-native schedules, so an India schedule and a US-East schedule coexist without UTC arithmetic or DST surprises.
- Time-bounded overrides with an admin-set maximum duration, notifications where the team lives (Slack or Teams), and a full audit trail. An override that can't expire is a deletion of the schedule with better manners.
- Reconciliation: every window, compare expected state against actual state and alert on mismatch. Success is measured, not assumed.
- Group-level operations: bundle an app's resources, start and stop them as a unit, in order, with a budget per group.
- Eligibility detection: something that continuously finds resources that could be scheduled but aren't (non-production instances running through weekends), because coverage decays as the fleet grows.
- Resilience to change: schedules must survive instance-type changes, autoscaling replacements, and re-tags, or drift eats the program in a quarter.
That checklist is also a fair evaluation rubric for any scheduling product. For what it's worth, this is the shape ZopNight's scheduling implements: dependency-aware sequencing (storage, compute, applications, with per-resource delays), first-class timezones so an Asia/Kolkata schedule runs beside an America/New_York one, overrides that are time-bounded with an admin-configurable maximum precisely so exceptions can't become permanent, and a 15-minute cycle that checks resources are in the state their schedule expects. The listed capabilities above aren't hypothetical; they're what the failure modes forced.
FAQ
How do I automatically start and stop EC2 instances on a schedule?
Four common paths: a cron-triggered Lambda calling StopInstances/StartInstances (fine at small scale), AWS Instance Scheduler (tag-driven, cross-account, DynamoDB-configured), EventBridge Scheduler calling the EC2 APIs directly, or a scheduling product. The mechanism is the easy part; dependency order, overrides, proof of firing, and drift are where the choice actually matters.
Is AWS Instance Scheduler free?
The solution is free to deploy; you pay for the underlying Lambda, DynamoDB, and CloudWatch usage (typically single-digit dollars monthly) plus the operational cost of managing it via CloudFormation and DynamoDB entries. The bigger cost is what it doesn't do: no sequencing, no bounded overrides, no firing reconciliation.
How do I schedule RDS start and stop?
RDS has native stop/start APIs with a catch: a stopped RDS instance automatically restarts after seven days, so weekly-cycle schedules must account for it. Multi-AZ and read-replica configurations have additional restrictions. Schedule databases to stop after their dependent applications, and start before them, with a delay for connection readiness.
What tags should drive instance scheduling?
One schedule tag (like schedule=office-hours-ist) mapping to a named schedule definition, not per-instance cron strings. Enforce it at provision time via IaC modules, and run a weekly report of non-production resources missing the tag, because untagged drift is how coverage decays.
How do I stop a shutdown script from breaking my application startup?
Sequence by tier with explicit delays: storage and databases first, application compute last on start, and the reverse on stop. Then verify: a post-start health check per group beats assuming the order worked. If your tooling can't express ordering, the workaround is separate schedules offset by conservative gaps, which wastes some savings but protects mornings.
Top comments (0)