How Do You Know Last Night's Cloud Shutdown Actually Fired? Measuring Schedule Success Rate

Here's a question almost no cloud team can answer with data: did last night's shutdown actually run?

Everyone can answer the neighboring question. "How much are the schedules saving?" has a dashboard: instances times hours times rate, a satisfying monthly number. But that number is a projection. It assumes the schedule fired, every night, on every resource. Nobody assumes their deploys succeeded; there's a pipeline status for that. Schedules, which touch production-adjacent infrastructure every single day, mostly run on faith.

The result is a specific and expensive failure pattern: a schedule silently stops working, the projected-savings dashboard keeps reporting the same number, and the gap between fiction and bill grows for weeks until someone reads an invoice carefully. The fix is a metric: schedule success rate.

How schedules fail silently

Every one of these is from the field, and none of them announces itself:

• The IAM change. A security sweep adds an explicit deny or rotates a role; the scheduler's stop calls start throwing AccessDenied at 8pm when nobody's watching. The function "runs successfully" in the sense that it executes and logs an exception someone will read in October.
• The disabled rule. Someone pauses the EventBridge rule during an incident, intending to re-enable it tomorrow. There is no tomorrow.
• Throttling half a fleet. Three hundred stop calls hit API rate limits; 190 succeed, 110 don't, and the script's exit code reflects whichever call happened last.
• Tag drift. An instance gets re-provisioned without its schedule tag. It didn't fail to stop; it silently left the population that was supposed to stop, which no per-run log will ever show.
• The seven-day surprise. Stopped RDS instances restart themselves after seven days by design. Your Friday-stop schedule works; the databases are quietly running again by the following Friday.

Notice the pattern: half of these aren't execution failures at all. They're population failures, resources drifting out of scheduling scope. Which is why logging "the Lambda ran" is not the metric.

The metric, defined

Schedule success rate = state transitions that actually happened ÷ state transitions that were supposed to happen, per window.

The denominator matters more than the numerator. It comes from intent (every resource carrying a schedule, and what state each should be in after the window), not from what the executor attempted. A resource the executor never tried to touch, because a tag vanished or a rule was disabled, still counts in the denominator. That's the difference between measuring execution and measuring the program.

Pair it with a second number: coverage, the share of schedule-eligible resources actually carrying a schedule. Success rate catches broken firing; coverage catches silent shrinkage of the population. A team holding 99% success on 40% coverage is doing a great job of a small job.

Building it in an afternoon

The reconciliation loop is small:

1. Fifteen to thirty minutes after each schedule boundary, run a checker (EventBridge rule, small Lambda).
2. Ask intent: which resources should now be stopped (or running)? Read the schedule tags or your schedule store.
3. Ask reality: DescribeInstances / DescribeDBInstances for actual state.
4. Emit the reconciliation as a metric and alert below 100%:

aws cloudwatch put-metric-data --namespace "Scheduling" \
  --metric-name ScheduleSuccessRate \
  --value "$(echo "$matched / $expected * 100" | bc -l)" \
  --unit Percent

1. On mismatch, emit which resources missed, because "97%" without names is a mystery novel.

CloudTrail adds the forensic layer: join expected transitions against actual StopInstances / StartInstances events to distinguish "never attempted" (population problem) from "attempted and failed" (execution problem). The two have different owners.

The one design rule: reconcile against expected state, not emitted commands. Checking your own homework by re-reading your own homework catches nothing.

What good looks like

A healthy scheduling program publishes three numbers weekly: success rate (target: 100%, alarmed below it), coverage (trending up), and savings (now credible, because it's built on the first two). When the success gauge dips, someone looks the next morning, not at invoice time. Scheduling products have started treating this as a first-class surface too; ZopNight, for instance, ships a Schedule Success Rate gauge that reconciles every scheduled run against what was supposed to fire, on top of a 15-minute cycle checking that resources are in the state their schedule expects, so a schedule that quietly stopped working shows up instead of hiding. However you get the number, the point is the same: a schedule you don't verify is a savings estimate, not a saving.

FAQ

How do I verify my EC2 scheduled shutdown actually ran?

Reconcile intent against state: shortly after each schedule boundary, list what should be stopped (from schedule tags or your schedule store), compare with actual instance state, and emit the match rate as a metric with an alarm below 100%. Checking the scheduler's own logs only catches execution errors, not population drift like missing tags or disabled rules.

What is a good schedule success rate?

100%, alarmed on anything less. Unlike most SLOs, there's no inherent noise floor: every legitimate exception should exist as an explicit, time-bounded override, which moves it into the "expected" column instead of eroding the metric. A team living at 96% has seven silent failures a week it has agreed not to look at.

Why did my stopped RDS instance start again by itself?

By design: AWS restarts stopped RDS instances after seven days so they don't miss maintenance. Weekly schedules must re-stop them, and your reconciliation should treat a self-started database as a mismatch to catch, not noise to ignore.

What should I monitor besides success rate?

Coverage: the percentage of schedule-eligible resources (non-production compute and databases with office-hours usage patterns) actually carrying a schedule. Success rate without coverage rewards shrinking the program; the pair keeps both failure directions visible.

Does CloudTrail show scheduled stops and starts?

Yes: StopInstances, StartInstances, StopDBInstance, and StartDBInstance events, with the calling identity. Joining "expected transitions" against CloudTrail separates never-attempted from attempted-and-failed, which is the difference between a tagging problem and an IAM problem.