My drift detector cried wolf every time my autoscaler scaled out — and the fix was a tag AWS already sets

A while back I wrote about the first time my self-hosted drift detector cried
wolf: a diff bug that flagged phantom changes on every scan. I fixed it and
felt clever.

Then I pointed it at an account with an Auto Scaling group, and it cried
wolf again — for a completely different reason, and this time the tool was
technically right.

Every time the ASG scaled out, the new instance had an instance ID the previous
scan had never seen. My drift report dutifully counted it as an added
resource. Scale in, scale out, Tuesday afternoon traffic — a steady drip of
"drift" that was nothing of the sort. The autoscaler launching an instance isn't
your infrastructure drifting from Terraform. It's the system working exactly as
designed.

Why "added" is the wrong verdict here

Attribute drift is easy to reason about: tfstate says t3.micro, live AWS says
t3.large, that's drift. "Added" is trickier, because a resource that wasn't
there and now is usually means someone created something out-of-band — the
exact console click I built this tool to catch.

But an ASG is different. Terraform describes the group — min, max, desired,
launch template. It does not describe each individual instance; those are the
autoscaler's runtime decisions. So the identity and count of instances inside an
ASG are never "drift from Terraform." They're supposed to change on their own.

Counting them as drift doesn't just inflate a number. It trains everyone to
ignore the drift report, which quietly kills the one signal you actually needed.

The fix I almost over-engineered

My first instinct was to call DescribeAutoScalingGroups, list each group's
members, and cross-reference. That means a new scanner and a new IAM permission
(autoscaling:DescribeAutoScalingGroups) on every deployment.

Then I looked at what I was already scanning. EC2 stamps every ASG-launched
instance with a reserved tag — aws:autoscaling:groupName — and my EC2 scanner
already reads instance tags. The ownership signal was sitting in my data the
whole time. No new API call, no new permission:

# asset_manager/autoscaling.py — one source of truth for "does an ASG own this?"
ASG_TAG = 'aws:autoscaling:groupName'

def autoscaling_group_of(raw_data):
    if not raw_data:
        return None
    # explicit key from the current scanner, else the tag we already had —
    # so instances scanned before this feature existed classify correctly too
    return raw_data.get('autoscaling_group') or (raw_data.get('tags') or {}).get(ASG_TAG) or None

Users can't set the aws: tag namespace themselves, so its presence is a
reliable "the autoscaler owns this" flag. This is the third time on this project
that the honest fix was "the data already told me" rather than a new API call —
I'm starting to think that's the default, not the exception.

The one line that keeps this from being a foot-gun

Here's the part I want to get right, because the naive version is dangerous.
It's tempting to say "ignore drift on ASG-owned instances." Don't. If you
suppress all drift on autoscaled instances, you go blind to a security group
opened to 0.0.0.0/0 on your entire web fleet — which is exactly the drift you
most want to catch.

So I suppress only the existence dimension — a first-sighting instance the
autoscaler created — and nothing else:

if not asset.raw_data_prev:
    # first sighting: churn if an ASG owns it, otherwise a real "added"
    if is_autoscaling_churn(asset.raw_data):
        autoscaling.append(meta)
    else:
        added.append(meta)
else:
    # an ASG instance that CHANGED attributes is still real drift — reported
    diff = _compute_raw_diff(asset.raw_data_prev, asset.raw_data)
    if diff:
        changed.append({**meta, 'changes': diff})

A new autoscaled instance → churn, filed away. That same instance later showing
an opened security group → drift, front and center. And I show the suppressed
instances in their own "Auto Scaling — not drift" section rather than hiding
them, so the report is honest about what it chose not to alarm on.

One source of truth, or the CLI lies again

My tool computes drift in a few places: the dashboard badge, the recorded
history snapshot, the drift report page — and, since last week, a syncvey
command line. If each of those decided "is this churn?" on its own, they'd
eventually disagree, and a tool that disagrees with itself is worse than no
tool.

So the ownership check lives in exactly one helper that all four import. Which
surfaced a nice catch: the CLI I shipped in my last post has a
syncvey drift --exit-code flag meant to fail a CI build on drift. Before this
change, a routine scale-out would have turned a pipeline red for no reason.
The same one-line fix closed that hole automatically, because the CLI reads the
same function as everything else.

What I didn't fix

Honesty section, as always:

• This suppresses scale-out churn (new instances). Scale-in leaves a stale asset row for a terminated instance — that's a separate "prune what's gone" problem I haven't solved here.
• It trusts the tag. That's a safe bet for aws:autoscaling:groupName, but it's a tag check, not a live ASG-membership lookup — I traded a little precision for zero new IAM, on purpose.
• It's on by default (DRIFT_SUPPRESS_AUTOSCALING), because churn-as-drift is almost never what you want — but you can turn it off if you really want to see every instance the autoscaler ever launched.

Takeaways

• Not every "added" resource is drift. If a system is designed to create and destroy resources on its own (autoscalers, spot fleets, Karpenter), counting that as drift just teaches people to mute the alarm.
• Before adding an API call and an IAM permission, check what you already collect. AWS reserved tags (aws:autoscaling:groupName, aws:cloudformation:*, aws:eks:*) carry a lot of ownership signal for free.
• Suppress narrowly. "Ignore this resource" is a foot-gun; "ignore this one dimension of this resource" (its existence, not its config) keeps you from going blind to the drift that matters.
• If several code paths answer the same question, give them one function to call. The bonus is every consumer — including a CLI you wrote later — inherits the fix.

---

This is a real, open-source (MIT) self-hosted tool that tracks how your live AWS
drifts from Terraform — one docker compose up:
syncvey.com. If you run autoscalers, how does your drift
tooling tell "the autoscaler did its job" apart from "someone changed something"?