The cross-reference approach you describe is the one I'd trust most — listing groups with no retention against the functions that still exist makes the orphan set explicit instead of guessed. I'd still lean toward deleting after the snapshot rather than letting retention fade them out, since a group with no owner and no retention is exactly the kind of thing that quietly accumulates. Do you store the snapshot somewhere the team can find it later, or is it more of a one-off record before the delete runs?
One-off, but one that ends up in git. Before the delete runs, the script writes the describe-log-groups output for the candidates (name, storedBytes, creationTime) to a dated file in the same ops repo, and the PR that ran it is the record. Enough so far: the question that comes back later is "what did we delete and when", not "give me the bytes". If someone wants the content, create-export-task to S3 before delete-log-group covers it.
I agree that fading them out is the wrong default. The guard I keep on the delete is list-tags-for-resource per candidate: anything tagged retain, or still in a Terraform state with skip_destroy, gets skipped and listed in that same file. A tag is the one thing a compliance rule can leave on a group with no owner.
That answers it neatly — a dated file in the ops repo, with the PR as the record, covers "what did we delete and when" without keeping the bytes around forever. I especially like the guard: checking list-tags-for-resource for retain and skip_destroy before deleting turns a cleanup script into something a compliance rule can actually trust. Have you ever had a group survive that check and later turn out to be genuinely needed, or does the skipped list mostly stay quiet?
Not yet. The skipped list has stayed short and quiet, and nothing on it has turned out to matter the other way either. The near miss I had was in the cross-reference, not in the guard: Lambda@Edge writes to /aws/lambda/us-east-1.<name> in every Region where the function ran, while the function itself only exists in us-east-1, so a per-Region list of "groups with no function" flags all of those as orphans. Since then the script also reads describe-log-streams --order-by LastEventTime --descending --limit 1 per candidate, and anything with an event in the last 30 days is not an orphan no matter what the function list says. Empty on both counts is the only thing that gets deleted.
That's a great catch on Lambda@Edge — the log groups living in every Region while the function only exists in us-east-1 is exactly the kind of false positive that would make a cleanup script scary to run. Adding the 30-day last-event check on top of the function list feels like the right belt-and-braces approach. Do you run this as a scheduled job, or is it still something you kick off manually when the orphan list gets long enough?
Split in two. The listing runs on a schedule (a weekly cron, read-only: describe-log-groups, the function cross-reference and the last-event check) and writes the candidate file to the ops repo as a PR. The delete never runs on a schedule: someone reads the PR, drops anything they recognise, and merges, and the merge is what runs the delete. Weekly keeps the list short enough that reading it takes a few minutes; the one time we let it pile up for a quarter nobody wanted to review 200 names.
I run servers for a company in China — OA, databases, NAS, the lot. At night I build tools I couldn't buy. Now: RansomShield, air-gapped backup for Windows that disconnects the drive after every run.
Same problem here, just with old SQL Server jobs and scheduled tasks instead of Lambda. What finally worked was making ownership a required attribute rather than a cleanup project: anything without an owner tag goes on a list, gets exported to cold storage with a name and date range manifest, then gets deleted 30 days later unless someone claims it. The export is what saves you in an audit, not the tag.
For the compliance side I keep a small allowlist file in git naming the resources we must keep, and the delete script skips them by name. Tags get edited without a trace. A file in git has a commit history and a reason attached to it.
The export-then-wait-30-days part is the piece most people skip, and I agree that is what saves you in an audit. One thing I would push back on for AWS: a tag edit on a log group is not traceless. CloudTrail records UntagResource and DeleteLogGroup for CloudWatch Logs, so who removed the tag and when is there. What a tag cannot carry is the reason, and that is where the file in git wins. Where I ended up is both: the allowlist in git names what must stay, and the tag is what the script reads at run time, because the file does not travel to the account. For new functions the fix is upstream: LoggingConfig.LogGroup lets the function log to a group Terraform created with retention and tags, instead of the one Lambda creates on first invoke.
We're a place where coders share, stay up-to-date and grow their careers.
The cross-reference approach you describe is the one I'd trust most — listing groups with no retention against the functions that still exist makes the orphan set explicit instead of guessed. I'd still lean toward deleting after the snapshot rather than letting retention fade them out, since a group with no owner and no retention is exactly the kind of thing that quietly accumulates. Do you store the snapshot somewhere the team can find it later, or is it more of a one-off record before the delete runs?
One-off, but one that ends up in git. Before the delete runs, the script writes the
describe-log-groupsoutput for the candidates (name,storedBytes,creationTime) to a dated file in the same ops repo, and the PR that ran it is the record. Enough so far: the question that comes back later is "what did we delete and when", not "give me the bytes". If someone wants the content,create-export-taskto S3 beforedelete-log-groupcovers it.I agree that fading them out is the wrong default. The guard I keep on the delete is
list-tags-for-resourceper candidate: anything tagged retain, or still in a Terraform state withskip_destroy, gets skipped and listed in that same file. A tag is the one thing a compliance rule can leave on a group with no owner.That answers it neatly — a dated file in the ops repo, with the PR as the record, covers "what did we delete and when" without keeping the bytes around forever. I especially like the guard: checking list-tags-for-resource for retain and skip_destroy before deleting turns a cleanup script into something a compliance rule can actually trust. Have you ever had a group survive that check and later turn out to be genuinely needed, or does the skipped list mostly stay quiet?
Not yet. The skipped list has stayed short and quiet, and nothing on it has turned out to matter the other way either. The near miss I had was in the cross-reference, not in the guard: Lambda@Edge writes to
/aws/lambda/us-east-1.<name>in every Region where the function ran, while the function itself only exists in us-east-1, so a per-Region list of "groups with no function" flags all of those as orphans. Since then the script also readsdescribe-log-streams --order-by LastEventTime --descending --limit 1per candidate, and anything with an event in the last 30 days is not an orphan no matter what the function list says. Empty on both counts is the only thing that gets deleted.That's a great catch on Lambda@Edge — the log groups living in every Region while the function only exists in us-east-1 is exactly the kind of false positive that would make a cleanup script scary to run. Adding the 30-day last-event check on top of the function list feels like the right belt-and-braces approach. Do you run this as a scheduled job, or is it still something you kick off manually when the orphan list gets long enough?
Split in two. The listing runs on a schedule (a weekly cron, read-only: describe-log-groups, the function cross-reference and the last-event check) and writes the candidate file to the ops repo as a PR. The delete never runs on a schedule: someone reads the PR, drops anything they recognise, and merges, and the merge is what runs the delete. Weekly keeps the list short enough that reading it takes a few minutes; the one time we let it pile up for a quarter nobody wanted to review 200 names.
Same problem here, just with old SQL Server jobs and scheduled tasks instead of Lambda. What finally worked was making ownership a required attribute rather than a cleanup project: anything without an owner tag goes on a list, gets exported to cold storage with a name and date range manifest, then gets deleted 30 days later unless someone claims it. The export is what saves you in an audit, not the tag.
For the compliance side I keep a small allowlist file in git naming the resources we must keep, and the delete script skips them by name. Tags get edited without a trace. A file in git has a commit history and a reason attached to it.
The export-then-wait-30-days part is the piece most people skip, and I agree that is what saves you in an audit. One thing I would push back on for AWS: a tag edit on a log group is not traceless. CloudTrail records
UntagResourceandDeleteLogGroupfor CloudWatch Logs, so who removed the tag and when is there. What a tag cannot carry is the reason, and that is where the file in git wins. Where I ended up is both: the allowlist in git names what must stay, and the tag is what the script reads at run time, because the file does not travel to the account. For new functions the fix is upstream:LoggingConfig.LogGrouplets the function log to a group Terraform created with retention and tags, instead of the one Lambda creates on first invoke.