DEV Community

Cover image for Deleting a Lambda leaves its log group behind. What do you do with the ones nobody owns?

Deleting a Lambda leaves its log group behind. What do you do with the ones nobody owns?

Gonzalo Gjuka on November 15, 2026

This week I went through an AWS account where Lambda functions come and go every sprint. The functions were gone. Their log groups were not. CloudW...
Collapse
 
iictp profile image
IICTP •

The cross-reference approach you describe is the one I'd trust most — listing groups with no retention against the functions that still exist makes the orphan set explicit instead of guessed. I'd still lean toward deleting after the snapshot rather than letting retention fade them out, since a group with no owner and no retention is exactly the kind of thing that quietly accumulates. Do you store the snapshot somewhere the team can find it later, or is it more of a one-off record before the delete runs?

Collapse
 
gonzalogjuka profile image
Gonzalo Gjuka •

One-off, but one that ends up in git. Before the delete runs, the script writes the describe-log-groups output for the candidates (name, storedBytes, creationTime) to a dated file in the same ops repo, and the PR that ran it is the record. Enough so far: the question that comes back later is "what did we delete and when", not "give me the bytes". If someone wants the content, create-export-task to S3 before delete-log-group covers it.

I agree that fading them out is the wrong default. The guard I keep on the delete is list-tags-for-resource per candidate: anything tagged retain, or still in a Terraform state with skip_destroy, gets skipped and listed in that same file. A tag is the one thing a compliance rule can leave on a group with no owner.

Collapse
 
iictp profile image
IICTP •

That answers it neatly — a dated file in the ops repo, with the PR as the record, covers "what did we delete and when" without keeping the bytes around forever. I especially like the guard: checking list-tags-for-resource for retain and skip_destroy before deleting turns a cleanup script into something a compliance rule can actually trust. Have you ever had a group survive that check and later turn out to be genuinely needed, or does the skipped list mostly stay quiet?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Not yet. The skipped list has stayed short and quiet, and nothing on it has turned out to matter the other way either. The near miss I had was in the cross-reference, not in the guard: Lambda@Edge writes to /aws/lambda/us-east-1.<name> in every Region where the function ran, while the function itself only exists in us-east-1, so a per-Region list of "groups with no function" flags all of those as orphans. Since then the script also reads describe-log-streams --order-by LastEventTime --descending --limit 1 per candidate, and anything with an event in the last 30 days is not an orphan no matter what the function list says. Empty on both counts is the only thing that gets deleted.

Thread Thread
 
iictp profile image
IICTP •

That's a great catch on Lambda@Edge — the log groups living in every Region while the function only exists in us-east-1 is exactly the kind of false positive that would make a cleanup script scary to run. Adding the 30-day last-event check on top of the function list feels like the right belt-and-braces approach. Do you run this as a scheduled job, or is it still something you kick off manually when the orphan list gets long enough?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Split in two. The listing runs on a schedule (a weekly cron, read-only: describe-log-groups, the function cross-reference and the last-event check) and writes the candidate file to the ops repo as a PR. The delete never runs on a schedule: someone reads the PR, drops anything they recognise, and merges, and the merge is what runs the delete. Weekly keeps the list short enough that reading it takes a few minutes; the one time we let it pile up for a quarter nobody wanted to review 200 names.

Collapse
 
mariosfong profile image
Mario Fang •

Same problem here, just with old SQL Server jobs and scheduled tasks instead of Lambda. What finally worked was making ownership a required attribute rather than a cleanup project: anything without an owner tag goes on a list, gets exported to cold storage with a name and date range manifest, then gets deleted 30 days later unless someone claims it. The export is what saves you in an audit, not the tag.

For the compliance side I keep a small allowlist file in git naming the resources we must keep, and the delete script skips them by name. Tags get edited without a trace. A file in git has a commit history and a reason attached to it.

Collapse
 
gonzalogjuka profile image
Gonzalo Gjuka •

The export-then-wait-30-days part is the piece most people skip, and I agree that is what saves you in an audit. One thing I would push back on for AWS: a tag edit on a log group is not traceless. CloudTrail records UntagResource and DeleteLogGroup for CloudWatch Logs, so who removed the tag and when is there. What a tag cannot carry is the reason, and that is where the file in git wins. Where I ended up is both: the allowlist in git names what must stay, and the tag is what the script reads at run time, because the file does not travel to the account. For new functions the fix is upstream: LoggingConfig.LogGroup lets the function log to a group Terraform created with retention and tags, instead of the one Lambda creates on first invoke.

Collapse
 
mariosfong profile image
Mario Fang •

Fair pushback — you're right that CloudTrail records the UntagResource and DeleteLogGroup calls, so the who and when is there. My SQL Server world doesn't have an equivalent, which is probably why I leaned so hard on the git file. I like your "both" setup: git holds the reason, the tag is what the script reads at runtime. And LoggingConfig.LogGroup is new to me — fixing it upstream so Lambda never creates the orphan group in the first place is clearly the better fix. Thanks for that.

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Nothing to add on the SQL Server side, that gap is real. The upstream fix has one condition worth knowing before you lean on it: the group named in LoggingConfig still has to be declared in the same stack with its retention, otherwise you only moved where the orphan is born.

Collapse
 
usernamevalid profile image
Shirley •

Thank you for this, and for naming the ownership problem rather than just the cost. The dollars are small, but a log group with no stack, no tag, and no owner is exactly the kind of resource nobody feels entitled to touch, which is why it survives for years.

On your question, I'd lean toward retention first and deletion second, mostly because the two actions carry very different risk. Setting an account-level retention policy is reversible in spirit (you can lengthen it later, though expired data is gone) and it stops the growth immediately. Deleting a group is permanent, and the one you delete is always the one someone needed during an incident review or an audit six months later. Letting empty groups fade to near-zero bytes costs almost nothing, so the urgency to delete is lower than it feels. What remains is clutter, which matters for hygiene but rarely for the bill.

For the compliance worry, a few guardrails I've seen work:

  • Separate "no function" from "no owner." Your cross of groups against existing functions identifies candidates, but it can't tell you whether a rule requires the logs to be kept. Treat the output as a review list, not a delete list.
  • Check for other consumers first. Subscription filters, metric filters, and export tasks to S3 are a good signal that someone downstream still cares about a group, even when the function is gone.
  • Use a quarantine step. Rather than deleting, export the group to S3 with a lifecycle rule, or tag it with a "scheduled for deletion" date and let a second run remove it after a cooling-off period. A tag also gives you the audit trail your current script lacks.
  • Use last-event time as evidence. A group with no ingestion in 90 or 180 days and no function behind it is a much safer candidate than one that merely lacks retention.

The longer-term fix is upstream, as you hinted: declare the log group in the same Terraform or CDK stack as the function, with retention set and a deletion policy chosen deliberately. Then the destroy takes the group with it, or leaves it with an explicit tag saying why. That turns the cleanup script from a habit you have to remember into a safety net for the legacy groups that predate the convention.

So my answer would be retention everywhere, deletion only for groups that are provably orphaned, idle, unsubscribed, and untagged for compliance, and a stack-level fix so the problem stops reproducing. Curious whether anyone runs the cleanup as a scheduled job with an approval step, since that seems like the natural next evolution of your script.

Collapse
 
gonzalogjuka profile image
Gonzalo Gjuka •

Agree on the order, and the consumer check is the piece my script lacks. Two things from running it. The signals you name are all queryable per group: describe-subscription-filters and describe-metric-filters take --log-group-name; describe-export-tasks does not, so you list them all and match logGroupName in the output. For idle, describe-log-streams --order-by LastEventTime --descending --limit 1 gives the lastEventTimestamp of the newest stream; it updates on eventual consistency, usually under an hour, so 90 days of nothing is a safe reading. On the approval step: mine has none. The until tag is the approval: a person writes it, the cron reads it on each pass and skips the group. No tag and no function means delete. Your quarantine export to S3 with a lifecycle rule is the version I would use where the logs carry a retention obligation.