DEV Community

Cover image for Deleting a Lambda leaves its log group behind. What do you do with the ones nobody owns?
Gonzalo Gjuka
Gonzalo Gjuka

Posted on

Deleting a Lambda leaves its log group behind. What do you do with the ones nobody owns?

This week I went through an AWS account where Lambda functions come and go every sprint. The functions were gone. Their log groups were not. CloudWatch creates /aws/lambda/<name> on the first invoke with retention set to "Never expire", and deleting the function does not touch it. The Lambda docs say it in one sentence; nobody reads that sentence until the bill does.

The count was not dramatic in dollars. A few hundred GB of logs nobody will ever open, at the storage rate that is a few dollars a month. The part that bothered me was the ownership. The team that wrote those functions moved on, the Terraform that created them never declared the log group, and the destroy that removed the function left the group with no stack, no tag and no owner. It sits in the list between groups that matter.

AWS added account-level retention policies this month, so now one put-account-policy with a /aws/lambda prefix puts an expiry on all of them, existing ones included. That stops the growth. It does not delete the group, and it does not tell you which ones still have a function behind them.

My current answer is a cross: list the groups with no retention, list the functions that exist, and treat the difference as orphans to delete after a snapshot of the names. It works, and it is a script I have to remember to run.

So the question for people running more accounts than I do: do you delete orphaned log groups at all, or do you set retention and let them fade to zero bytes? And if you delete, what stops you from deleting the one that a compliance rule wanted kept?

Top comments (29)

Collapse
 
iictp profile image
IICTP •

The cross-reference approach you describe is the one I'd trust most — listing groups with no retention against the functions that still exist makes the orphan set explicit instead of guessed. I'd still lean toward deleting after the snapshot rather than letting retention fade them out, since a group with no owner and no retention is exactly the kind of thing that quietly accumulates. Do you store the snapshot somewhere the team can find it later, or is it more of a one-off record before the delete runs?

Collapse
 
gonzalogjuka profile image
Gonzalo Gjuka •

One-off, but one that ends up in git. Before the delete runs, the script writes the describe-log-groups output for the candidates (name, storedBytes, creationTime) to a dated file in the same ops repo, and the PR that ran it is the record. Enough so far: the question that comes back later is "what did we delete and when", not "give me the bytes". If someone wants the content, create-export-task to S3 before delete-log-group covers it.

I agree that fading them out is the wrong default. The guard I keep on the delete is list-tags-for-resource per candidate: anything tagged retain, or still in a Terraform state with skip_destroy, gets skipped and listed in that same file. A tag is the one thing a compliance rule can leave on a group with no owner.

Collapse
 
iictp profile image
IICTP •

That answers it neatly — a dated file in the ops repo, with the PR as the record, covers "what did we delete and when" without keeping the bytes around forever. I especially like the guard: checking list-tags-for-resource for retain and skip_destroy before deleting turns a cleanup script into something a compliance rule can actually trust. Have you ever had a group survive that check and later turn out to be genuinely needed, or does the skipped list mostly stay quiet?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Not yet. The skipped list has stayed short and quiet, and nothing on it has turned out to matter the other way either. The near miss I had was in the cross-reference, not in the guard: Lambda@Edge writes to /aws/lambda/us-east-1.<name> in every Region where the function ran, while the function itself only exists in us-east-1, so a per-Region list of "groups with no function" flags all of those as orphans. Since then the script also reads describe-log-streams --order-by LastEventTime --descending --limit 1 per candidate, and anything with an event in the last 30 days is not an orphan no matter what the function list says. Empty on both counts is the only thing that gets deleted.

Thread Thread
 
iictp profile image
IICTP •

That's a great catch on Lambda@Edge — the log groups living in every Region while the function only exists in us-east-1 is exactly the kind of false positive that would make a cleanup script scary to run. Adding the 30-day last-event check on top of the function list feels like the right belt-and-braces approach. Do you run this as a scheduled job, or is it still something you kick off manually when the orphan list gets long enough?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Split in two. The listing runs on a schedule (a weekly cron, read-only: describe-log-groups, the function cross-reference and the last-event check) and writes the candidate file to the ops repo as a PR. The delete never runs on a schedule: someone reads the PR, drops anything they recognise, and merges, and the merge is what runs the delete. Weekly keeps the list short enough that reading it takes a few minutes; the one time we let it pile up for a quarter nobody wanted to review 200 names.

Thread Thread
 
iictp profile image
IICTP •

That split is the part I like most — the listing can run on a schedule because it's read-only, but the delete only ever happens through a human merging a PR. Keeping the list short enough to actually read in a few minutes is a nice detail too; a 200-name backlog would just get rubber-stamped. Do the reviews usually catch anything, or has the candidate list been clean enough that the PR is mostly a formality by now?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Mostly a formality, and I think that is the healthy state: the script already did the lookups, so the reviewer only adds the one thing it cannot see, which is intent. What a review has actually caught is a group kept on purpose. One was a deleted function whose last months of logs someone still wanted for an audit. Another had a metric filter feeding an alarm nobody had removed. The listing now runs describe-metric-filters and describe-subscription-filters per candidate so that second case shows up in the PR body, but "keep this for the audit" only lives in a reviewer's head, and that is the part worth the two minutes.

Thread Thread
 
iictp profile image
IICTP •

Those two catches are exactly the kind of thing a script can't see — the audit-retention one especially, since 'keep this for the audit' only lives in a reviewer's head. Running describe-metric-filters and describe-subscription-filters per candidate so the alarm case shows up in the PR body is a nice touch; it turns a silent dependency into something the reviewer can actually judge. Once a group is kept for an audit, do you tag it so the next listing skips it on its own, or does the reviewer have to re-catch it every run?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Tag it, and give the tag an end. When a reviewer keeps a group on purpose, the merge of that PR runs aws logs tag-resource --resource-arn <group arn> --tags keep=audit,until=2027-03 on it, and the listing calls list-tags-for-resource on each candidate and drops the ones whose until is still in the future. The next run does not re-ask. Once the date passes the group comes back on the list, so "keep this" is a decision with an expiry instead of a memory. I also stopped leaving kept groups on Never expire: the same PR sets put-retention-policy --retention-in-days 400 (the valid values jump from 365 to 400, then 545), so even if nobody ever revisits the tag, the bytes stop growing on their own.

Thread Thread
 
iictp profile image
IICTP •

The expiry on the tag is the elegant part — it turns a permanent exception into a decision with an end date, and the listing not re-asking is exactly right. The 400-day retention backstop is a nice safety net too, since it caps the cost even if nobody revisits the tag. Do you re-check the until dates on a schedule so a group surfaces again on time, or does it only come back when the next cleanup pass happens to run?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Only when the pass runs. There is no second job watching the dates: the listing is the schedule, and the until tag is read right there, in the list-tags-for-resource call on each candidate. So a kept group comes back on the first pass after its month, which with a weekly cron means up to a week late, and that gap is what the 400-day retention is for: by then the bytes are already capped, so being late costs nothing. If I ever needed it on the day, I would put the until month on the same calendar as the cron rather than build a watcher for one tag.

Thread Thread
 
iictp profile image
IICTP •

That makes a lot of sense — 'the listing is the schedule' is a neat way to put it, and reading the until tag right in the list-tags-for-resource call keeps the whole thing to one pass. The weekly cron plus the 400-day cap feels like the right trade: a week late costs nothing when the bytes are already bounded. Do you run the sweep as a plain scheduled Lambda, or is it part of a wider housekeeping job that handles other orphaned resources too?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Plain script, not a Lambda. It runs from the same scheduled runner that already holds read credentials for every account, loops the regions, and writes the candidate list into a ticket that a person reads. Putting it in a Lambda would mean one function per account plus somewhere to collect the output, and the sweep takes under a minute, so the function buys nothing. It is also on purpose separate from the other housekeeping. Disks with no VM, reserved IPs with nothing behind them and the NIC that keeps an IP hostage after a VM delete I look at in Control Cloud, where I work: the listing is automatic there and the delete is a button a person presses. Log groups are not in that inventory, which is why the script exists at all. Do you have one job for everything, or one per resource type?

Thread Thread
 
iictp profile image
IICTP •

One job for everything makes sense to me here — the listing is already the schedule, and splitting it per resource type would just multiply the runners for the same read-only sweep. The ticket as the review surface is the part I like most, since the human decision stays in one place. Do the other housekeeping jobs — disks, IPs, NICs — share the same runner and ticket format, or does each have its own?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Same runner, different shape. Disks, IPs and NICs are the read-only inventory I kept in the product: every item gets a verdict, orphan, in-use or unknown, plus a reason (detached, unassociated, nicNoVm, noAttachments), and unknown is never treated as orphan. That rule came from a real trap: an ENI with no instance id is not abandoned, NAT gateways, load balancers, RDS and VPC endpoints all carry interfaces with no instance, so the NIC check believes the provider's in-use flag before it believes the empty attachment. Log groups stay a separate script because the signal is different: nothing to attach to, only retention, bytes and the until tag. The ticket is the same table on purpose, id, account, region, monthly cost, reason, until, so the reviewer reads one shape. Deletion is manual in both.

Thread Thread
 
iictp profile image
IICTP •

Thanks for the detail, Gonzalo — the 'unknown is never treated as orphan' rule is the part that sticks with me, and the ENI example shows exactly why it matters. With log groups the signal is thinner, just retention, bytes and the until tag — when a log group lands in unknown there, does it get resolved by age, or does it sit for a human to check?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

It sits, and age never promotes it. Unknown in the log group script is not a verdict about the group, it means a lookup did not finish: describe-log-streams throttled (25 transactions per second, which a sweep over some hundred groups hits), an AccessDenied in one account, a Region where the function list failed. Those rows go on the ticket with the error text and the delete skips them. The next weekly pass runs the same calls, so a throttle clears itself and a permissions error comes back every week until a person fixes the role. The one data case I treat the same way is a group with no streams at all: there is no lastEventTimestamp to read, so I fall back to creationTime. Older than 30 days, no function, zero bytes, the script deletes it; younger waits for the next pass, because a group created ahead of its first invocation looks the same.

Thread Thread
 
iictp profile image
IICTP •

That distinction between a verdict about the data and a verdict about the group is the part I'll remember — age as 'runs since someone last had to decide' puts it sharply. It also pairs well with the tag-expiry idea: the unknown stays unknown until either a human closes the loop or the clock does. Has an unknown ever sat long enough that the retention policy quietly finished it before anyone decided?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Once, and it taught me that retention never finishes anything: it deletes events, not the group. One group sat in unknown for three weeks behind an AccessDenied in a member account. When the role got fixed, the 14-day retention had already emptied it: storedBytes at 0 (events marked for deletion stop counting, up to 72 hours after the date), no function, and the streams still there with a lastEventTimestamp five weeks old, because that field moves on ingestion and nothing else. The next pass deleted it as a plain orphan. Nobody decided, but there was nothing left to decide about: the data was already gone, which is the one case where the delete costs nothing. What I changed after that: the ticket row for an unknown now carries the group's retention, so whoever reads it sees how long the evidence survives while the lookup stays broken.

Thread Thread
 
iictp profile image
IICTP •

That's a great war story — and a sharp lesson that retention is a clock on the evidence, not a cleanup step. The detail that storedBytes hits 0 while the streams still sit there with a five-week-old timestamp is exactly the kind of thing that makes 'unknown' dangerous to treat as 'safe to delete later.' Putting the retention on the ticket row so the reviewer sees how long the evidence survives is a really practical fix. Has that changed how long you let an unknown sit before escalating the broken lookup itself?

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka • • Edited

Yes, and the number follows the retention now, not the calendar. Before that case an unknown had no clock: the weekly pass retried, the ticket carried the error, and nothing nagged anyone. The rule now is two passes, with one exception. A throttle never escalates, the next pass clears it. A permissions or Region error that survives two consecutive passes, a week of broken lookup, goes to the account owner as its own item, separate from the cleanup ticket, with the error text and the days of retention left. On a 14-day group that leaves one week of evidence; on a 7-day group that is already late, so those escalate on the first failed pass. Never-expire groups keep the two-pass rule, their data is not going anywhere.

Thread Thread
 
iictp profile image
IICTP •

That answers my question exactly — and the 7-day carve-out is the part that really lands for me, since a week of broken lookup on a 7-day group means the evidence is already gone by the time a second pass would even run. Splitting the escalation into its own item for the account owner, separate from the cleanup ticket, also feels right: the cleanup keeps its rhythm while the broken lookup gets its own clock. Do the account owners ever push back on owning those items, or has the error text plus the days of retention left on the ticket been enough to make it stick?

Collapse
 
mariosfong profile image
Mario Fang •

Same problem here, just with old SQL Server jobs and scheduled tasks instead of Lambda. What finally worked was making ownership a required attribute rather than a cleanup project: anything without an owner tag goes on a list, gets exported to cold storage with a name and date range manifest, then gets deleted 30 days later unless someone claims it. The export is what saves you in an audit, not the tag.

For the compliance side I keep a small allowlist file in git naming the resources we must keep, and the delete script skips them by name. Tags get edited without a trace. A file in git has a commit history and a reason attached to it.

Collapse
 
gonzalogjuka profile image
Gonzalo Gjuka •

The export-then-wait-30-days part is the piece most people skip, and I agree that is what saves you in an audit. One thing I would push back on for AWS: a tag edit on a log group is not traceless. CloudTrail records UntagResource and DeleteLogGroup for CloudWatch Logs, so who removed the tag and when is there. What a tag cannot carry is the reason, and that is where the file in git wins. Where I ended up is both: the allowlist in git names what must stay, and the tag is what the script reads at run time, because the file does not travel to the account. For new functions the fix is upstream: LoggingConfig.LogGroup lets the function log to a group Terraform created with retention and tags, instead of the one Lambda creates on first invoke.

Collapse
 
mariosfong profile image
Mario Fang •

Fair pushback — you're right that CloudTrail records the UntagResource and DeleteLogGroup calls, so the who and when is there. My SQL Server world doesn't have an equivalent, which is probably why I leaned so hard on the git file. I like your "both" setup: git holds the reason, the tag is what the script reads at runtime. And LoggingConfig.LogGroup is new to me — fixing it upstream so Lambda never creates the orphan group in the first place is clearly the better fix. Thanks for that.

Thread Thread
 
gonzalogjuka profile image
Gonzalo Gjuka •

Nothing to add on the SQL Server side, that gap is real. The upstream fix has one condition worth knowing before you lean on it: the group named in LoggingConfig still has to be declared in the same stack with its retention, otherwise you only moved where the orphan is born.

Collapse
 
usernamevalid profile image
Shirley •

Thank you for this, and for naming the ownership problem rather than just the cost. The dollars are small, but a log group with no stack, no tag, and no owner is exactly the kind of resource nobody feels entitled to touch, which is why it survives for years.

On your question, I'd lean toward retention first and deletion second, mostly because the two actions carry very different risk. Setting an account-level retention policy is reversible in spirit (you can lengthen it later, though expired data is gone) and it stops the growth immediately. Deleting a group is permanent, and the one you delete is always the one someone needed during an incident review or an audit six months later. Letting empty groups fade to near-zero bytes costs almost nothing, so the urgency to delete is lower than it feels. What remains is clutter, which matters for hygiene but rarely for the bill.

For the compliance worry, a few guardrails I've seen work:

  • Separate "no function" from "no owner." Your cross of groups against existing functions identifies candidates, but it can't tell you whether a rule requires the logs to be kept. Treat the output as a review list, not a delete list.
  • Check for other consumers first. Subscription filters, metric filters, and export tasks to S3 are a good signal that someone downstream still cares about a group, even when the function is gone.
  • Use a quarantine step. Rather than deleting, export the group to S3 with a lifecycle rule, or tag it with a "scheduled for deletion" date and let a second run remove it after a cooling-off period. A tag also gives you the audit trail your current script lacks.
  • Use last-event time as evidence. A group with no ingestion in 90 or 180 days and no function behind it is a much safer candidate than one that merely lacks retention.

The longer-term fix is upstream, as you hinted: declare the log group in the same Terraform or CDK stack as the function, with retention set and a deletion policy chosen deliberately. Then the destroy takes the group with it, or leaves it with an explicit tag saying why. That turns the cleanup script from a habit you have to remember into a safety net for the legacy groups that predate the convention.

So my answer would be retention everywhere, deletion only for groups that are provably orphaned, idle, unsubscribed, and untagged for compliance, and a stack-level fix so the problem stops reproducing. Curious whether anyone runs the cleanup as a scheduled job with an approval step, since that seems like the natural next evolution of your script.

Collapse
 
gonzalogjuka profile image
Gonzalo Gjuka •

Agree on the order, and the consumer check is the piece my script lacks. Two things from running it. The signals you name are all queryable per group: describe-subscription-filters and describe-metric-filters take --log-group-name; describe-export-tasks does not, so you list them all and match logGroupName in the output. For idle, describe-log-streams --order-by LastEventTime --descending --limit 1 gives the lastEventTimestamp of the newest stream; it updates on eventual consistency, usually under an hour, so 90 days of nothing is a safe reading. On the approval step: mine has none. The until tag is the approval: a person writes it, the cron reads it on each pass and skips the group. No tag and no function means delete. Your quarantine export to S3 with a lifecycle rule is the version I would use where the logs carry a retention obligation.