Build a CloudWatch Logs Cost Agent: Find Never-Expire Log Groups, Ingestion Hot Spots, and Debug Floods Before the Bill Does

💡 Originally published on devtocash.com — where this guide stays updated. I write hands-on DevOps/SRE deep-dives there weekly.

The log bill is an ingestion bill, and nobody owns it

Open Cost Explorer, filter to AmazonCloudWatch, group by usage type. In most accounts one line dominates: DataProcessing-Bytes, which is log ingestion at $0.50 per GB. Storage at $0.03 per GB-month is a rounding error next to it. That single fact decides what a CloudWatch Logs cost agent should do: retention fixes are the easy, safe, boring win, but the money is in what gets written, and that needs a per-source diagnosis a human never has time to run across 900 log groups.

This post builds that agent. Deterministic code collects the evidence for every log group: 30-day ingestion from the IncomingBytes metric, stored bytes, retention, log class, subscription and metric filters, tags, and a bounded Logs Insights sample of how much of the stream is DEBUG noise. The LLM classifies each group by what produced it and proposes one of five actions. Retention changes ship as a Terraform pull request under a plan-stage gate. Log-level changes go to the owning team as a diff they can read. The agent's role cannot delete a log group or a single event.

It sits next to the other AWS FinOps agents here: the cost anomaly agent tells you the CloudWatch line jumped, and this one tells you which log group did it and what to change.

Where CloudWatch Logs money hides

Leak	Signal	What it costs (us-east-1 list)
Never-expire groups	retentionInDays absent	$0.03/GB-month, growing linearly forever
Debug floods	High IncomingBytes, high share of DEBUG/TRACE lines	$0.50/GB ingested, Standard class
Lambda default groups	/aws/lambda/* auto-created, no retention, no log level set	Ingestion plus unbounded storage per function
Container Insights performance logs	/aws/containerinsights/*/performance	One JSON event per pod per minute, at $0.50/GB
EKS control plane audit logs	/aws/eks/*/cluster with audit enabled	Often the largest group in the account
Vended flow logs into Logs	/aws/vpc/flowlogs* or similar	$0.50/GB tiered, versus ~$0.25/GB to S3
Logs Insights habit	DataScanned-Bytes usage type	$0.005/GB scanned per query, dashboards re-run it every refresh

Two things in that table surprise people. First, Infrequent Access class is half price on ingestion ($0.25/GB) but can only be chosen at creation, so it's a recreate-and-repoint decision, not a flag flip. Second, the EKS audit log is frequently the top ingestion source in an account, and turning it off is a security decision, not a cost one. The agent has to know which groups it is allowed to have opinions about.

Get the account-level picture first so you can check the agent's arithmetic against the bill:

aws ce get-cost-and-usage \
  --time-period Start=2026-08-01,End=2026-09-01 --granularity MONTHLY \
  --metrics UnblendedCost \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["AmazonCloudWatch"]}}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE \
  --query 'ResultsByTime[0].Groups[?contains(Keys[0], `Bytes`) || contains(Keys[0], `ByteHrs`)].[Keys[0], Metrics.UnblendedCost.Amount]' \
  --output table

The usage type prefix varies by region (USE1-, EUW1-). DataProcessing-Bytes is ingestion, TimedStorage-ByteHrs is storage, DataScanned-Bytes is Logs Insights, VendedLog-Bytes is flow logs and friends. If ingestion is under half the total, your account is unusual and the retention part of this agent matters more than the rest.

Step 1: A role that can read everything and change nothing

Same design as the S3 storage cost agent: no write path at all, plus an explicit deny so a future "helpful" policy attachment can't add one.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadLogGroupsAndMetrics",
      "Effect": "Allow",
      "Action": [
        "logs:DescribeLogGroups", "logs:DescribeSubscriptionFilters",
        "logs:DescribeMetricFilters", "logs:ListTagsForResource",
        "logs:StartQuery", "logs:GetQueryResults", "logs:StopQuery",
        "cloudwatch:GetMetricData", "ce:GetCostAndUsage",
        "lambda:GetFunctionConfiguration", "lambda:ListFunctions"
      ],
      "Resource": "*"
    },
    {
      "Sid": "NeverEvenIfSomeoneAddsIt",
      "Effect": "Deny",
      "Action": [
        "logs:DeleteLogGroup", "logs:DeleteLogStream", "logs:PutRetentionPolicy",
        "logs:DeleteRetentionPolicy", "logs:PutSubscriptionFilter",
        "logs:DeleteSubscriptionFilter", "logs:CreateLogGroup"
      ],
      "Resource": "*"
    }
  ]
}

logs:StartQuery is the one permission with a cost attached, because every Logs Insights query is billed per GB scanned. The sampler in step 2 bounds that by time window, and the deny on PutRetentionPolicy means the agent can only ever propose a retention change.

Step 2: Evidence per log group, computed not prompted

IncomingBytes in the AWS/Logs namespace is published per log group and is exactly what you're billed on for ingestion. One GetMetricData call takes 500 queries, so a thousand groups is three API calls.

import boto3
from datetime import datetime, timedelta, timezone

logs = boto3.client("logs")
cw = boto3.client("cloudwatch")
NOW = datetime.now(timezone.utc)
INGEST_PRICE = {"STANDARD": 0.50, "INFREQUENT_ACCESS": 0.25}
STORAGE_PRICE = 0.03

def all_log_groups():
    for page in logs.get_paginator("describe_log_groups").paginate():
        yield from page["logGroups"]

def incoming_bytes(names, days=30):
    """Sum of IncomingBytes per group over the window, 500 groups per call."""
    out = {}
    for i in range(0, len(names), 500):
        chunk = names[i:i + 500]
        queries = [{
            "Id": f"q{j}",
            "MetricStat": {
                "Metric": {"Namespace": "AWS/Logs", "MetricName": "IncomingBytes",
                           "Dimensions": [{"Name": "LogGroupName", "Value": n}]},
                "Period": days * 86400, "Stat": "Sum"},
        } for j, n in enumerate(chunk)]
        resp = cw.get_metric_data(MetricDataQueries=queries,
                                  StartTime=NOW - timedelta(days=days), EndTime=NOW)
        for r in resp["MetricDataResults"]:
            out[chunk[int(r["Id"][1:])]] = sum(r["Values"])
    return out

def source_of(name):
    rules = [("/aws/lambda/", "lambda"), ("/aws/eks/", "eks_control_plane"),
             ("/aws/containerinsights/", "container_insights"), ("/aws/rds/", "rds"),
             ("/aws/vpc/", "vended_flow_logs"), ("API-Gateway-Execution-Logs", "api_gateway"),
             ("/aws/codebuild/", "codebuild"), ("/aws/cloudtrail", "audit"), ("/ecs/", "ecs")]
    return next((label for prefix, label in rules if name.startswith(prefix)), "application")

def evidence(g, ingest):
    name = g["logGroupName"]
    cls = g.get("logGroupClass", "STANDARD")
    gb_in = ingest.get(name, 0.0) / 1e9
    stored_gb = g.get("storedBytes", 0) / 1e9
    arn = g.get("logGroupArn") or g["arn"].removesuffix(":*")
    subs = logs.describe_subscription_filters(logGroupName=name)["subscriptionFilters"]
    return {
        "name": name, "source": source_of(name), "class": cls,
        "retention_days": g.get("retentionInDays"),      # None means never expire
        "created": datetime.fromtimestamp(g["creationTime"] / 1000, timezone.utc).date().isoformat(),
        "stored_gb": round(stored_gb, 2),
        "ingest_gb_30d": round(gb_in, 2),
        "ingest_usd_30d": round(gb_in * INGEST_PRICE[cls], 2),
        "storage_usd_month": round(stored_gb * STORAGE_PRICE, 2),
        "subscriptions": [s["destinationArn"] for s in subs],
        "metric_filters": len(logs.describe_metric_filters(logGroupName=name)["metricFilters"]),
        "tags": logs.list_tags_for_resource(resourceArn=arn).get("tags", {}),
    }

Only groups above an ingestion threshold get the expensive check. A one-hour sample of the last day tells you whether the stream is mostly DEBUG, and one hour bounds the scan cost to whatever that group ingests per hour, which you already know:

import time

def debug_share(name, hours=1):
    q = logs.start_query(
        logGroupName=name,
        startTime=int((NOW - timedelta(hours=hours)).timestamp()), endTime=int(NOW.timestamp()),
        queryString='stats count(*) as total, '
                    'sum(strcontains(@message, "DEBUG")) as debug, '
                    'sum(strcontains(@message, "TRACE")) as trace')
    while (r := logs.get_query_results(queryId=q["queryId"]))["status"] in ("Scheduled", "Running"):
        time.sleep(1)
    row = {f["field"]: float(f["value"]) for f in r["results"][0]} if r["results"] else {}
    total = row.get("total", 0) or 1
    return round((row.get("debug", 0) + row.get("trace", 0)) / total, 3)

For Lambda groups the collector also pulls LoggingConfig from GetFunctionConfiguration. A function with no ApplicationLogLevel set and a 60% debug share is the clearest possible fix in the whole account.

Step 3: The LLM classifies and proposes, inside a schema

The model sees a list of evidence records, never the account. Its output is constrained to a tool call whose retention_days enum is the exact list CloudWatch accepts, so a hallucinated "45 days" is a schema error, not a failed apply.

{
  "name": "propose_log_group_actions",
  "description": "Propose one action per log group from the evidence records provided.",
  "input_schema": {
    "type": "object",
    "properties": {
      "proposals": {
        "type": "array",
        "items": {
          "type": "object",
          "properties": {
            "log_group": {"type": "string"},
            "classification": {"enum": ["lambda_default", "debug_flood", "vended_flow_logs",
                                         "container_insights", "eks_control_plane", "audit",
                                         "application", "unknown"]},
            "action": {"enum": ["set_retention", "reduce_log_level",
                                 "recreate_as_infrequent_access", "route_to_s3", "no_change"]},
            "retention_days": {"enum": [1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365,
                                         400, 545, 731, 1096, 1827, 2192, 2557, 2922, 3288, 3653, null]},
            "monthly_saving_usd": {"type": "number"},
            "saving_requires": {"enum": ["nothing", "retention_ageout", "code_or_config_change"]},
            "rationale": {"type": "string"},
            "confidence": {"enum": ["high", "medium", "low"]}
          },
          "required": ["log_group", "classification", "action", "monthly_saving_usd",
                       "saving_requires", "rationale", "confidence"]
        }
      }
    },
    "required": ["proposals"]
  }
}

The system prompt carries the rules that the schema can't:

You are a FinOps reviewer for CloudWatch Logs. You receive evidence records and
propose actions. Rules:
1. Never propose retention below the "retention:min" tag if present, and never
   below 365 for classification "audit" or any group whose name contains
   "cloudtrail", "audit", or "security". Set action=no_change and say why.
2. Ingestion savings (reduce_log_level, route_to_s3, recreate_as_infrequent_access)
   require a change by the owning team. Set saving_requires=code_or_config_change
   and never count them as realised.
3. Retention savings are storage only: stored_gb * 0.03 per month, realised
   after the old data ages out. Do not add ingestion cost to that number.
4. recreate_as_infrequent_access is only valid when metric_filters == 0 and
   subscriptions is empty; the IA class supports neither.
5. A group with a subscription to a SIEM or S3 destination already has a copy
   elsewhere; 30 days retention is the default proposal there.
6. If tags carry no owner and the name gives no clue, classification=unknown,
   action=no_change, confidence=low. Do not guess an owner.
Explain every proposal in one sentence a service owner would accept.

Rule 4 is the kind of thing that costs a day when a human forgets it: an IA group silently has no metric filters, so the alarm built on one disappears with the recreate. Rule 2 keeps the agent honest about the number it will be judged on. The prompt is versioned and diffed like the ones in the Terraform plan review agent, and a fixture of twenty hand-labelled log groups runs against every prompt change so a "small wording tweak" that starts proposing 7-day retention on audit logs fails in CI.

Step 4: Retention ships as a PR, log levels ship as a diff to the owner

Retention proposals become a Terraform commit. Groups that Terraform already manages get their attribute changed. Groups Lambda created on its own get an import block so they come under management in the same PR, which also ends the never-expire default for good:

import {
  to = aws_cloudwatch_log_group.lambda_thumbnailer
  id = "/aws/lambda/thumbnailer"
}

resource "aws_cloudwatch_log_group" "lambda_thumbnailer" {
  name              = "/aws/lambda/thumbnailer"
  retention_in_days = 30
}

The plan-stage gate refuses any bot PR that does anything other than update log groups:

terraform show -json plan.bin | jq -e '
  [.resource_changes[]
   | select(.change.actions != ["no-op"])
   | select(.type != "aws_cloudwatch_log_group" or .change.actions != ["update"])]
  | length == 0' || { echo "bot PR may only update existing log groups"; exit 1; }

Log-level changes never go through that lane. They land as a separate PR against the service's own module, opened in the owner's repo with the evidence in the description, because a debug flood is the team's bug to fix. For Lambda the diff is small and the payoff is immediate:

resource "aws_lambda_function" "thumbnailer" {
  # ...existing config...
  logging_config {
    log_format            = "JSON"
    application_log_level = "INFO"
    system_log_level      = "WARN"
    log_group             = aws_cloudwatch_log_group.lambda_thumbnailer.name
  }
}

With application_log_level = "INFO" the Lambda runtime drops DEBUG lines before they are ingested, which is the only place a $0.50/GB cost can actually be avoided. Approval follows reversibility, as in the human-in-the-loop gates post: a retention increase or a 30-day default needs one reviewer, anything under 14 days or any change to a group tagged retention:min needs the owner, and audit groups are excluded by rule 1 before the PR exists.

Step 5: Stop the never-expire default from coming back

The agent's second month should find nothing new, and that only happens if creation is fixed too. An EventBridge rule on the CloudTrail CreateLogGroup event triggers a tiny function that sets a 30-day retention on anything born without one:

{
  "source": ["aws.logs"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["logs.amazonaws.com"],
    "eventName": ["CreateLogGroup"]
  }
}

import boto3
logs = boto3.client("logs")

def handler(event, _):
    name = event["detail"]["requestParameters"]["logGroupName"]
    groups = logs.describe_log_groups(logGroupNamePrefix=name)["logGroups"]
    g = next((x for x in groups if x["logGroupName"] == name), None)
    if g and g.get("retentionInDays") is None:
        logs.put_retention_policy(logGroupName=name, retentionInDays=30)

Note the exact-name match: logGroupNamePrefix is a prefix, and /aws/lambda/api also matches /aws/lambda/api-canary. This function is the one place in the design with PutRetentionPolicy, it runs on a role the LLM process cannot assume, and Terraform-managed groups that set their own retention later simply overwrite it. Pair it with the AWS Config managed rule cw-loggroup-retention-period-check so an audit surface exists that isn't your agent's own report.

What this does not solve

• Retention is the small lever. On a typical account the first run cuts storage cost by a lot and the total CloudWatch bill by a little. The ingestion proposals are where the money is, and they depend on teams merging a log-level change. Report the two numbers separately or the agent will look like it under-delivered.
• Log class is set at creation. Moving a chatty group to Infrequent Access means a new group, a repointed producer, and losing metric filters, subscription filters, Live Tail and S3 export on that group. The agent proposes it only when rule 4 holds, and a human still has to weigh it.
• Vended logs have a cheaper home. Flow logs, Route 53 resolver logs and similar are usually better delivered straight to S3 at roughly half the price and queried with Athena, which is the same argument the network cost agent makes about its own telemetry. route_to_s3 is a proposal, not something the agent can do.
• The sampler has a bill. A one-hour Logs Insights window per hot group is cheap, but running it across every group every day is not. Gate it on ingestion, run it weekly, and watch DataScanned-Bytes in the cost breakdown so the agent doesn't become its own top finding.
• Cost Explorer lags a day and the metric lags minutes. The dollar figures the agent reports come from IncomingBytes and list prices, so they can disagree with the invoice by a few percent. That's fine for ranking, and wrong for a finance report.

Run it monthly, ship the retention PR the first week, and put the top ten ingestion sources in front of their owners with the debug share next to each. In most accounts three log groups are half the bill, and at least one of them is DEBUG output nobody has read since the incident it was turned on for.

---

📌 Read the latest version of this guide — plus the full library of DevOps, SRE, Kubernetes, observability & cloud-cost guides — on devtocash.com.