Which log group is causing a sudden increase in my CloudWatch Logs bill?

4 minute read
2

My Amazon CloudWatch Logs bill is unusually high, and I want to determine which log group is increasing my CloudWatch Logs costs.

Short description

Sudden increases in the CloudWatch Logs billed amount often result from an increase in ingested or storage data in a particular log group. Use CloudWatch Logs metrics to check data usage, and then review your AWS bill to identify the log group that's responsible for the increase in your billed amount.

Resolution

Check the amount of data that you're ingesting

The IncomingBytes metric shows you the amount of ingested data in your CloudWatch log groups in near real time. This metric can help you to determine the following points:

  • Which log group is the highest contributor towards your bill
  • Whether there's a spike in the incoming data to your log groups or a gradual increase due to new applications
  • How much data is pushed during a particular period of time

Query a small set of log groups using the CloudWatch console

  1. Open the Amazon CloudWatch console.
  2. In the navigation pane, choose All metrics.
  3. For each of your log groups, complete the following steps:
    Choose the log's namespace.
    Choose Logs, Log group metrics.
    Select the individual IncomingBytes metrics.
    Choose the Graphed metrics tab.
  4. For Statistic, choose Sum.
  5. For Period, choose 30 Days.
  6. Choose the Graph options tab, and then choose Number.
  7. At the top right of the graph, choose Custom, and then choose Absolute to set the time range. Select a start and end date that corresponds with the last 30 days.
    Note: This step works only with the Absolute option. If you choose Relative, then the graph returns incorrect data.

Query a large set of log groups using the CloudWatch console

  1. Open the Amazon CloudWatch console.
  2. In the navigation pane, choose All metrics.
  3. Choose the Graphed metrics tab. Then, from the Add math dropdown list, choose Start with an empty expression.
  4. Copy the following math expression, and then paste it into the Edit math expression field:
SORT(REMOVE_EMPTY(SEARCH('{AWS/Logs,LogGroupName} MetricName="IncomingBytes"', 'Sum', 2592000)),SUM, DESC)

After you paste the expression, choose Apply.

  1. Choose the Graph options tab, and then choose Number.

  2. At the top right of the graph, choose Custom. Then, choose Absolute. Select a start and end date that corresponds with the last 30 days. You can graph up to 500 metrics using this method.
    Note: This step only works with the Absolute option. If you choose Relative, then the graph returns incorrect data.

Query a large set of log groups using an API call

Note: Before you run the following API calls, review the associated costs with making API calls. It's a best practice to distribute the ListMetrics call to avoid throttling. The default limit for ListMetrics is 25 transactions per second. However, you can request a limit increase, if necessary.

  1. Make a ListMetrics call. This call finds all log group names that ingested data in the past 14 days. Use the following parameters:
    Namespace: AWS/Logs
    MetricName: IncomingBytes
  2. Make a GetMetricData call. This call finds the sum of all incoming bytes in a month for every log group name that you get from the ListMetrics call. Use the following parameters:
    Namespace: AWS/Logs
    MetricName: IncomingBytes
    Dimensions: As received from the ListMetrics call
    StartTime: [Date and time 14 days before the current date]
    EndTime: [Current date and time]
    Period: [EndTime - StartTime, in seconds]
    Statistics: Sum
  3. To display the log group names with the highest Ingested data amounts, sort the resulting data points in descending order.

To be sure that ingested data charges don't exceed a specified limit in the future, create a CloudWatch alarm.

Review your storage data usage

Check your most recent AWS bill to see how much storage data you used in the previous billing cycle.

Related information

CloudWatch billing and cost

AWS OFFICIAL
AWS OFFICIALUpdated 9 months ago