How do I troubleshoot CloudWatch logs that fail to export to S3 buckets?

Resolution

Troubleshoot your logs that fail to export to Amazon S3 buckets based on whether they fail during or after you create the export task.

During export task creation

To troubleshoot tasks that fail when you're creating them, check the following:

• AWS Region: Confirm that your CloudWatch log streams and Amazon S3 buckets are in the same Region.
• Amazon S3 bucket policies: By default, Amazon S3 buckets and objects are private. Only the resource owner has access to the bucket and objects within the bucket. For more information, see Step 3: Set permissions on an Amazon S3 bucket.
  Note: When you set export permissions on your Amazon S3 bucket, you must specify the account IDs that can export logs to your bucket. List these accounts under the aws:SourceAccount key. Also, you can't add aws:SourceAccount in the s3:GetBucketAcl action.
• Amazon S3 bucket prefixes: When you set the Amazon S3 bucket policy, it's a best practice to include a randomly generated string as the prefix for the bucket. If you use a prefix, then you must specify the generated string in the Amazon S3 bucket prefix settings when you create the export task. If you don't specify the generated string, then the export task creation fails.
• AWS Identity and Access Management (IAM) policies: Confirm that the IAM user or role that created the export task has full access to Amazon S3 and CloudWatch Logs. For more information, see Step 2: Set up access permissions.
• Check the CloudTrail logs: Search for the CreateExportTask event in AWS CloudTrail to identify the error message. CreateExportTask is an asynchronous operation. If you provide all the necessary information, then an export task is initiated and returns the task's ID. When the task begins, use DescribeExportTasks to check the status. For more information, see Errors.
• Resource quotas: The CloudWatch Logs service quota allows only one running or pending export task per AWS account per Region. You can't change this quota. Make sure that you're within the allowed quota.
• Type of server side encryption: Make sure that you use a supported server-side encryption for your Amazon S3 buckets. Supported encryptions include Advanced Encryption Standard (AES)-256 and server-side encryption with AWS Key Management Service (AWS KMS) keys, known as SSE-KMS.

After export task creation

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

To troubleshoot tasks that fail after you create them, check the time range setting. If you export log streams with large amounts of data and specify a long time range, then the export task might fail. To resolve this issue, split the task into multiple shorter time ranges.

To check the status of the time range setting, run the following command:

Note: Replace example-task-id with your task ID.

-- aws logs describe-export-tasks --task-id example-task-id

To use the AWS CLI to export data to Amazon S3, run the following command:

Note: Replace example-task-name with your task name, example-log-group-name with your log group name, and example-bucket-name with your destination bucket.

aws logs create-export-task --task-name "example-task-name" --log-group-name "/aws/lambda/example-log-group-name" --destination "example-bucket-name --from 1710378735 --to 1710465135

To check the status of the export task, run the following command:

Note: Replace example-task-id with your task ID.

-- aws logs describe-export-tasks --task-id example-task-id

Note: It might take up to 12 hours for the logs to be available for export. The export task can also take up to a few hours, depending on the size of the data that you're exporting. It's a best practice to use log group-level subscription filters to process in real time or to continuously archive new data to Amazon S3. For more information, see Real-time processing of log data with subscriptions.

Related information

I configured Amazon CloudWatch to export log data to Amazon S3, but the log data is either missing or invalid. How do I resolve this issue?

How do I retrieve log data from CloudWatch Logs?

Exporting log data to Amazon S3