How do I troubleshoot cross-account logging issues in CloudWatch?

Short description

You can send CloudWatch logs within your account or to another AWS account in near real-time. Sharing your logs with another AWS account or receiving their logs is called cross-account logging. You can set up cross-account logging using Amazon Kinesis or Amazon Kinesis Data Firehose stream. Or you can stream the data to Amazon OpenSearch using a subscription filter. The CloudWatch console supports all these options. For more information on using cross-account logging, see these articles:

• Real-time processing of log data with subscriptions
• How do I create, configure, and troubleshoot a subscription filter to Kinesis using the CloudWatch console?
• Streaming CloudWatch Logs data to Amazon OpenSearch Service

Resolution

Troubleshoot Kinesis Data Stream/Kinesis Data Firehose delivery stream as target

1.    Make sure that your Kinesis stream or Kinesis Data Firehose is in an active state. You can view the status on the Kinesis console, or you can use the DescribeStream or DescribeDeliveryStream API calls.

2.    Check that the CloudWatch log group Region and the Kinesis data stream Region match.

3.    Check that you have created an AWS Identity and Access Management (IAM) role with the correct trust permissions in place. The IAM role must allow your CloudWatch logs to send log events to Kinesis data stream or Kinesis Data Firehose.

In this example, the log data recipient account has an AWS account ID of 999999999999. The log data sender AWS account ID is 111111111111. Make sure that Account 111111111111 is allowed in the recipient account trust policy for Kinesis Data Streams or Kinesis Data Firehose stream. This is an example IAM trust policy for Kinesis Data Streams or Kinesis Data Firehose delivery stream:

{
  "Statement": {
    "Effect": "Allow",
    "Principal": {
      "Service": "logs.region.amazonaws.com"
    },
    "Condition": {
      "StringLike": {
        "aws:SourceArn": [
          "arn:aws:logs:region:sourceAccountId:*",
          "arn:aws:logs:region:recipientAccountId:*"
        ]
      }
    },
    "Action": "sts:AssumeRole"
  }
}

This example shows an IAM role policy for Kinesis Data Streams:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "kinesis:PutRecord",
      "Resource": "arn:aws:kinesis:region:999999999999:stream/RecipientStream"
    }
  ]
}
 
IAM role policy for kinesis Firehose stream.
 
{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:*"
      ],
      "Resource": [
        "arn:aws:firehose:region:999999999999:*"
      ]
    }
  ]
}

4.    Verify that the Regions and resource ARNs in your IAM policy are correct.

5.    Make sure that you don't select Kinesis Firehose when you are configuring the subscription filter for Kinesis Data Streams.

6.    After you start streaming, check the metrics for the subscription filter. Confirm that the filter pattern is valid and matches the incoming log events. For more information, see Monitoring with CloudWatch metrics. Use these metrics:

• ForwardedBytes - The volume of log events in compressed bytes forwarded to the subscription destination.
• ForwardedLogEvents - The number of log events forwarded to the subscription destinations.

7.    To confirm that there are no errors when streaming log events to the destination, check these metrics for destination errors. You might find a datapoint that confirms that the log event received an error when streaming log events to the destination.

• DeliveryErrors - The number of log events that CloudWatch logs received an error for when forwarding data to the subscription destination.
• DeliveryThrottling - The number of log events that CloudWatch logs throttled when forwarding data to the subscription destination.

8.    If you have a dedicated stream that gives insights for your Kinesis Data Streams functionality, then check the metrics for your Kinesis Data Stream or Kinesis Firehose. For more information, see Monitoring the Amazon Kinesis Data Streams Service with Amazon CloudWatch and Monitoring Kinesis Data Firehose using CloudWatch Metrics.

Troubleshoot OpenSearch as destination target

1.    Check that the OpenSearch domain allows either public access or VPC access. For more information on creating a domain, see Creating and managing Amazon OpenSearch Service domains.

2.    Be sure that the AWS Lambda function has an IAM execution role that has trust relation for lambda.amazonaws.com with this role policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "es:*"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:es:region:account-id:domain/target-domain-name/*"
    }
  ]
}

3.    If the target OpenSearch domain uses VPC access, check that the role has the AWSLambdaVPCAccessExecutionRole policy attached.

---

Related information

Create cross account & cross Region CloudWatch dashboards