How do I troubleshoot CloudWatch Logs so that it streams to my OpenSearch Service domain?

Resolution

I can't stream multiple CloudWatch log groups to the same OpenSearch Service domain

By default, Amazon CloudWatch creates only one AWS Lambda function for each OpenSearch Service domain. If you set up multiple log groups to index data into your domain, then all the multiple log groups invoke the same Lambda function. When the first log group invokes a Lambda function, the invocation creates an index and a type field in your domain.

Indices that are created in OpenSearch Service 6.0.0 or later can contain only a single mapping type. Indices that are created in 5.x with multiple mapping types continue to function in OpenSearch Service 6.x. For more information about OpenSearch Service mapping types deprecation, see Removal of mapping types on the Elastic website.

When other log groups try to invoke the same Lambda function, the invocation fails with the following error:

"reason": "Rejecting mapping update to [<index_name>] as the final mapping would have more than 1 type: [log-group-1, log-group-2]”

To resolve this issue, first update your Lambda function with the following syntax:

var indexName = [
     'cwl-' + payload.logGroup.toLowerCase().split('/').join('-') + '-' + timestamp.getUTCFullYear(),
     ('0' + (timestamp.getUTCMonth() + 1)).slice(-2),
     ('0' + timestamp.getUTCDate()).slice(-2) 
     ].join('.');

This syntax creates multiple indices for the different log groups that are streaming into your OpenSearch Service domain.

Save the updated Lambda function to create separate indices for the multiple log groups that are streaming into your domain.

I can't stream to a VPC-based OpenSearch Service domain in the same AWS account

Important: Before streaming the CloudWatch log groups to your virtual private cloud (VPC)-based OpenSearch Service domain, update your AWS Identity and Access Management (IAM) role policy. The IAM role that's attached to the corresponding Lambda function must have the AWSLambdaVPCAccessExecutionRole policy attached to it.

Here's an AWSLambdaVPCAccessExecutionRole policy in JSON format:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "ec2:CreateNetworkInterface",
        "ec2:DescribeNetworkInterfaces",
        "ec2:DeleteNetworkInterface"
      ],
      "Resource": "*"
    }
  ]
}

Note: This managed policy allows the Lambda function to write the CloudWatch log group to your cluster in the VPC.

After you attach the policy to your Lambda function, you can begin streaming the logs to your OpenSearch Service domain in the VPC.

I can't stream my CloudWatch log group to an OpenSearch Service domain when fine-grained access control is activated

If you stream your CloudWatch logs to an OpenSearch Service domain with fine-grained access control, then you might receive an error similar to the following one:

"{\"statusCode\":403,\"responseBody\":{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::123456789101:role/lambda_opensearch_execution, roles=[arn:aws:iam::123456789101:role/lambda_opensearch_execution], requestedTenant=null]\"}],\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::123456789101:role/lambda_opensearch_execution, roles=[arn:aws:iam::123456789101:role/lambda_opensearch_execution], requestedTenant=null]\"},\"status\":403}}"

If you received this error message from your Lambda function logs, then the role mapping is incomplete.

Note: By default, OpenSearch Service creates an Lambda function for you.

OpenSearch Service domains running 7.9 and later (including OpenSearch version 1.x)

To resolve the error message, complete the following steps:

1.    Open OpenSearch Dashboards. You can find a link to OpenSearch Dashboards in the domain summary of your OpenSearch Service console.

2.    From the navigation pane, choose Security.

3.    Choose Roles.

4.    Choose the all_access role.

5.    Choose the Mapped users tab.

6.    On the Mapped users dialog page, choose Manage mapping.

7.    Under Backend roles, enter the Lambda function execution role ARN.

8.    Choose Map. Your logs now stream to your OpenSearch Service domain.

For more information about role mapping, see Mapping roles to users.

OpenSearch Service domains running 7.8 and earlier

To resolve the error message, complete the following steps:

1.    Open OpenSearch Dashboards. You can find a link to OpenSearch Dashboards in the domain summary of your OpenSearch Service console.

2.    On the navigation pane, choose the lock icon.

3.    Choose Role mappings.

4.    Choose all_access and security_manager as your roles.

Note: The all_access role provides access only to your cluster. Based on your use case, you can also add fine-grained access control to your cluster.

5.    Edit the mapping for all_access.

6.    For Backend Role, add the Lambda function's execution role and choose Submit. Your logs now stream to your OpenSearch Service domain.

My CloudWatch logs aren't being delivered to my OpenSearch Service domain

When you use the default Lambda function to stream CloudWatch logs to your OpenSearch Service domain, you might get the following error:

"errorMessage": "{\"statusCode\":200,\"responseBody\":{\"took\":42,\"errors\":true}}","

Note: By default, Lambda errors are returned as 200 OK responses.

To troubleshoot this error message, complete the following steps:

1.    Open the default Lambda function.

2.    Find the following line of code:

"var logFailedResponses = false;"

3.    Update the var logFailedResponses value to true. This update provides additional information for any new indexing requests that use the Lambda function. You can use the additional information to identify the cause of your indexing issue.

I'm getting a cluster_block_exception error

A lack of free storage space or excessive JVM memory pressure causes cluster block exceptions.

For more information, see How do I troubleshoot high JVM memory pressure on my Amazon OpenSearch Service cluster?

My CloudWatch subscription filter fails to send data to my cluster through a default Lambda function (OpenSearch Service 2.0 and later)

If you have a CloudWatch subscription filter to send logs to OpenSearch Service 2.x using the default Lambda function, then you might get an error. If the subscription filter fails to ingest the logs and you get the following error, then a deactivated parameter caused the error:

"{\"statusCode\":400,\"responseBody\":{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}}"

In OpenSearch Service versions 2.0 and later, the _type parameter is removed from API endpoints. To resolve this error, you must also remove the parameter from your Lambda function's code.

1.    Open the AWS Lambda console.

2.    Select the default Lambda function for your subscription filter.

3.    See the code for your function under Code source.

4.    Find the transform function that's defined in the code. Within this function, the data is converted into the JSON indexing format for OpenSearch Service. The first lines of this code are similar to the following code:

function transform(payload) {
    if (payload.messageType === 'CONTROL_MESSAGE') {
        return null;
    }

    var bulkRequestBody = '';

    payload.logEvents.forEach(function(logEvent) {
                var timestamp = new Date(1 * logEvent.timestamp);

5.    Under the transform function, find the _type parameter. In most cases, this is in line 79. Remove or comment out the line of code that adds the _type parameter. After removal, your code is similar to the following code:

var action = {
    "index": {}
};
action.index._index = indexName;
//action.index._type = payload.logGroup;
action.index._id = logEvent.id;

bulkRequestBody += [

You can now successfully send indexing requests.