Why is my Kinesis Data Streams trigger not able to invoke my Lambda function?

Short description

The following are common causes of Lambda function errors:

• Insufficient permissions in the Lambda function's execution role
• No incoming data to the Kinesis data stream
• Inactive event source mapping caused by the recreation of a Kinesis data stream, Lambda function, or Lambda execution role
• A Lambda function that exceeds the maximum execution time and causes a timeout error in the Lambda function
• Lambda breaches its limits of concurrent executions

If there's a Lambda function error, then your function isn't invoked. The function also doesn't process records from the batch. An error can cause Lambda to retry the batch of records until the process succeeds or the batch expires. For more information about Lambda function and Kinesis errors, see Using AWS Lambda with Amazon Kinesis.

Resolution

Troubleshoot your Lambda function

To identify why your Lambda function isn't invoked, complete the following steps:

1. Check the Invocations metric in Amazon CloudWatch with statistics set as Sum for the Lambda function. The Invocations metric can help you verify whether the Lambda function is invoked.

2. Check the IteratorAge metric to see how old the last record in the batch was or when the process completed. When your Lambda consumer isn't able to invoke, the iterator age of your stream increases.

3. Check the Lambda function's CloudWatch logs. The logs use the .../aws/lambda/function_name format for their names. Look for corresponding entries against the function error. For example, if the error occurs because of Lambda's AWS Identity and Access Management (IAM) role permissions, then modify the IAM role policy.

4. Confirm that your IAM role has the proper permissions to access CloudWatch:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents"
         ],
         "Resource": "*"
       }
     ]
   }

5. (Optional) If you experience a permissions error, then update your Lambda function policy and grant it access to Kinesis Data Streams:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "kinesis:DescribeStream",
           "kinesis:DescribeStreamSummary",
           "kinesis:GetRecords",
           "kinesis:GetShardIterator",
           "kinesis:ListShards",
           "kinesis:ListStreams",
           "kinesis:SubscribeToShard",
           "logs:CreateLogGroup",
           "logs:CreateLogStream",
           "logs:PutLogEvents"
         ],
         "Resource": "*"
       }
     ]
   }

   Note: The AWSLambdaKinesisExecutionRole policy includes these permissions.

Additional troubleshooting

To perform additional troubleshooting, complete the following tasks:

Lambda execution function timeout

If the error is related to a Lambda execution function timeout, then increase the timeout value to accommodate faster processing.

AWS KMS encrypted data stream

If you use AWS Key Management Service (AWS KMS) to encrypt your Kinesis data stream, then the consumer and producer must have proper access. The Kinesis data stream must be able to access the AWS KMS keys that are used for encryption and decryption. The Lambda function's execution role must also have read access to the KMS key to successfully read data from the Kinesis data stream.

Internal Lambda function error

If the error is caused by an internal Lambda function error, then there is an issue with stream processing. To avoid control plane API throttling, restrict each stream to 4 to 5 event source mappings. These restrictions help avoid too many event source mappings with the same data stream. Multiple event source mappings with the same stream can result in a breach of Kinesis and Amazon DynamoDB control plane limits.

Connection timeout error

If you get a connection timeout error, add logging statements before and after the API calls made in your code. Then, identify the exact line of code where the function begins to fail.

Slow or stalled shards

If you have slow or stalled shards, then configure the event source mapping to retry with a smaller batch size. You can also limit the number of retries or discard older records.

"Memory used" error

If you see a "memory used" error message in your CloudWatch logs, then increase the memory of your Lambda function.

Maximum timeout exceeded

If you exceeded the maximum timeout for your Lambda function, then modify the client library and client timeouts. To modify the timeout session based on the remaining time in the Lambda container, use the context.GetRemainingTimeInMillis function. The context.GetRemainingTimeInMillis function returns the amount of time left in the Lambda container before it times out.

Lambda function code error

If you receive errors from the Lambda function's code, then your Lambda function might get stuck trying the same record. Use a try-catch block to catch the failed data. Then, use an Amazon Simple Queue Service (Amazon SQS) queue or Amazon Simple Notification Service (Amazon SNS) topic to record it. You can also add a Lambda trigger to the Amazon SQS queue with the processing logic to separately retry the failed requests.

SQS DLQ

Set up an Amazon SQS dead-letter queue (DLQ) to manually invoke the Lambda function. Configure the DeadLetterConfig property when you create or update your Lambda function. You can provide an Amazon SQS queue or an Amazon SNS topic as the TargetArn for your DLQ. Lambda then writes the event object and invokes the Lambda function to the specified endpoint after the standard retry policy is exhausted.

Lambda with X-Ray

Use Lambda with AWS X-Ray to detect, analyze, and optimize performance issues with Lambda applications. X-Ray collects metadata from the Lambda service and generates graphs that depict issues that affect the performance of the Lambda application. For example, if there's a call that takes a long time to run, then you can use AWS X-Ray to confirm the issue.