How do I troubleshoot Lambda function failures?

7 minute read
0

When I try to invoke my AWS Lambda function, it fails and returns an error.

Resolution

To troubleshoot Lambda function failures, determine what's causing the error by using one or more of the AWS services and features listed in this article. Then, follow the links provided to review the troubleshooting best practices for each issue.

Identify and troubleshoot any networking errors

If there are issues with your Lambda networking configuration, you see many types of errors. The following are some of the most common Lambda networking-related errors:

If your function isn't in a virtual private cloud (VPC) and you tried to access resources using a private DNS name, then you see following error:

UnknownHostException
Error: getaddrinfo ENOTFOUND

If your function is in a VPC and then loses internet access or times out, you see the following error:

connect ETIMEDOUT 176.32.98.189:443
Task timed out after 10.00 seconds

If the VPC that your function is in reaches its elastic network interface limit, you see the following error:

ENILimitReachedException: The elastic network interface limit was reached for the function's VPC.

If the Transmission Control Protocol (TCP) connection is dropped, you see the following error:

Connection reset by peer
ECONNRESET
ECONNREFUSED

To troubleshoot Lambda networking errors

1.    Confirm that there's a valid network path to the endpoint that your function is trying to reach for your Amazon Virtual Private Cloud (Amazon VPC). For more information, see Configuring a Lambda Function to Access Resources in a VPC.

2.    Confirm that your function has access to the internet. For more information, see How do I give internet access to a function that's connected to an Amazon VPC? Also, see How do I troubleshoot timeout issues with a Lambda function that's in an Amazon VPC?

3.    To troubleshoot DNS resolution related issues, make sure that the VPC is configured for private resource access. If you're not using AWS provided DNS use an EC2 instance to make sure that the custom provided DHCP option resolves DNS name correctly. For more information, see How does DNS work and how do I troubleshoot partial or intermittent DNS failures?

Note: If you can't determine why your function code isn't reaching a public endpoint after reviewing your VPC configuration, turn on VPC Flow Logs. VPC Flow Logs allow you to see all the network traffic flowing to and from a VPC. VPC Flow Logs also allows you to determine why a specific request was denied or didn't route. For more information, see Troubleshoot networking issues in Lambda.

Identify and troubleshoot any permission errors

If the security permissions for your Lambda deployment package are incorrect, you see one of the following errors:

  • EACCES: permission denied, open '/var/task/index.js'
  • cannot load such file -- function
  • [Errno 13] Permission denied: '/var/task/function.py'

The Lambda runtime needs permission to read the files in your deployment package. You can use the chmod command to change the file mode. The following example commands make all files and folders in the current directory readable by any user:

chmod -R o+rX .

For more information, see Troubleshoot deployment issues in Lambda.

If your AWS Identity and Access Management (IAM) identities don't have permission to invoke a function, then you receive the following error:

User: arn:aws:iam::123456789012:user/developer is not authorized to perform: lambda:InvokeFunction on resource: my-function

To troubleshoot Lambda permissions errors

Review your Lambda log file entries in AWS CloudTrail. The requester making calls to Lambda must have the IAM permissions required to invoke your function. To grant the required permissions, update your Lambda function permissions.

For more information, see the following topics:

Identify and troubleshoot any code errors

If there are issues with your Lambda code, you see many types of errors. The following are some of the more common Lambda code-related errors:

  • Unable to marshal response: Object of type AttributeError is not JSON serializable
  • Issue: The AWS SDK included on the runtime is not the latest version
  • (Node.js) Function returns before code finishes executing
  • KeyError

To troubleshoot Lambda code errors

1.    Review your Amazon CloudWatch Logs for Lambda.

You can use CloudWatch to view all logs generated by your function's code and identify potential issues. For more information, see Accessing Amazon CloudWatch Logs for AWS Lambda. For details on function logging, see the following Lambda function logging instructions for the programming language that you're using:

Note: If your function is returning a stack trace, then the error message in the stack trace specifies what's causing the error.

2.    Use AWS X-Ray to identify any code performance bottlenecks. If your Lambda function uses downstream AWS resources, microservices, databases, or HTTP web APIs, then you can use AWS X-Ray to troubleshoot code performance issues. For more information, see Using AWS Lambda with AWS X-Ray.

3.    Confirm that your function's deployment package can import any required dependencies. Follow the Lambda deployment packages instructions for the programming language that you're using:

Note: You can also use Lambda layers to add dependencies that are outside of your deployment package.

4.    (For code deployed as a container image) Confirm that you're installing the runtime interface client and deploying the image correctly. Follow the container image instructions for the programming language that you're using:

Identify and troubleshoot any throttling errors

If your function gets throttled, you see the following error:

Rate exceeded
429 TooManyRequestsException

To troubleshoot Lambda throttling errors

Review your CloudWatch metrics for Lambda. For more information, see Working with Lambda function metrics.

Key metrics to monitor:

  • ConcurrentExecutions
  • UnreservedConcurrentExecutions
  • Throttles

Note: If requests to invoke your function arrive faster than the function can scale or exceed your concurrency limit, then requests fail with a 429 throttling error. For more information, see Lambda function scaling. Also, How do I troubleshoot Lambda function throttling with "Rate exceeded" and 429 "TooManyRequestsException" errors?

Identify and troubleshoot any Invoke API 500 and 502 errors

If your invoke request fails, then you see any of the following 502 or 500 server-side errors:

  • InvalidRuntimeException
  • InvalidSecurityGroupIDException
  • InvalidZipFileException
  • KMSAccessDeniedException
  • KMSNotFoundException
  • You have exceeded the maximum limit for Hyperplane ENIs for your account
  • SubnetIPAddressLimitReachedException

To troubleshoot Lambda Invoke API 500 and 502 errors

Follow the instructions in How do I troubleshoot HTTP 502 and HTTP 500 status code (server-side) errors from AWS Lambda? For a list of possible errors with descriptions, see Errors in the Lambda invoke API reference.

Identify and troubleshoot any container image errors

If you're using container images and there's an issue with a container image, you see any of the following errors:

  • "errorType": "Runtime.InvalidEntrypoint"
  • Error: You are using an AWS CloudFormation template, and your container ENTRYPOINT is being overridden with a null or empty value.

To troubleshoot Lambda container image errors

Follow the instructions in Troubleshoot container image issues in Lambda.


Related information

Monitoring and troubleshooting Lambda applications

Error handling and automatic retries in Lambda

AWS OFFICIAL
AWS OFFICIALUpdated a year ago