How do I resolve the error "CannotPullContainerError: You have reached your pull rate limit" in Amazon ECS?

Short description

This error occurs when you try to pull an image from the public Docker Hub repository (on the Docker Hub website) after you reach your Docker pull rate limit (from the Docker Hub website). Exceeding your rate limit returns an HTTP status code of 429. Docker Hub uses IP addresses to authenticate the users, and pull rate limits are based on individual IP addresses. For anonymous users, the rate limit is set to 100 pulls per 6 hours per IP address. For authenticated users with a Docker ID, the pull rate is set to 200 pulls per 6-hour period. If your image pull request exceeds these limits, Amazon ECS denies these requests until the 6-hour window elapses. If you're running your Amazon ECS or Amazon Elastic Kubernetes Service (Amazon EKS) workload, then data is pulled through a NAT gateway with a fixed IP address. In this case, the request is throttled when you exceed the pull limit.

Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook to troubleshoot the errors for Amazon ECS tasks that fail to start. This automation reviews the following configurations:

• Network connectivity to the configured container registry
• Missing AWS Identity and Access Management (IAM) permissions that the execution role requires
• Virtual private cloud (VPC) endpoint connectivity
• Security group rule configuration
• AWS Secrets Manager secrets references
• Logging configuration

Resolution

Important:

• Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook in the same AWS Region where your ECS cluster resources are located.
• When using the runbook, you must use the most recently failed Task ID. If the failed task is part of Amazon ECS, then use the most recently failed task in the service. The failed task must be visible in ECS:DescribeTasks during the automation execution. By default, stopped ECS tasks are visible for 1 hour after entering the Stopped state. Using the most recently failed task ID prevents the task state cleanup from interrupting the analysis during the automation.

Note: If the runbook's output doesn't provide recommendations, then use one of the manual troubleshooting approaches in the following section.

To run the AWSSupport-TroubleshootECSTaskFailedToStart runbook:

1.    Open the AWS Systems Manager console.

2.    In the navigation pane, under Change Management, choose Automation.

3.    Choose Execute automation.

4.    Choose the Owned by Amazon tab.

5.    Under Automation document, search for TroubleshootECSTaskFailedToStart.

6.    Select the AWSSupport-TroubleshootECSTaskFailedToStart card.
Note: Make sure that you select the radio button on the card and not the hyperlinked automation name.

7.    Choose Next.
Note: After execution, analysis results are populated in the Global output section. However, wait for the status of the document to move to Success. Also, watch for any exceptions in the Output section.

8.    For Execute automation document, choose Simple execution.

9.    In the Input parameters section, for AutomationAssumeRole, enter the ARN of the role that allows Systems Manager Automation to perform actions.
Note: Be sure that either the AutomationAssumeRole or the IAM user or role has the required IAM permissions to run the AWSSupport-TroubleshootECSTaskFailedToStart runbook. If you don't specify an IAM role, then Systems Manager Automation uses the permissions of the IAM user or role that runs the runbook. For more information about creating the assume role for Systems Manager Automation, see Task 1: Create a service role for Automation.

10.    For ClusterName, enter the cluster name where the task failed to start.

11.    For TaskId, enter the identification for the task that most recently failed.

12.    Choose Execute.

Based on the output of the automation, use one of the following manual troubleshooting steps.

Copy public images into an Amazon ECR private registry

Create an Amazon Elastic Container Registry (Amazon ECR) repository, and then push the image into this new repository. When you pull the images from the Amazon ECR repository, you might avoid exceeding the Docker Hub pull limit.

1.    Run a command similar to the following to pull the image from Docker Hub:

docker pull example-image

2.    Run a command similar to the following to authenticate your Docker client to access the Amazon ECR registry:

aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 1111222233334444.dkr.ecr.eu-west-1.amazonaws.com

3.    Run a command similar to the following to tag the image to push to your repository:

docker tag myrepository:latest 1111222233334444.dkr.ecr.eu-west-1.amazonaws.com/myrepository:latest

4.    Run a command similar to the following to push the Docker image to an Amazon ECR registry:

docker push 1111222233334444.dkr.ecr.eu-west-1.amazonaws.com/myrepository:latest

5.    Run a command similar to the following to update the Docker file to use the newly pushed Amazon ECR image as the base image:

FROM 1111222233334444.dkr.ecr.eu-west-1.amazonaws.com/myrepository:tag

In the preceding commands, replace the following values with your values:

• example-image with the name of the public image that you want to push
• 1111222233334444 with your account ID
• myrepository:latest with the name of your Amazon ECR registry
• eu-west-1 with the Region of your choice

Authenticate the Docker Hub pull

When you authenticate with Docker Hub, you have more rate limits as an authenticated user and are rate limited based on the Docker username. Store your Docker Hub username and password as a secret in AWS Secrets Manager, and then use this secret to authenticate to Docker Hub.

Create a Secrets Manager secret for Docker Hub credentials

To create a secret for your Docker Hub credentials, use the instructions under the section To create a basic secret in Turning on private registry authentication.

Update your task execution IAM role

To grant the Amazon ECS task access to the secret, manually add the required permissions as an inline policy to the task execution role.

1.    Open the IAM console.

2.    In the navigation pane, choose Roles.

3.    Search the list of roles for ecsTaskExecutionRole, and then choose the role to view the attached policies.

4.    On the Permissions tab, choose Add permissions, and then choose Create inline policy.

5.    In the Create policy page, choose JSON, and then copy and paste the following policy:

{
	"Version": "2012-10-17",
	"Statement": [{
		"Effect": "Allow",
		"Action": [
			"secretsmanager:GetSecretValue",
			"kms:Decrypt"
		],
		"Resource": [
			"arn:aws:secretsmanager:eu-west-1:1111222233334444:secret:dockerhub-0knT",
			"arn:aws:kms:eu-west-1:1111222233334444:key/mykey"
		]
	}]
}

In the preceding policy, replace the following values with your values:

• 1111222233334444 with your account ID
• eu-west-1 with the Region of your choice
• mykey with your AWS KMS key

Note: Include kms:Decrypt only if your key uses a custom AWS Key Management Service (AWS KMS) key. Add the ARN for your custom key as a resource.

6.    Choose Review policy.

7.    For Name, enter the name of the policy (ECSSecrets).

8.    Choose Create policy.

Create a task definition that uses the secret for Docker authentication

Follow the instructions in Creating a task definition using the classic console to create your Amazon ECS task definition. For Task execution role, select the task execution IAM role that you updated in the preceding section.

In the Container definitions section, complete the following steps:

1.     Choose Add container.

2.    For Container name, enter the name of your container.

3.     For Image, enter the name of the image, or include the path to your private image (example: repository-url/image.tag).

4.    Choose Private repository authentication.

5.    For Secrets Manager ARN or name, enter the ARN of the secret that you created.

6.    Choose Add.

Create an Amazon ECS cluster and run the Amazon ECS task

Create an Amazon ECS cluster. Then, use the task definition that you created to run the task.

Use Amazon ECR public registry for public container images

Identify the public images that you're using in the Docker file. Use the appropriate search filters to search for these images on the Amazon ECR Public Gallery. You don't need to authenticate to browse the public repositories and pull images. The Amazon ECR Public contains popular base images, including operating systems, AWS-published images, Kubernetes add-ons, and artifacts. Pull images from the Amazon ECR public registry to avoid reaching the Docker Hub's rate limit.

Use these images as the source for the container image in your task definition:

ContainerDefinitions: [
 {
 ...
 Image: 'public.ecr.aws/cloudwatch-agent/cloudwatch-agent:latest'
 ...
 }
 ]

You can also choose to use these images as the base image in your Docker file:

Docker File  
FROM public.ecr.aws/amazonlinux/amazonlinux:latest

Upgrade to a Docker Pro or Team subscription

If you require more pulls, then upgrade your plan to a Docker Pro or Team subscription that offers 50,000 pulls in a 24-hour period. For more information on pricing plans, see Pricing and subscriptions (from the Docker Hub website).

Related information

Amazon ECR pricing

Amazon ECR Public service quotas