Why are properly functioning Amazon ECS tasks registered to ELB marked as unhealthy and replaced?

2 minute read
0

Elastic Load Balancing (ELB) is repeatedly flagging properly functioning Amazon Elastic Container Service (Amazon ECS) tasks as unhealthy. These incorrectly flagged tasks are stopped, and then new tasks are started instead.

Short description

Some Amazon ECS tasks have several dependencies and lengthy bootstrapping processes that can exceed the ELB health check grace period, even when functioning as intended. When Amazon ECS tasks don't respond to ELB health checks within the grace period, they're flagged as unhealthy. To increase the health check grace period for your service, complete the following steps.

To troubleshoot ECS tasks failing an Application Load Balancer health check, see How can I get my Amazon ECS tasks running using the Amazon EC2 launch type to pass the Application Load Balancer health check in Amazon ECS?

Resolution

If no grace period is configured, then the service scheduler immediately replaces any targets marked as unhealthy. Change the grace period to allow more time for your Amazon ECS tasks to complete their processes and pass the health check.

Note: To change the grace period, use the earlier version of the ECS console. To change to the earlier version of the console, toggle off New ECS Experience at the top of the navigation pane. Then, complete the following steps.

  1. Open the AWS Management Console.
  2. In the navigation bar, choose Services, and then select ECS from the list.
  3. Select your service from the Service Name list.
  4. Choose Update.
  5. Choose Next step.
  6. On the Step 2: Configure network page, change the Health check grace period to an appropriate time period for your service. The maximum time period is 2,147,483,647 seconds.
    Caution: To prevent delayed replacement of legitimately unhealthy Amazon ECS tasks, carefully estimate the required grace period for your lengthiest tasks. When setting your grace period, consider all relevant factors, such as bootstrap time and time to pull container images.
  7. Choose Next step, and then choose Update Service.

You can also use these ways to increase the grace period:

  • Use the HealthCheckGracePeriodSeconds parameter defined in the AWS::ECS::Service resource in AWS CloudFormation.
  • Run the UpdateService command in the AWS Command Line Interface (AWS CLI), and increase the --health-check-grace-period-seconds value.

Related information

Amazon ECS adds ELB health check grace period

AWS OFFICIAL
AWS OFFICIALUpdated a year ago
No comments