How do I troubleshoot missing container logs for Amazon ECS or Amazon EKS?

4 minute read
0

I can't find the container logs for my Amazon Elastic Container Service (Amazon ECS) task or Amazon Elastic Kubernetes Service (Amazon EKS) pod.

Short description

Your containerized application doesn't write some or all logs when you:

  • Run the "docker logs yourContainerName" command on a container instance in Amazon ECS.
  • Use the awslogs log driver for a task in Amazon ECS.
  • Run the "kubectl logs yourPodName" command for an Amazon EKS cluster.

You can experience issues when:

  • There are problems with the host instance.
  • Your logs are not visible to the Docker daemon because your containerized application isn't writing to the correct location.

Resolution

Find the logs for your Amazon ECS tasks

To confirm that your task is configured correctly so that you can find your logs, consider the following:

  • The log driver is set by the logConfiguration parameter in the container definition section of your ECS task definition. Log drivers are set per container. If your ECS task has multiple container definitions, then verify that the log configuration is correct for the containers with your logs.
  • The Amazon ECS container agent automatically removes log files to reclaim free space. To preserve your log files for longer on your container instance, reduce the frequency of your task cleanup.

If your tasks are using the awslogs log driver, then the following conditions are true:

  • Logs are streamed to Amazon CloudWatch Logs. These logs are never written to the container instance.
  • The "docker logs yourContainerName" command returns the following error message: "Error response from daemon: configured logging driver does not support reading."

To allow EC2 launch types to stream to CloudWatch Logs, grant permissions on the IAM role used by your container instance.

To allow AWS Fargate launch types to stream to CloudWatch Logs, grant permissions on the task execution role used by your task.

If your logs are still missing, complete the steps in the Troubleshoot the container section.

Find the logs for your Kubernetes pods on Amazon EKS

To return the log files generated from a pod's container, run the following kubectl command:

kubectl logs yourPodName

Note: The kubelet automatically removes log files after a pod exits. For more information, see Container collection on the Kubernetes website. To preserve these log files for longer on a worker node, configure the kubelet to run garbage collection less frequently.

If your logs are still missing, complete the steps in the Troubleshoot the container section.

Troubleshoot the container

Consider the following options for your application:

  • Set your application to the correct log level during your container build.
    Note: Your application could require you to set logging through an environment variable or in a configuration file.
  • Make your application the ENTRYPOINT of the container. For more information, see ENTRYPOINT on the Docker Docs website.
    Note: The ENTRYPOINT in your dockerfile is the process where STDOUT and STDERR I/O streams are logged on the Docker Docs website. For more information, see View logs for a container or service on the Docker Docs website.
  • Build your container with application log files linked to STDOUT and STDERR, or configure your application to log directly to /proc/1/fd/1 (stdout) and /proc/1/fd/2 (stderr). For examples, see the official nginx and httpd container images on the Docker Hub website.
    Note: If possible, make your application process the parent process in your container. If your container uses a shell script as the ENTRYPOINT, configure the script to initialize your container data at runtime. For more information, see ENTRYPOINT on the Docker Docs website.

AWS OFFICIAL
AWS OFFICIALUpdated 4 years ago