How do I troubleshoot logging issues in Elastic Beanstalk?

3 minute read
0

My log file is empty, or my log stream suddenly stopped streaming log messages. I want to resolve this issue in AWS Elastic Beanstalk.

Resolution

Choose one of the following options based on the logging issue that you experience:

Find your empty log files

By default, Elastic Beanstalk keeps all logs on the Amazon Elastic Compute Cloud (Amazon EC2) instances in your environment. Elastic Beanstalk periodically rotates your logs to manage file size, and conserve disk space. Elastic Beanstalk exports your logs to Amazon Simple Storage Service (Amazon S3). For more information, see Log rotation settings on Linux. To access recently rotated log messages, check /var/log/rotated.

Determine why your log file and log stream stopped outputting new log messages

To continuously stream your log files from your Elastic Beanstalk EC2 instances, use Amazon CloudWatch Logs.

If log streaming doesn't work after the first log rotation occurs on your EC2 instance, then your application might not be logging unique lines. Your application log might look like the following example:

Hello my application!- at time 1:00 PM
Hello my application!
- at time 2:00 PM
...

In the preceding example, not every line is unique. A unique entry spans two lines. It takes a minimum of two lines to uniquely identify this log file.

Note: To detect when log rotation occurs, the CloudWatch Logs agent identifies your log file and uses the unique number of lines that your file contains.

To get log streaming to work again, try one of the following solutions:

Edit the application log format to include a unique identifier, such as a timestamp, at the start of each line.

-or-

Edit the file_fingerprint_lines value to count the lines of your file until enough lines are included to uniquely identify the file.

Note: To edit this value, you must customize the CloudWatch Logs agent configuration. The Elastic Beanstalk integration with CloudWatch Logs doesn't directly support streaming the custom log files that your application generates. To stream custom logs, use a configuration file to install the CloudWatch Logs agent and configure the files to push.

For example, the following .ebextension customizes file_fingerprint_lines for Node.js application log files:

files:  "/etc/awslogs/config/beanstalklogs.conf":
    mode: "000644"
    user: root
    group: root
    content: |
      [/var/log/nodejs/nodejs.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/nodejs/nodejs.log
      log_stream_name={instance_id}
      file=/var/log/nodejs/nodejs.log*
      file_fingerprint_lines=2-5

       [/var/log/nginx/error.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/nginx/error.log
      log_stream_name={instance_id}
      file=/var/log/nginx/error.log*

       [/var/log/nginx/access.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/nginx/access.log
      log_stream_name={instance_id}
      file=/var/log/nginx/access.log*

       [/var/log/httpd/error.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/httpd/error.log
      log_stream_name={instance_id}
      file=/var/log/httpd/error.log*

       [/var/log/httpd/access.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/httpd/access.log
      log_stream_name={instance_id}
      file=/var/log/httpd/access.log*

       [/var/log/eb-activity.log]
      log_group_name=/aws/elasticbeanstalk/streamlogs/var/log/eb-activity.log
      log_stream_name={instance_id}
      file=/var/log/eb-activity.log*
 commands:
  01_remove_eb_stream_config:
    command: 'rm -fr /etc/awslogs/config/beanstalklogs.conf.bak'
  02_restart_log_agent:
    command: 'service awslogs restart'

Troubleshoot the failed creation of a log stream

If you turned on log streaming but your log stream still isn't showing up, then see Troubleshooting CloudWatch Logs integration.

Related Information

Using Elastic Beanstalk with Amazon CloudWatch Logs

Custom log file streaming

logs-streamtocloudwatch-linux.config on the GitHub website

AWS OFFICIAL
AWS OFFICIALUpdated 7 months ago