How do I troubleshoot high latency on my ELB Classic Load Balancer?

Short description

High latency on a Classic Load Balancer occurs because of the following reasons:

• Network connectivity issues
• Incorrect configuration of the Classic Load Balancer
• High memory (RAM) utilization on backend instances
• High CPU utilization on backend instances
• Improper web server configuration on backend instances
• Problems with web application dependencies that run on backend instances, such as external databases or Amazon Simple Storage Service (Amazon S3) buckets

Resolution

1.    Troubleshoot network connectivity issues for your Classic Load Balancer.

2.    Configure the Classic Load Balancer for your use case.

3.    Check the access logs for your Classic Load Balancer to determine which backend instances experience high latency. Review backend_processing_time to find backend instances with latency issues.
To verify that a backend instance's web application server experiences high latency, use curl to measure the first byte response:

[ec2-user@ip-192.0.2.0 ~]$ for X in `seq 6`; do curl -Ik -w "HTTPCode=%{http_code} TotalTime=%{time_total}\n" http://www.example.com/ -so /dev/null; done  
High Latency sample output:  
HTTPCode=200 TotalTime=2.452  
HTTPCode=200 TotalTime=1.035
Low latency sample output:  
HTTPCode=200 TotalTime=0.515  
HTTPCode=200 TotalTime=0.013

4.    Check the Average statistic of the Amazon CloudWatch Latency metric for your Classic Load Balancer. If the value is high, then there's a problem with backend instances or application dependency servers.

5.    Check the Maximum statistic of the Latency metric. If the value meets or exceeds the idle timeout value, then requests time out and result in HTTP 504 errors.

6.    Check for patterns in the Latency metric. Metric spikes at regular intervals indicate performance problems with backend instances because of overhead from scheduled tasks.

7.    Check the CloudWatch SurgeQueueLength metric for ELB. If requests to the Classic Load Balancer exceed the maximum value (1024), then the requests are rejected and the load balancer generates an HTTP 503 error. The sum statistic of the SpilloverCount metric measures the total number of rejected requests. For more information, see How do I troubleshoot Classic Load Balancer capacity issues in ELB?

8.    Review the Apache that's processed on your backend instances to check for memory issues.

Example command:

watch -n 1 "echo -n 'Apache Processes: ' && ps -C apache2 --no-headers | wc -l && free -m"

Example output:

Every 1.0s: echo –n 'Apache Processes: ' && ps –C apache2 –no-
headers | wc -1 && free –m
Apache Processes: 27
          total     used     free     shared     buffers     cached
Mem:      8204      7445     758      0          385         4567
-/+ buffers/cache:  2402     5801
Swap:     16383     189      16194

9.    Check the CloudWatch CPUUtilization metric of your backend instances. Look for high CPU utilization or spikes in CPU utilization. For high CPU utilization, upgrade your instances to a larger instance type.

10.    Check the MaxClient setting for the web servers on your backend instances. This setting defines how many simultaneous requests that the instance can serve. For instances with appropriate memory and CPU utilization that experience high latency, increase the MaxClient value.

Compare the number of processes that Apache (httpd) generates with the MaxClient setting. If the number of Apache processes frequently reaches the MaxClient value, then increase the value.

Example command:

[root@ip-192.0.2.0 conf]# ps aux | grep httpd | wc -l 15

Example output:

<IfModule prefork.c>  
StartServers         10  
MinSpareServers      5  
MaxSpareServers      10  
ServerLimit          15  
MaxClients           15  
MaxRequestsPerChild  4000  
</IfModule>

11.    Check for dependencies that cause latency issues on your backend instances.

Here are some of the dependencies that can cause latency issues:

• Shared databases, external resources, such as S3 buckets
• external resource connections, such as network address translation (NAT) instances
• remote web services
• proxy servers

Related information

Monitor your Classic Load Balancer