How can I use CloudWatch Logs Insights queries with my VPC flow log?

4 minute read
2

I want to use Amazon CloudWatch Logs Insights queries to process my Amazon Virtual Private Cloud (Amazon VPC) flow logs that are in a log group. How can I do this?

Short description

After you turn on VPC flow logs targeting CloudWatch Logs, you see one log stream for each elastic network interface. CloudWatch Logs Insights is a query tool that can perform complex queries on log events stored in log groups. If an issue occurs, you can use CloudWatch Logs Insights to identify potential causes and validate deployed fixes.

For information on supported log types, see Supported logs and discovered fields.

Resolution

Run a query

To run a query, do the following:

1.    Open the Cloudwatch console.

2.    Select Logs, Logs Insights.

3.    On the Logs Insights dashboard, select the log group that you want to analyze and visualize data for.

4.    You can create a query, or you can run one of the provided sample queries for VPC flow logs. If you're creating a custom query, start by reviewing the tutorials provided in the Amazon CloudWatch documentation. For information on query syntax, see CloudWatch Logs Insights query syntax.

5.    Select History to view your previously executed queries. You can run queries again from History.

6.    To export your results, select Export results and then choose a format.

Example queries

Scenario 1

You have a webserver, application server, and DB server. The application isn't working as expected. For example, you're receiving a timeout or HTTP 503 error and you're trying to determine the cause of the error.

Example variables:

  • Action is set to "REJECT" so that only rejected connections are returned.
  • The query includes only internal networks.
  • The list of server IPs shows both inbound and outbound connections (srcAddr and dstAddr).
  • The Limit is set to 5 so that only the first five entries are shown.
  • Web server IP: 10.0.0.4
  • App server IP: 10.0.0.5
  • DB server IP: 10.0.0.6
filter(
action="REJECT" and
dstAddr like   /^(10\.|192\.168\.)/and
srcAddrlike   /^(10\.|192\.168\.)/ and
(
srcAddr = "10.0.0.4" or
dstAddr = "10.0.0.4" or
srcAddr = "10.0.0.5" or
dstAddr = "10.0.0.5" or
srcAddr = "10.0.0.6" or
dstAddr = "10.0.0.6" or
)
)|
stats count(*) as records by srcAddr,dstAddr,dstPort,protocol |
sort records desc |
limit 5

Scenario 2

You're experiencing intermittent timeouts on a given elastic network interface. The following query checks for any rejects on the elastic network interface over a period of time.

fields @timestamp, interfaceId, srcAddr, dstAddr, action
| filter (interfaceId = 'eni-05012345abcd' and action = 'REJECT')
| sort @timestamp desc
| limit 5

Scenario 3

The following query example analyzes VPC flow logs to produce a report on a specific elastic network interface. The query checks the amount of traffic that's being sent to different ports.

fields @timestamp, @message
 | stats count(*) as records by dstPort, srcAddr, dstAddr as Destination
 | filter interfaceId="eni-05012345abcd"
 | filter dstPort="80" or dstPort="443" or dstPort="22" or dstPort="25"
 | sort HitCount desc
 | limit 10

Scenario 4

The following query filters VPC flow logs to list IP addresses that are trying to connect with a specific IP or CIDR in your VPC.

For a specific IP:

fields @timestamp, srcAddr, dstAddr
 | sort @timestamp desc
 | limit 5
 | filter srcAddr like "172.31."

For a specific CIDR:

fields @timestamp, srcAddr, dstAddr
 | sort @timestamp desc
 | limit 5
 | filter isIpv4InSubnet(srcAddr,"172.31.0.0/16)

Note: For additional example queries, see Sample queries.


Related information

Analyzing log data with CloudWatch Logs Insights

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago