How can I analyze my CloudTrail Logs with CloudWatch Logs Insights?

Short description

You can configure CloudTrail to log to CloudWatch Logs. Doing so allows you to use CloudWatch Logs Insights to analyze the CloudTrail logs to monitor specific account activity.

To demonstrate Logs Insights ability to analyze CloudTrail logs, the following resolution contains sample queries. These queries cover the most common use cases:

• Isolating specific log fields.
• Filtering on different conditions.
• Aggregating events.
• Building a time series.

Resolution

The following queries explore Amazon Simple Storage Service (Amazon S3) bucket and object activity. By default, CloudTrail doesn't capture S3 data events. You can turn on event logging in CloudTrail. For more information, see Enabling CloudTrail event logging for S3 buckets and objects.

You can build on these example queries to create additional and more complex Logs Insights queries aligned to your use case. You can also integrate queries with your CloudWatch dashboard to visualize your queries as charts and graphs alongside related metrics.

Query 1: Latest events

Objective

Retrieves the most recent CloudTrail Log events with the default @timestamp and @message fields.

Query

#Retrieve the most recent CloudTrail events
fields @timestamp, @message
| sort @timestamp desc
| limit 2

Results

@timestamp	@message
2022-02-18 17:52:31.118	{"eventVersion":"1.08","userIdentity":{"type":"AssumedRole","principalId":"AROAWZKRRJU47ARZN7ECC:620d7d78144334d6933c27195cae2a98", "arn":"arn:aws:sts::123456789012:assumed- role/Amazon_EventBridge_Invoke_Run_Command_371790151/620d7d78144334d6933c27195cae2a98","accountId":"123456789012", "accessKeyId":"ASIAWZKRRJU4Y45M4SC6","sessionContext":{"sessionIssuer": {"type":"Role","principalId":"AROAWZKRRJU47ARZN7ECC","arn":"arn:aws:iam::123456789012:role/service- role/Amazon_EventBridge_Invoke_Run_Command_371790151","accountId":"123456789012","userName": "Amazon_EventBridge_Invoke_Run_Command_371790151" (output truncated)
2022-02-18 17:51:52.137	{"eventVersion":"1.08","userIdentity":{"type":"AssumedRole","principalId":"AROAWZKRRJU43YP4FHR2N:StateManagerService","arn":"arn:aws:sts::123456789012:assumed-role/AWSServiceRoleForAmazonSSM/StateManagerService","accountId":"123456789012","sessionContext":{"sessionIssuer":{"type":"Role","principalId":"AROAWZKRRJU43YP4FHR2N","arn":"arn:aws:iam::123456789012:role/aws-service-role/ssm.amazonaws.com/AWSServiceRoleForAmazonSSM","accountId":"123456789012","userName":"AWSServiceRoleForAmazonSSM"}, "webIdFederationData":{},"attributes":{"creationDate":"2022-02-18T17:50:06Z","mfaAuthenticated":"false"}},"invokedBy":"ssm.amazonaws.com"},"eventTime":"2022-02-18T17:50:06Z","eventSource":"ec2.amazonaws.com","eventName":"DescribeInstances","awsRegion":"eu-west-1","sourceIPAddress":"ssm.amazonaws.com","userAgent":"ssm.amazonaws.com","requestParameters":{"maxResults":50,"instancesSet": (output truncated)

Query 2: Break out individual fields

Objective

• Isolate the individual fields in @message.
• Display select fields in the CloudTrail event.

Query

#Breakout Individual Fields
fields @timestamp, awsRegion, eventCategory, eventSource, eventName, eventType, sourceIPAddress, userIdentity.type
| sort @timestamp desc
| limit 2

Results

@timestamp	awsRegion	eventCategory	eventSource	eventName	eventType	sourceIPAddress	userIdentity.type
2022-02-18 18:00:09.647	ca-central-1	Management	sts.amazonaws.com	AssumeRole	AwsApiCall	cloudtrail.amazonaws.com	AWSService
2022-02-18 18:00:09.647	ca-central-1	Management	sts.amazonaws.com	AssumeRole	AwsApiCall	cloudtrail.amazonaws.com	AWSService

Query 3: Filter by Amazon Elastic Compute Cloud (Amazon EC2) RunInstances

Objective

• Retrieve specific fields in the Cloudtrail event.
• Rename fields with more meaningful labels.
• Filter the latest EC2 instances launched in this account based on API call.

Query

#EC2: Recently Launched Instances
fields eventTime, eventName as API, responseElements.instancesSet.items.0.instanceId as InstanceID, userIdentity.sessionContext.sessionIssuer.type as IssuerType, userIdentity.type as IdentityType, userIdentity.sessionContext.sessionIssuer.userName as userName
| filter eventName = 'RunInstances'
| sort eventTime desc
| limit 2

Results

eventTime	API	InstanceID	IssuerType	IdentityType	userName
2022-02-18T17:36:38Z	RunInstances	i-0325b4d6ae4e93c75	Role	AssumedRole	AWSServiceRoleForAutoScaling
2022-02-18T13:45:18Z	RunInstances	i-04d17a8425b7cb59a	Role	AssumedRole	AWSServiceRoleForAutoScaling

Query 4: Filter by console login: Most recent

Objective

• Retrieve specific fields in the Cloudtrail event.
• Rename fields with more meaningful labels.
• Filter the latest logins via the AWS Console based on API call.

Query

#Console Login: Most Recent API Calls
fields eventTime, eventName, responseElements.ConsoleLogin as Response, userIdentity.arn as ARN, userIdentity.type as User_Type
| filter eventName = 'ConsoleLogin'
| sort eventTime desc
| limit 10

Results

eventTime	eventName	Response	ARN	User_Type
2022-02-18T17:35:44Z	ConsoleLogin	Success	arn:aws:iam::123456789012:user/test_user	IAMUser
2022-02-17T13:53:58Z	ConsoleLogin	Success	arn:aws:sts::123456789012:assumed-role/Admin/test_user	AssumedRole

Query 5: Filter by console login: Failed authentication

Objective

• Retrieve specific fields (for example, username, user type, source IP) in the Cloudtrail event.
• Rename fields with more meaningful labels.
• Filter the latest unsuccessful logins via the AWS Console.

Query

#ConsoleLogin: Filter on Failed Logins
fields eventTime, eventName, responseElements.ConsoleLogin as Response, userIdentity.userName as User, userIdentity.type as User_Type, sourceIPAddress, errorMessage
| filter eventName = 'ConsoleLogin' and responseElements.ConsoleLogin = 'Failure'
| sort eventTime desc
| limit 10

Results

eventTime	eventName	Response	User	User_Type	sourceIPAddress	errorMessage
2022-02-18T20:10:55Z	ConsoleLogin	Failure	echo	IAMUser	12.34.56.89	Failed authentication
2022-02-18T20:10:43Z	ConsoleLogin	Failure	echo	IAMUser	12.34.56.89	Failed authentication

Query 6: Filter by Amazon Simple Storage Solution (Amazon S3) object upload

Objective

• Retrieve specific fields in the Cloudtrail event.
• Rename fields with more meaningful labels.
• Filter on API call and target S3 bucket.

Query

#Filter PutObject API Calls on a specific S3 Bucket
fields @timestamp, eventName as API, requestParameters.bucketName as BucketName, requestParameters.key as Key, userIdentity.sessionContext.sessionIssuer.userName as UserName
| filter eventName = 'PutObject' and BucketName = 'target-s3-bucket'
| sort @timestamp desc
| limit 2

Results

@timestamp	API	BucketName	Key	UserName
2022-02-12 17:16:07.415	PutObject	test_bucket1	w4r9Hg4V7g.jpg
2022-02-12 16:29:43.470	PutObject	test_bucket2	6wyBy0hBoB.jpg

Query 7: Summarize S3 activity

Objective

• Filter based on S3 service.
• Aggregate all the matching events based on count statistic.
• Splice the results based on API, S3 bucket, and key.
• Use the stats command to rename fields.
• Sort by descending order.

Query

#S3 Activity: Bucket & Key Details
filter eventSource = 's3.amazonaws.com'
| stats count(*) as Hits by eventName as API, requestParameters.bucketName as BucketName, requestParameters.key as Key
| sort Hits desc
| limit 5

Results

API	BucketName	Key	Hits
ListAccessPoints			44
GetBucketAcl	team1-ctrail-multi-region		27
GetBucketAcl	team2-dub-cloudtrail		27
GetBucketAcl	aws-cloudtrail-logs-123456789012-ba940dd7		26
GetObject	devsupport-prod	rdscr/individual/123456789012	18

Query 8: Summarize AWS Key Management Service (AWS KMS) decrypt activity

Objective

• Filter based on KMS service and Decrypt API.
• Use the fields command to rename fields and aggregate on the user-friendly names.
• Aggregate all the matching events based on count statistic.
• Splice the results based on KMS Key and User.
• Sort by descending order.

Query

#KMS Decrypt Activity: Key & User Details
fields resources.0.ARN as KMS_Key, userIdentity.sessionContext.sessionIssuer.userName as User
| filter eventSource='kms.amazonaws.com' and eventName='Decrypt'
| stats count(*) as Hits by KMS_Key, User
| sort Hits desc
| limit 2

Results

KMS_Key	User	Hits
arn:aws:kms:us-east-1:123456789012:key/03f2923d-e213-439d-92cf-cbb444bd85bd	AWSServiceRoleForConfig	12
arn:aws:kms:us-east-1:123456789012:key/03f2923d-e213-439d-92cf-cbb444bd85bd	FoxTrot-1UQJBODTWZYZ6	8

Query 9: Summarize API calls with errors

Objective

• Filter based on the presence of the errorCode field.
• Aggregate all of the matching events based on the count statistic.
• Splice the results based on AWS Service, API, and errorCode.
• Use the stats command to rename fields.
• Sort by largest number of matches.

Query

#Summarize API Calls with Errors
filter ispresent(errorCode)
| stats count(*) as Num_of_Events by eventSource as AWS_Service, eventName as API, errorCode
| sort Num_of_Events desc
| limit 5

Results

AWS_Service	API	errorCode	Num_of_Events
s3.amazonaws.com	GetBucketPublicAccessBlock	NoSuchPublicAccessBlockConfiguration	79
lambda.amazonaws.com	GetLayerVersionPolicy20181031	ResourceNotFoundException	66
s3.amazonaws.com	GetBucketPolicyStatus	NoSuchBucketPolicy	60
s3.amazonaws.com	HeadBucket	AccessDenied	47
logs.amazonaws.com	CreateLogStream	ResourceNotFoundException	21

Query 10: Summarize S3 API calls with error codes

Objective

• Filter based on S3 Service and the presence of errorCode field.
• Aggregate all the matching events based on the count statistic.
• Splice the results based on errorCode and errorMessage.
• Sort by largest number of matches.

Query

#S3: Summarize Error Codes
filter eventSource = 's3.amazonaws.com' and ispresent(errorCode)
| stats count(*) as Hits by errorCode, errorMessage
| sort Hits desc
| limit 5

Results

errorCode	errorMessage	Hits
AccessDenied	Access Denied	86
NoSuchBucketPolicy	The bucket policy does not exist	80
NoSuchPublicAccessBlockConfiguration	The public access block configuration was not found	79
ObjectLockConfigurationNotFoundError	Object Lock configuration does not exist for this bucket	3
ServerSideEncryptionConfigurationNotFoundError	The server side encryption configuration was not found	3

Query 11: Summarize AccessDenied/UnauthorizedOperation API calls by AWS service, API, and AWS Identity and Access Management (IAM) user

Objective

• Filter on either AccessDenied or UnauthorizedOperation CloudTrail events.
• Aggregate all the matching events based on count statistic.
• Splice the results based on errorCode, AWS Service, API, and IAM user/role.
• Use the stats command to rename fields.
• Sort in descending order.

Query

#Summarize AccessDenied/UnauthorizedOperation API Calls by AWS Service, API, IAM User
filter (errorCode='AccessDenied' or errorCode='UnauthorizedOperation')
| stats count(*) as NumberOfEvents by errorCode, eventSource as AWS_Service, eventName as API, userIdentity.type as IdentityType, userIdentity.invokedBy as InvokedBy
| sort NumberOfEvents desc
| limit 10

Results

errorCode	AWS_Service	API	IdentityType	InvokedBy	NumberOfEvents
AccessDenied	s3.amazonaws.com	HeadBucket	AWSService	delivery.logs.amazonaws.com	83
AccessDenied	s3.amazonaws.com	GetObject	AssumedRole		9

Query 12: Time series: KMS hourly call volume

Objective

• Filter based on KMS Service and Decrypt API.
• Aggregate all the matching events into 1 hour bins.
• Visualize the results on a line graph.

Query

#KMS: Hourly Decrypt Call Volume
filter eventSource='kms.amazonaws.com' and eventName='Decrypt'
| stats count(*) as Hits by bin(1h)

Results

bin(1h)	Hits
2022-02-18 19:00:00.000	16
2022-02-18 18:00:00.000	25
2022-02-18 17:00:00.000	28
2022-02-18 16:00:00.000	14
2022-02-18 15:00:00.000	16

---

Related information

Monitor AWS CloudTrail log data in Amazon CloudWatch (Video)

Add query to dashboard or export query results

CloudWatch Logs Insights sample queries

CloudWatch Logs Insights query commands