How do I troubleshoot an unresponsive list command after I perform a large deletion in Amazon S3?

3 minute read
0

I performed a large number of file deletions, and I want to run the list command on my Amazon Simple Storage Service (Amazon S3) bucket. However, the list command stops responding and doesn't return any output.

Resolution

If you deleted a large number of files and the ls or list-objects command stops responding, then follow these troubleshooting steps:

Remove expired object delete markers

In a versioning-enabled bucket, when you use the delete operation for a versioned object and don't specify a version ID, a delete marker is created. When lifecycle configuration rules set previous object versions to expire, the bucket contains the delete markers that are associated with the expired objects. If there are thousands of delete markers, then the list operation might time out. Therefore, it's a best practice to remove the expired object delete markers from the bucket to improve the performance of a list operation.

To efficiently clean up expired object delete markers, use a lifecycle rule:

  1. Open the Amazon S3 console.
  2. From the list of buckets, choose the bucket that contains the expired object delete markers.
  3. Choose the Management tab.
  4. Choose Add lifecycle rule and enter a rule name. Then, choose Next.
  5. For Storage class transition, skip this section and choose Next.
  6. For Configure expiration, choose Clean up expired object delete markers. Then, choose Next.
  7. Review the lifecycle rule settings, and then choose Save.

Try the list-objects-v2 command or the latest version of the AWS CLI

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

The list-objects-v2 command, or ListObjectsV2 API call, is an updated version of the list operation that includes some performance improvements. The following example command requests a list of objects in awsexamplebucket:

aws s3api list-objects-v2 --bucket awsexamplebucket

Note: For backward compatibility, Amazon S3 continues to support the ListObjects API call.

Use an Amazon S3 inventory report

Instead of a list operation, use an Amazon S3 inventory report to get a list of your objects. When you run multiple list operations for a bucket that has hundreds of millions of objects, the process can be challenging and costly.

You can get daily or weekly Amazon S3 inventory reports for a bucket or prefix, and include additional object metadata in your reports. You can also pass an inventory report as an input for a batch operation, and use Amazon Athena to query a report.

Related information

Amazon S3 pricing

AWS OFFICIAL
AWS OFFICIALUpdated 6 months ago