How do I use a lifecycle configuration rule to empty an Amazon S3 bucket?
I have an Amazon Simple Storage Service (Amazon S3) bucket that stores millions of objects. I want to use a lifestyle configuration rule to empty the bucket so that I'm not charged for storage.
Resolution
Important: The following resolution permanently deletes all data in your Amazon S3 bucket. Because you can't recover the data, review all objects and data in the bucket before you delete them.
To create a lifecycle configuration rule that expires current versions of objects and permanently deletes previous versions of objects, complete the following steps:
- Open the Amazon S3 console.
- From the list of buckets, choose the bucket that you want to empty.
- Choose the Management tab.
- Choose Create lifecycle rule.
- For Lifecycle rule name, enter a rule name.
- For Choose a rule scope, choose Apply to all objects in the bucket.
- Select I acknowledge that this rule will apply to all objects in the bucket.
- For Lifecycle rule actions, select the following options:
Expire current versions of objects
Permanently delete noncurrent versions of objects
Delete expired object delete markers or incomplete multipart uploads - In the Expire current versions of objects field, for the Days after object creation field, enter 1.
- In the Permanently delete noncurrent versions of objects field, for the Days after objects become noncurrent field, enter 1.
- To delete all versions, leave the Number of newer versions to retain (Optional) field empty.
- Select Delete incomplete multipart uploads, and then enter 1 for the Number of days field.
- Choose Create rule.
- To create a second lifecycle rule, repeat steps 4-7.
- Select the following option: Delete expired object delete markers or incomplete multipart uploads.
- Select Delete expired object delete markers.
- Choose Create rule.
Amazon S3 runs lifecycle rules once a day. After the first time that Amazon S3 runs the rules, all objects that are eligible for expiration are marked for deletion. You're no longer charged for objects that are marked for deletion.
Amazon S3 asynchronously expires object versions and removes delete markers. It might take the rules a few days to run before the bucket is empty. For more information about asynchronous object removal in Amazon S3, see Expiring objects.
You can also use the AWS Command Line Interface (AWS CLI) to create a lifecycle rule to empty your S3 bucket.
JSON Example:
{ "Rules": [{ "Expiration": { "Days": 1 }, "ID": "FullDelete", "Filter": { "Prefix": "" }, "Status": "Enabled", "NoncurrentVersionExpiration": { "NoncurrentDays": 1 }, "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 } }, { "Expiration": { "ExpiredObjectDeleteMarker": true }, "ID": "DeleteMarkers", "Filter": { "Prefix": "" }, "Status": "Enabled" } ] }
Related information
Removing expired object delete markers
Managing your storage lifecycle
I want to empty a bucket using the AWS SDK for JavaScript. According to the documentation, "If you want the S3 Lifecycle rule to apply to all objects in the bucket, specify an empty prefix". However, I'm not sure how to do this. When I don't include a "Prefix" or a "Filter" property or include a "Filter" with an undefined "Prefix", I receive an error response of "MalformedXML: The XML you provided was not well-formed or did not validate against our published schema". The rule I'm trying to send is shown below:
const params = {
Bucket: "bucket-name",
LifecycleConfiguration: {
Rules: {
ID: "Empty bucket s3",
Filter: {
}
Status: "Enabled",
Expiration: {
Days: 1
},
AbortIncompleteMultipartUpload: {
DaysAfterInitiation: 1
}
}
};
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
This is not possible using boto3, the example:
lifecycle_configuration = {
'Rules': [
{
'Expiration': {
'Days': 1,
},
'ID': 'DeleteAfter1Days',
'Status': 'Enabled',
'Prefix': '',
'NoncurrentVersionExpiration': {
'NoncurrentDays': 1
},
'AbortIncompleteMultipartUpload': {
'DaysAfterInitiation': 1
}
},
{
'Expiration': {
'ExpiredObjectDeleteMarker': True
},
'ID': 'ExpireDeleteMarkers',
'Status': 'Enabled',
'Prefix': '*'
}
]
}
print(f'Creating lifecyle for {bucket_name}')
s3.put_bucket_lifecycle_configuration(
Bucket=bucket_name,
LifecycleConfiguration=lifecycle_configuration
)
will result in "An error occurred (InvalidRequest) when calling the PutBucketLifecycleConfiguration operation: Found overlapping prefixes '*' and '' for same action type 'Expiration'"
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
I did this for my bucket, but it's been 2 days now and there is still data in my bucket? Does the rule not run automatically everyday?
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
In the past, if you had a large number of objects in a bucket, you could get a very large bill for LIST API calls before the DELETE calls. Can you verify that if the Lifecycle Policy above is used that you will not get a large bill for deleting millions/billions of objects?
Thank you for your comment. We'll review and update the Knowledge Center article as needed.
Relevant content
- Accepted Answerasked a year ago
- asked 3 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 5 months ago
