Remove objects from a bucket automatically with AWS CLI

What is a lifecycle policy?

A lifecycle policy is a file written in XML (or JSON) format that contains rules for automating the movement of objects from one stage of their lifecycle to another. The movement is typically based on the “age” and size of the object, as well as other parameters. You can find more information on the different parameters in the Amazon documentation.

There are several ways to utilize a Lifecycle Policy. Gcore supports a lifecycle policy, which allows you to set an expiration time for an object and automatically remove it from the bucket once the expiration time is reached.

Lifecycle policy logic

Object deletion is processed around midnight UTC. The table below shows when an object is deleted based on the scenario.

Scenario	Upload date	Policy set	Expiration (days)	Deleted on
Policy exists before upload	Jan 2	Before Jan 2	1	Jan 4
Policy set after upload	Jan 1	After Jan 1	1	Jan 3
Policy removed	Any	Removed	—	Never

The midnight UTC processing window means deletion happens at the next processing cycle after the expiration period ends — not at the exact moment the period expires.

Gcore lifecycle configuration elements

Here is an example of a completed XML file with the lifecycle policy:

<LifecycleConfiguration>  
 <Rule>  
  <ID>...</ID>  
  <Prefix/>  
  <Status>Enabled</Status>  
  <Expiration>  
   <Days>...</Days>  
  </Expiration>  
</Rule>  
</LifecycleConfiguration>

• <Rule>...</Rule>. This section includes all the settings below.
• <ID>...</ID>. Specify a unique identifier for the rule. The ID can be up to 255 characters long and include numbers, Latin letters, and underscores.
• <Prefix/>. Leave the tag empty if you want to apply the rule to the entire bucket or specify the folder’s name.
• <Status>...</Status>. Specify either “Enabled” or “Disabled”. If a rule is in the “Disabled” status, no actions specified in the rule will be performed.
• <Expiration><Days>...</Days></Expiration>. Specify the days you want the object to be kept in your bucket in the “Expiration” section.

Manage a lifecycle policy with AWS CLI

Add a lifecycle policy to your bucket

1. Create an XML file:

<LifecycleConfiguration>  
 <Rule>  
  <ID>one_day</ID>  
  <Prefix/>  
  <Status>Enabled</Status>  
  <Expiration>  
   <Days>1</Days>  
  </Expiration>  
 </Rule>  
</LifecycleConfiguration>

Where:

• ID is one_day.
• The rule is applied to the entire bucket (<Prefix/>).
• The rule is active (status is Enabled).
• The file expiration time is 1 day.

2. Convert the XML file to the JSON format because the AWS CLI does not support the XML format. Your policy will look like this:

{  
  "Rules": [  
    {  
      "ID": "one_day",   
      "Prefix": "",  
      "Status": "Enabled",   
      "Expiration": {  
        "Days": 1  
      }  
    }  
  ]  
}

Note : The following header value ("Prefix": "") applies the Lifecycle Configuration to the entire bucket. Specify the folder name if you need to apply the policy to a specific folder in the bucket. For example, if the objects are in the deleteme folder, the value will be "Prefix": "deleteme /".

3. Save the file as lifecycle.json.

4. Start the AWS CLI from the directory with the lifecycle.json file and run the following command:

aws s3api put-bucket-lifecycle --bucket my_bucket --lifecycle-configuration file://lifecycle.json --endpoint-url=https://luxembourg-2.storage.gcore.dev 

Replace:

• my_bucket name in the example with your bucket name.
• https://luxembourg-2.storage.gcore.dev with your storage endpoint. To choose the correct value for this parameter, use the “S3 service URLs and default region names” guide.

5. Check if the policy was uploaded correctly with the following command:

aws s3api get-bucket-lifecycle-configuration --bucket my_bucket --endpoint-url=https://luxembourg-2.storage.gcore.dev 

Replace:

• my_bucket name in the example with your bucket name.
• https://luxembourg-2.storage.gcore.dev with your storage endpoint. To choose the correct value for this parameter, use the “S3 service URLs and default region names” guide.

In the response, you should receive the uploaded JSON file.

Delete a lifecycle policy

To delete the policy from the bucket, use the following command:

aws s3api delete-bucket-lifecycle --bucket my-bucket --endpoint-url=https://luxembourg-2.storage.gcore.dev

Replace:

• my_bucket name in the example with your bucket name.
• https://luxembourg-2.storage.gcore.dev with your storage endpoint. To choose the correct value for this parameter, use the “S3 service URLs and default region names” guide.

Manage a lifecycle policy with the UI

If you are using S3 storage in Luxembourg, you can manage your lifecycle policy for buckets in the Gcore Customer Portal according to the “Manage buckets via the Gcore Customer Portal” guide.

Protect objects from deletion with Object Lock

To prevent objects from being deleted or overwritten, use S3 Object Lock. See Protect objects with Object Lock.