Move objects between Storage Classes using S3 Bucket Lifecycle Management
Let’s take advantage of S3 Bucket Lifecycle Management to move objects between storage classes so that they are stored cost effectively throughout their lifecycle.
We will describe the steps to transition objects between STANDARD and COLD Storage Classes after 7 days.
Adding a Storage Class
To add a new storage class named COLD to the default-placement target, start by adding it to the zonegroup:
$ radosgw-admin zonegroup placement add \ --rgw-zonegroup default \ --placement-id default-placement \ --storage-class COLD
Then provide the zone placement info for that storage class:
$ radosgw-admin zone placement add \ --rgw-zone default \ --placement-id default-placement \ --storage-class COLD \ --data-pool default.rgw.cold.data \ --compression lz4
Create data pool with desired pg_num and data protection mecanism:
ceph osd pool create default.rgw.cold.data 32 replicated
Bucket Lifecycle Management
Specify a filter to transition all object to COLD Storage Class after 7 days.
Create S3 Lifecycle configuration file lifecycle-to-cold.xml
<LifecycleConfiguration>
<Rule>
<ID>Archive all object older than 7 days</ID>
<Filter>
<Prefix></Prefix>
</Filter>
<Status>Enabled</Status>
<Transition>
<Days>7</Days>
<StorageClass>COLD</StorageClass>
</Transition>
</Rule>
</LifecycleConfiguration>
Set lifecycle on bucket named my-bucket
$ s3cmd setlifecycle lifecycle-to-cold.xml s3://my-bucket s3://my-bucket/: Lifecycle Policy updated
Display lifecycle policy applied on bucket
$ s3cmd getlifecycle s3://my-bucket
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>Archive all object older than 7 days</ID>
<Prefix/>
<Status>Enabled</Status>
<Transition>
<Days>7</Days>
<StorageClass>COLD</StorageClass>
</Transition>
</Rule>
</LifecycleConfiguration>
$ s3cmd put /tmp/toto.txt s3://my-bucket/toto.txt upload: '/tmp/toto.txt' -> 's3://my-bucket/toto.txt' [1 of 1]
Objects in bucket my-bucket will transition from pool default.rgw.buckets.data to default.rgw.cold.data after 7 days.
To found out if an object has been moved to COLD storage class
$ s3cmd info s3://my-bucket/toto.txt s3://my-bucket/toto.txt (object): File size: 15 Last mod: Tue, 31 Mar 2020 12:02:56 GMT MIME type: text/plain Storage: COLD MD5 sum: 60e3aba64259bc73f47dd0ab4078f24e SSE: none Policy: none CORS: none ACL: Test user: FULL_CONTROL x-amz-meta-s3cmd-attrs: atime:1585656149/ctime:1585656149/gid:0/gname:root/md5:60e3aba64259bc73f47dd0ab4078f24e/mode:33188/mtime:1585656149/uid:0/uname:root
We can see objects increasing in pools with
ceph df
More lifecycle configuration examples are available on AWS documentation.
Related Articles
May 31st, 2022
HTML on documentation.suse.com – Shape it up!
Jul 14th, 2022
‘Contain’ your excitement: The s3gw is in town
Run SAP
SUSE for Public Cloud
Security
No comments yet