Move objects between Storage Classes using S3 Bucket Lifecycle Management

April 7, 2020 | By: Benoit Loriot

Let’s take advantage of S3 Bucket Lifecycle Management to move objects between storage classes so that they are stored cost effectively throughout their lifecycle.

We will describe the steps to transition objects between STANDARD and COLD Storage Classes after 7 days.

Adding a Storage Class

To add a new storage class named COLD to the default-placement target, start by adding it to the zonegroup:

$ radosgw-admin zonegroup placement add \
--rgw-zonegroup default \
--placement-id default-placement \
--storage-class COLD

Then provide the zone placement info for that storage class:

$ radosgw-admin zone placement add \
--rgw-zone default \
--placement-id default-placement \
--storage-class COLD \
--data-pool default.rgw.cold.data \
--compression lz4

Create data pool with desired pg_num and data protection mecanism:

ceph osd pool create default.rgw.cold.data 32 replicated

Bucket Lifecycle Management

Bucket Lifecycle Management Process

Specify a filter to transition all object to COLD Storage Class after 7 days.

Create S3 Lifecycle configuration file lifecycle-to-cold.xml

<LifecycleConfiguration>
  <Rule>
    <ID>Archive all object older than 7 days</ID>
    <Filter>
      <Prefix></Prefix>
    </Filter>
    <Status>Enabled</Status>
    <Transition>
      <Days>7</Days>
      <StorageClass>COLD</StorageClass>
    </Transition>
  </Rule>
</LifecycleConfiguration>

Set lifecycle on bucket named my-bucket

$ s3cmd setlifecycle lifecycle-to-cold.xml s3://my-bucket
s3://my-bucket/: Lifecycle Policy updated

Display lifecycle policy applied on bucket

$ s3cmd getlifecycle s3://my-bucket
<?xml version="1.0" ?>
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Rule>
    <ID>Archive all object older than 7 days</ID>
    <Prefix/>
    <Status>Enabled</Status>
    <Transition>
      <Days>7</Days>
      <StorageClass>COLD</StorageClass>
    </Transition>
  </Rule>
</LifecycleConfiguration>

Put an object

$ s3cmd put /tmp/toto.txt s3://my-bucket/toto.txt
upload: '/tmp/toto.txt' -> 's3://my-bucket/toto.txt' [1 of 1]

Objects in bucket my-bucket will transition from pool default.rgw.buckets.data to default.rgw.cold.data after 7 days.

To found out if an object has been moved to COLD storage class

$ s3cmd info s3://my-bucket/toto.txt
s3://my-bucket/toto.txt (object):
 File size: 15
 Last mod: Tue, 31 Mar 2020 12:02:56 GMT
 MIME type: text/plain
 Storage: COLD
 MD5 sum: 60e3aba64259bc73f47dd0ab4078f24e
 SSE: none
 Policy: none
 CORS: none
 ACL: Test user: FULL_CONTROL
 x-amz-meta-s3cmd-attrs: atime:1585656149/ctime:1585656149/gid:0/gname:root/md5:60e3aba64259bc73f47dd0ab4078f24e/mode:33188/mtime:1585656149/uid:0/uname:root

We can see objects increasing in pools with