SES4: Deleting files within a Rados Block Device does not free up space on the cluster.

This document (7022066) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Enterprise Storage 4

Situation

Customer is using Rados Block Devices to store data. Customer moved data from one rbd device to another, expecting the first rbd device to release the freed space back to the SES cluster.

Resolution

There is good news and a few options to address this issue.

If the goal is to free up space, so osd's are not marked as full, then it may be best to add more disks to the cluster.
If space is available, create a new rbd device and copy the data to the new device, then delete the original rbd device. Deleting the rbd device will return the space to SES.
In some cases "fstrim" can return the "freed" space back to SES, assuming that the stack, filesystem, drivers, kernel, HBA, and hard drives support the functionality. fstrim should be run on the filesystem within the rbd block device.

Example:
List block devices:
rbd list PooName
Map the image to a new block device.
rbd map --pool PoolName ImageName
List all mapped devices:
rbd showmapped
Mount the device:
mount /dev/rbd0 /mnt
run fstrim:
fstrim /mnt

Mounting the file system with "-o discard" could provide same results with out having to run "fstrim".

Example: mount -o discard /dev/rbd0 /mnt

Cause

It is normal behavior for a rbd device to not return "freed" space back to the SES cluster when a files are deleted. Only the pointers to the files are removed. The blocks of data that made up the files, remain on the file system. The filesystem within the rbd device will recognize that there is free space, however SES is not aware that the files were deleted and can not distinguish deleted files from files that have not been deleted. This behavior is the same, if the operation were done on a normal hard drive with a posix file system.

Additional Information

https://bugzilla.suse.com/show_bug.cgi?id=1062152

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.