Maintaining and Adding Storage to SUSE Multi-Linux Manager 5.x persistent container volumes

SUSE Multi-Linux Manager 5.0

SUSE Multi-Linux Manager (SMLM) 5.0 is a containerized system, running on top of a container host, and managing storage with this configuration is different than it has been in the past.  Maintaining storage now is simpler once you understand how it works. This TID is meant to demystify the process of expanding or segmenting your SUSE Multi-Linux Manager 5.0 storage. 

If you have run out of disk space in a monitored directory in the SMLM container environment you may get an alert and SUSE Multi-Linux Manager may have shut down, by design.  The directory disk space monitoring tool is explained in the documentation link below:

https://documentation.suse.com/suma/5.0/en/suse-manager/administration/space-management.html

This article can also help you to segment your SMLM storage to different block devices in your environment beyond or best practices.

The recommended persistent container volume configuration is to have a single device backing all the persistent container volumes and another that backs just the postgresql database data. This keeps things simple and easy to manage.  The best practice is to use our mgr-storage-server tool to setup persistent container volumes. To see an explanation of the mgr-storage-server tool “Additional Information” section below, or the deployment documentation. The ability to use more then the two backing block devices is possible. Below we will explain what is possible, pertaining to backing block devices for persistent container volumes, and how to expand capacity.

First, we need to understand that containers have persistent volumes that are defined when the container is created. Persistent container volumes are file system locations where data can be written in the container to be retained. If data is attempted to be written to a non-persistent volume location the data will not be retained. For SUSE Multi-Linux Manager 5.0 you will find a list of the persistent container volume paths on the container host and mount points within the container in our documentation, see link below:

https://documentation.suse.com/suma/5.0/en/suse-manager/installation-and-upgrade/container-management/persistent-container-volumes.html 

It is important to note that the base podman persistence container volume path on the container host and default for SUSE MLM is /var/lib/containers/storage/volumes/, this is where all the persistent data for the SUSE MLM container is stored.  This base storage path can be changed, not recommended, but for our purposes below we will assume the default path. We also need to understand that when we are setting up additional persistent container volumes, they will be set up on the container host and NOT from within the container running SUSE MLM.

As an example the below is the persistent container volume for what the SUSE Multi-Linux Manager container sees as the /var/spacewalk directory path. This is the path where all the synced rpm's for SUSE MLM channels are stored and is a common path that may need its own volume due there being a lot of data being synced here.  The default path for the persistent container volume, on the container host system is:

/var/lib/containers/storage/volumes/var-spacewalk

This is just one example of many persistent paths that can be seen in the table from the link above.

The initial step for following any of these scenarios below is to assure the SUSE Multi-Linux Manager container is NOT running on the container host.  Stop the container from the container host with:

mgradm stop

*Note the recommended filesystem for persistent storage volumes is XFS.

**Note NFS storage is not recommended.

Choose the scenario below that suits your situation

#1 Adding a block device to back a specific persistent container volume

#2 Cloud/Virtualization growing the current persistent container volume size when you can expand the backing device dynamically.  

#3 Bare Metal or other situations where you need to grow the current persistent container volume size but do NOT have the ability to do it dynamically.

Scenario #1 - Adding a block device to back a specific persistent container volume.

1. Attach the new block device to the container host system.
   Example -
       We will be mounting a device at /var/lib/containers/storage/volumes/var-spacewalk for what the container sees as /var/spacewalk

2. Format the new storage device from the container host.  mkfs.xfs $device
   Example -
       mkfs.xfs /dev/sdb

3. Make a temporary mount point in a read-write location of the filesystem (we will use /root).  Using the "mount" command you can find other read-write locations as well.
   Example -
     mkdir /root/tmpmnt

4. Temporarily mount the new device so data can be moved.
   Example -
       mount /dev/sdb /root/tmpmnt

5. Move the data to the new device.  The move command (mv) is preferred to preserve selinux context. 
   Example -
       mv /var/lib/containers/storage/volumes/var-spacewalk/* /root/tmpmnt

6. Unmount new device from temp mount point after data is moved.
   Example -
       umount /tmpmnt

7. Update the fstab to mount the new device at the persistent volume location.
   Example - edit /etc/fstab file
       UUID=234234-23424wser-324234-2342 /var/lib/containers/storage/volumes/var-spacewalk xfs defaults,nofail 1 2

8. Mount the new device.
   Example -
       mount /var/lib/containers/storage/volumes/var-spacewalk

9. Start the SMLM container from the container host and confirm function.
   Example -
       mgradm start

Scenario #2 - Cloud/Virtualization growing the current persistent container volume size when you can expand the backing device dynamically.

* Note - this procedure could be done with the SUSE MLM container running but we recommend stopping it as a precaution.

1. From your cloud provider or VM management console allocate additional disk space to the device backing the container volumes.
   Example -
       We will consider our recommended default setup where the /var/lib/containers/storage/volumes directory is backed by a separate block device on the container host which is /dev/vdb1. 

2. From the container host of the SMLM server you will need to check if the disk you resized is partitioned or not.  If the disk you just resized is partitioned then expand the partition to take up the additional space.  If it is not partitioned then continue to step c.
   Example -
       The /dev/vdb1 device is a partition so use fdisk or parted to expand the partition.

3. From the container host of the SMLM server you will need to grow the filesystem to occupy the additional space.
   Example -
       xfs_growfs -d /dev/vdb1

4. Start the SUSE MLM container from the container host.
   Example -
       mgradm start

5. Log into the terminal of the SMLM container, mgrctl term, and you should see additional space now available.

Scenario #3 - Bare Metal or other situations where you need to grow the current persistent container volume size but do NOT have the ability to do it dynamically.

1. To resize/grow the current persistent container volume, if it can not be done dynamically or is a physical disk, you will have to first attach/install the new block device on the system.
   Example -
       A device named /dev/sdc was added to replace /dev/sdb which is currently mounted at /var/lib/containers/storage/volumes.

2. Format the new block device with the xfs filesystem.
   Example -
       mkfs.xfs /dev/sdc

3. Make a temporary mount point in a read-write location of the filesystem (we will use /root).  Using the "mount" command you can find other read-write locations.
   Example -
     mkdir /root/tmpmnt

4. Temporarily mount the new storage device.
   Example -
       mount /dev/sdc /root/tmpmnt

5. Move the data from /var/lib/containers/storage/volumes to the temp mount that is being used to replace the current one.  A mv operation is preferred here to preserve selinux context.
   Example -
       mv /var/lib/containers/storage/volumes/* /root/tmpmnt

6. Unmount the temporary mount point for the new disk.
   Example -
       umount /root/tmpmnt

7. Unmount the current mounted disk that is backing your persistent storage location.
   Example -
       umount /var/lib/containers/storage/volumes

8. Edit the fstab to now mount the new disk at the desired volume location and remove/comment out the original.
   Example - edit /etc/fstab comment out old device and add new.
       ##old backing device
    # UUID=234234-23424wser-324234-2342 /var/lib/containers/storage/volumes/var-spacewalk xfs defaults,nofail 1 2
    ##new backing device
     UUID=1234-4321-das4987-4987 /var/lib/containers/storage/volumes/var-spacewalk xfs defaults,nofail 1 2

9. Mount the new device via fstab. 
   Example -
       mount  /var/lib/containers/storage/volumes 

10. Start the SMLM container. 
    Example -
        mgradm start

11. Confirm SMLM it is working as expected.

A better understanding of what the documented deployment command for persistence container volumes is doing could also be helpful.  When setting up persistence container volumes per the documentation we are presented with the below command:

mgr-storage-server <storage-disk-device> [<database-disk-device>]

The above can be used with or without the [<database-disk-device>] portion. If you use the mgr-storage-server command without [<database-disk-device>] then whatever block device you define for <storage-disk-device> will be formatted, added to the container hosts fstab file, current data synced to the new mount volume and the new device mounted at /var/lib/containers/storage/volumes/.  This new device will act as the entire backing device for the SUSE MLM persistent container volume including database data (ex. mgr-storage-server /dev/sdb). 

If you use both options then the device defined for <storage-disk-device> will still be formatted, added to the container hosts fstab file, data synced and mounted at /var/lib/containers/storage/volumes/ but the [<database-disk-device>] device will also be formatted, added to the fstab, data synced and mounted at /var/lib/containers/storage/volumes/var-pgsql to back only the data for the postgresql database (ex. Mgr-storage-server /dev/sdb /dev/sdc). 

The mgr-storage-server command should not be run while the SMLM container is running.  Stop the container with “mgradm stop” before using it. The mgr-storage-server command can only be used one time and if any changes are needed to persistent container volumes after the command is initially run they will have to follow a similar manual process as discussed in the Resolution section above.