RMT mirroring fails when /tmp runs out of disk space

This document (000021403) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 Repository Monitoring Tool (RMT)

Situation

During RMT mirroring operations, the filesystem hosting /tmp runs out of free disk space and so mirroring fails:

W, [2024-01-02T18:16:12.000122 #49915] WARN -- : The following errors occurred while mirroring:
W, [2024-01-02T18:16:12.000168 #49915] WARN -- : Repository 'SUSE-PackageHub-15-Standard-Pool’ (3059): Error while mirroring packages: No space left on device @ io_write - /tmp/rmt20240102-49915-1ng0dl.
W, [2024-01-02T18:16:12.000199 #49915] WARN -- : Repository 'SUSE-PackageHub-15-SPS-Backports-Pool’ (6113): Error while mirroring packages: No space left on device @ io_write - /tmp/rmt20240102-49915-yvfj0b.
W, [2024-01-02T18:16:12.000233 #49915] WARN -- : Mirroring completed with errors.
The command exited with errors.

Resolution

The workaround to resolve this issue is as follows:

1. make sure you do this on a filesystem with abundant free space. If you have already enough space at /var/lib/rmt/tmp it is suggested to use this one and skip directly to step 3 as it already have to correct permissions. Maybe you need to add additional space somewhere. Create a new directory at this location, for example:

mkdir /var/rmttmp

2. Set the proper filesystem permission for the new folder:

chown -R _rmt:nginx /var/rmttmp

3. Edit the RMT mirror service file to redirect temporary files to the location created above:

systemctl edit rmt-server-mirror.service

4. Add the statement between the commented section as instructed and use the chosen location.

[Service]
Environment=
Environment="LOG_TO_JOURNALD=1" "LANG=en" "TMPDIR=/var/rmttmp/"

5. Restart the services:

systemctl daemon-reload
systemctl restart rmt-server-mirror.service

On restarting the rmt-server-mirror.service, resume the mirroring process (rmt-cli mirror / rmt-cli mirror product xxxx) and /tmp will no longer be in use, but rather the new location created above will be used, and thus the RMT server will not run out of disk space.

Cause

When RMT mirrors repos from SCC, it writes large temporary files in /tmp. If /tmp is too small, or is part of the / (root) filesystem and that filesystem is small in size, the result can be /tmp running out of space quickly, causing the mirroring to fail.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.