Disaster recovery of SMT

This document (7004986) is provided subject to the disclaimer at the end of this document.

Environment

Subscription Management Tool 1.0
Subscription Management Tool 1.1
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10 SP3
SUSE Linux Enterprise Server 10 SP2

Situation

The first section of this document explains what data to proactively back up in order to provide for a smooth recovery of an SMT server in case it for some reason should become unusable.
The second section describes how to recover by utilizing the information gathered in the first section.

NOTE:
The Novell Customer Center (NCC) and nu.novell.com mentioned below are not available anymore. SLE10 and SLE11 Systems still registered to the Novell Customer Center (NCC) can be migrated to the SUSE Customer Center (SCC) as outlined at: https://www.suse.com/support/kb/doc/?id=000019508

Resolution

Preparation / backup of relevant information.
A normal rule is to ensure to have a usable backup of the complete system.
Besides that it is recommended to create easily-accessible copies of the data mentioned in the following steps, which should all be performed while being authenticated as the root user.

To back up the server certificate and be able to create a new server certificate when it expires, the easiest way is to create a copy of /var/lib/CAM/*
Record the IP address, DNS and routing configuration of the server in a file.
Create a copy of the following :
- If special configuration of apache2 has been done, then make a copy of the modified files (e.g. /etc/apache2/default-server.conf)
- /etc/smt.conf
- /etc/smt.d/*
- /etc/zypp/credentials.d/NCCcredentials
Regularly create a backup of the SMT database.
Since the database is constantly updated with changes it is recommended to create a script that backs up the database. In the examples below the file is named smt-db-backup.sh
- Create a script in /usr/sbin/ that includes the following commands :
```
rcsmt stop
mysqldump -uroot -p<mysql-root-user-password> smt > /<some-directory>/smt-db-backup.sql
rcsmt start
```
- Set the appropriate permissions to the script – also to prevent normal users from being able to read it, since it contains the password of the MySQL root user :
- ```
# chmod 750 /usr/sbin/smt-db-backup.sh
```
- Include the job in the SMT framework by appending a line like the following to the cron table in /etc/smt.d/novell.com-smt:
  22 2 * * * root /usr/sbin/smt-db-backup.sh
- This will execute the script daily at 02:22
- Restart the cron daemon to pick up the new job :
```
# rccron restart
```
Depending on factors like :
- the amount of repository data mirrored
- backup capacity and costs
- internet bandwidth constraints
- requirements for SMT service restoration
it might or might not be feasible to back up the repository structure (repo/), which is located in the directory specified in the MirrorTo parameter in /etc/smt.conf.
By default this means /srv/www/htdocs/repo/ and everything below

Disaster recovery :

In case the SMT server needs to be reinstalled from scratch, the information saved during the preparation can be used to get the server back to operational state as quickly as possible.
Always recover to the same version of SMT that the backups were created on !
The execution order of the individual steps is key and should be performed in the sequentially as listed below.

Install SLES 10 (for SMT 1.0) or SLES 11 (SMT 1.1) and configure the same hostname IP address etc. as the server had before.
Restore the backup copy of the /etc/zypp/credentials.d/NCCcredentials file.
Register it against an update source and apply the current updates (reboot).
Restore the Certificate Authority and server certificate from the removable device :
- Overwrite /var/lib/CAM/ directory structure with the backup copy of the same directory – e.g. :
```
# cp -vr /media/disk/smt-backup/CAM/* /var/lib/CAM/
```
- Verify the server certificate has been restored correctly and export the server certificate :
  - Start YaST and open CA Management under Security and Users
  - Enter the CA
  - Verify that the correct server certificate is present and valid
  - Select it, and click on Export | Export as common server certificate
    Enter the certificate password, click OK and a confirmation that "Certificate has been written as common server certificate" should appear.
  - Exit the YaST module.
  - With the next restart of SMT the Certificate Authority is copied to the apache document root directory
Restore the repository structure
Since the uid of the smt user might be different, the permissions to
/etc/zypp/credentials.d/NCCcredentials must be set with
```
# setfacl -m u:smt:r /etc/zypp/credentials.d/NCCcredentials
```
Verify that the owner/group of /srv/www/htdocs/repo/ is smt.www
If not then set it recursively with
```
# chown -R smt.www /srv/www/htdocs/repo
```
Install SMT as per the documentation :
SMT 1.0

SMT 1.1
and complete the setup wizard.
Apply the current updates for SMT
Stop the SMT services :
```
# rcsmt stop
```
Restore and verify the backup of the SMT database :
- ```
# mysql -uroot -p smt < <path-and-name-of-backup-file>
```
  After entering the password it will silently load the backup file and return to the prompt.
- Enter the MySQL monitor and verify that the tables in the smt database are present :
```
# mysql -uroot -p smt
```
  Enter the password and when the monitor is loaded execute the command :
```
mysql> show tables;
```
  It should return the following output :

- Exit the monitor :
```
mysql> exit
```
Adapt the configuration files to the customizations in the backed up ones and start SMT with
```
# rcsmt start
```
Syncronize data between local database and the Novell Customer Center with
```
# smt-ncc-sync
```
Verify the catalogs/repositories enabled for mirroring with
```
# smt-catalogs -o
```
List the registered clients with
```
# smt-list-registrations
```
SMT 1.1 only : View the job queue with
```
# smt-job
```

Kick off a mirror job to fix up the database :

SMT 1.0 :

# smt-mirror -d -L /var/log/smt-mirror.log &

SMT 1.1 :

# smt-mirror -d -L /var/log/smt/smt-mirror.log &

Once the mirror job has completed the SMT server should be fully operational.
To verify client functionality perform a service refresh on a client and check that the "Last Contact" time-stamp for its entry of registered clients on the server gets updated.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.