Migration from SUSE Manager 4.1 to 4.3 fails

SUSE Manager 4.1

SUSE Manager 4.1 went out of support at the end of October 2022. Migrations to either 4.2 or, much better as in the case of this article, 4.3, should have happened before that date. The migration includes two parts, a product migration (4.1 to 4.3 in this case, which includes the migration of the underlaying operating system, from SLES 15 SP2 to SP4), and then a database migration, from postgresql 12 to 14.
The first part of the migration has usually less issues, once the python 2 module has been removed, as explained in the documentation:
https://documentation.suse.com/suma/4.3/en/suse-manager/installation-and-upgrade/server-y.html
However, finding errors in the script that runs the database migrations is possible, specially since the migration stopped being tested after the product end of life. The error messages can be diverse.

One usual advise is to have a roll back plan. In many infrastructures, SUSE Manager is a virtualized instance to which snapshots are taken. Even if snapshots are no real backups, they can be used as such with the proper strategy.
One approach that was useful to customers with problems in the migrations was to removed all unneeded products. A SUSE Manager instance should be dedicated and treated like an appliance. It should have nothing else, namely no legacy or package hub modules, not to mention desktop application modules.
The suggestion is to roll back to the snapshot taken before the migration, and remove the modules that are not needed, in the same way that the Python 2 module needs to be removed for the "zypper migration" command to work. The output of "zypper pd" should be similar to this:

# zypper pd
Refreshing service 'Basesystem_Module_15_SP2_x86_64'.
Refreshing service 'Python_2_Module_15_SP2_x86_64'.
Refreshing service 'SUSE_Manager_Server_4.1_x86_64'.
Refreshing service 'SUSE_Manager_Server_Module_4.1_x86_64'.
Refreshing service 'Server_Applications_Module_15_SP2_x86_64'.
Refreshing service 'Web_and_Scripting_Module_15_SP2_x86_64'.
Loading repository data...
Reading installed packages...
S  | Repository                                  | Internal Name                  | Name                                     | Version | Arch   | Is Base
---+---------------------------------------------+--------------------------------+------------------------------------------+---------+--------+--------
i+ | SLE-Module-Basesystem15-SP2-Pool            | sle-module-basesystem          | Basesystem Module                        | 15.2-0  | x86_64 | No
i+ | SLE-Module-Python2-15-SP2-Pool              | sle-module-python2             | Python 2 Module                          | 15.2-0  | x86_64 | No
i+ | SLE-Module-Python2-15-SP2-Updates           | sle-module-python2             | Python 2 Module                          | 15.2-0  | x86_64 | No
i+ | SLE-Module-Python2-15-SP2-Updates           | sle-module-python2             | Python 2 Module                          | 15.2-0  | x86_64 | No
i+ | SLE-Module-Python2-15-SP2-Updates           | sle-module-python2             | Python 2 Module                          | 15.2-0  | x86_64 | No
i+ | SLE-Module-SUSE-Manager-Server-4.1-Pool     | sle-module-suse-manager-server | SUSE Manager Server 4.1 Module           | 4.1-0   | x86_64 | No
i+ | SLE-Module-Server-Applications15-SP2-Pool   | sle-module-server-applications | Server Applications Module               | 15.2-0  | x86_64 | No
i+ | SLE-Module-Web-Scripting15-SP2-Pool         | sle-module-web-scripting       | Web and Scripting Module                 | 15.2-0  | x86_64 | No
i  | SLE-Product-SUSE-Manager-Server-4.1-Pool    | SUSE-Manager-Server            | SUSE Manager Server 4.1                  | 4.1-0   | x86_64 | Yes
   | SLE-Product-SUSE-Manager-Server-4.1-Updates | SLES-LTSS                      | SUSE Linux Enterprise Server 15 SP2 LTSS | 15.2-0  | x86_64 | No

In a nutshell, the only modules should be the ones in a default installation, namely:

• Basesystem module
• Python 2 module
• SUSE Manager Server module
• Server Applications module
• Web and Scripting module

Modules that should not be part of the system configuration:

• Legacy module
• Package hub

Modules can be removed with SUSEConnect:

SUSEConnect -d -p $name/$version/$arch

Here "$name", "$version" and "$arch" should be the actual values (for example, "PackageHub/15.2/x86_64"). 
Sometimes the above command might not be enough, and a manual removal of the release rpm package might be needed. 
Those can be fetched by grepping the string "release" from the list of packages:

# rpm -qa | grep -i release
sle-module-web-scripting-release-15.2-44.1.x86_64
SUSE-Manager-Server-release-4.1-44.2.x86_64
sle-module-python2-release-15.2-55.1.x86_64
sle-module-server-applications-release-15.2-44.1.x86_64
release-notes-susemanager-4.1.15-150200.3.80.1.x86_64
sle-module-basesystem-release-15.2-44.1.x86_64
sle-module-suse-manager-server-release-4.1-10.3.x86_64

And then the release package can be removed with zypper (notice that the version number and architecture is not added, only the package name):

zypper rm -u sle-module-XXX

The command should not try to remove a lot of packages. In case it tries to, or if there is any doubt, it is better to contact Support at this point.

After several iterations, the output of the "zypper pd" command should look very similar to the one previously shared.

At this point, it is very recommended to patch the system (a new snapshot could also be taken here). Before doing so, SUSE Manager services should be stopped:

spacewalk-service stop

As some repositories have been removed, some packages might need to be reinstalled from the remaining repositiories:

zypper patch --allow-vendor-change

There could still be some version dependency issues, that should be easy to handle with the interactive assistant from zypper. The command might need to be run twice, as if there were any patches for the zypper stack itself, those would need to be applied separately. In the second iteration, the rest of patches would be applied. Or easier to remember, the command can be run until it shows no more patches remaining.

After a reboot (if needed, when basic packages such as the kernel, systemd, glibc... have been updated), it is a good point to make a new snapshot and the migration can be run again, as explained in the documentation (see additional information section). The first step would be to remove the Python 2 module (otherwise the migration will not show the SUSE Manager 4.3 target), then the "zypper migration" command would be next, followed by a reboot and ending with the database migration.
 

Most of the times, some basic packages coming from Package Hub overwrite system packages, and this can affect the database migration script. However, as explained before, the causes can vary from case to case.

Please be advised, in case SUSE Manager has a different life cycle than the underlying SLES version, updates to the OS will be provided via the SUSE Manager update channels. Example: SUSE Manager 4.2 will receive updates for SLES 15 SP3 until October 2023 (end of life of the product).

SUSE Manager upgrade guide: https://documentation.suse.com/suma/4.3/en/suse-manager/installation-and-upgrade/server-y.html

SUSE lifecycle: https://www.suse.com/lifecycle