Patching latest salt 3000/3002 minion throws an error regarding authentication to master

This document (000020625) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Manager 4.2
SUSE Manager 4.1
SUSE Linux Enterprise Server 15
SLE 12 Module: Advanced Systems Management Module
SLE Micro 5.X

Situation

The problem described in this article can affect any supported salt environment, no matter whether it is within a SUSE Manager environment or not, as salt can also be installed as a stand-alone component.
After patching salt minion(s) to the latest version, starting any of them throws an error similar to the following one:

2022-03-28 13:19:41,880 [salt.crypt       :743 ][ERROR   ][15942] Sign-in attempt failed: {'publish_port': 4505, 'pub_key': '-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA1KQQu6D9s5+dsTnksfrn\nLhsfpHhQ2yhBZshQXeBAOnMgQWBy/sovc/5I6FR/6J7xd5L83iYzKG7MBsMFbqLN\nFlDibTynNOSCpETD/GbzQ68JrxBG+vbAbOwf1IjqWnCRmhVKgsFj66ZZY2/IW7bb\nkwPvcFSChFTn1PovCNxYUZNEa6cL2/UUzRbiHGVJXscQnhyAUmAOr0Zu5+eqEkD1\nugshWOc3whEPzP6Rem1onLfXAU6KkXAZpEvjvpvBGwxzLvxeTsNux53jXBTZrArH\nrd1kCVss1dBwvEvIAyTiYwaDiviER5vtcF8xMcZBSFj3eXC8pdUnHytRn3qNn/pY\nLQIDAQAB\n-----END PUBLIC KEY-----\n', 'enc': 'pub', 'sig': "\x15\x03\xe5\xbb\xbf<\x90\x87t\xa0\xfb\x19\xa2\xa6\xda\x14\x9e\x89nlo\xbed\xeb\xbf\xb8j\xfeV\xa3\xe6\xffF\xf0y\xd2\xa7\x8ac\xbc\xb1\xae\x86\x8b\x0bi\x93\x97\xa2wqy\x9by<\xeel\xe1\x93\xb3\\+\xaa\x91\xf0\x84o\x92\xef\xc0\xc0j\x8f\xd3\x10o\xae\xe4\x8e\xeb\xda\xd8\r\xfdN\x15\xb2#\xda\r\x8e\xec\xab\xfd~>`` \xa1\x7fK\xb6\xe9v\x96\xa5\xb5\x1b\xb0\xabic\xc7;\x1d\xeb\xb0\xc9bj\xf0\xb3-cA\x04\xdd\x9a@\xdb`\xd1\xce\xe8[\xc2\xa4r\xea\xf98\xc1\xbf\xb3\x1emq\xb0$q\x88\xc9\x05e\xf4BKH\xec\x8a\x89\x0b\xfe\x89\x9d\xb0\xa7YV\x7f\xce\x06h\x9cY\x9f\x0c\x9b9A\x9c\xff\x9e\xf2\xbfsB\xc6u\x1b\xaa\x1dFz\x8bV\xf4N\x89\xaa\xfe:v<'\xc9m\x90J|[\xb5\x8e\xdb\xb5o\x87\xddn\xf68\xa2\xeae\xf9\xe1\xdc\xf6\xdc\x88\x99_\x8f\xe4\xd5\x99\xc07\xb0\xb2\xf1J\xa3\x8f\x10\xb9 \xf7\xee\xfbH\xa6\xf5\x06f", 'aes': "\x10\x7f\xa8\x92\x0cZ3B\x9ds\xf5]!\xb9\x06\x0c\x81c\xfd\xce\xdf-\r-y@\xc9`\x0f/`\x9fq\xd3\xf9\x99v\x13R\x8c\xb8\xae~\xb8eg\xff\xf9:\xaf\xbf/1\xca\x19\x95\xcc\xf8\xc5Y\xaa\x0c*\x7f4}\xb8\xa8\n5&\xbc\x91\xec\xf3\xc9K\xad\xf3\xcf?\xbd\x08\xdaolK\x1e\xa2\x0ezb\x97\xd7>\x83=:w\x94\xe2\xf3X\xc4\x8d\x9e\x1e8\x99.6\x8f\xc2p\x19q\xff\xc4\x1d\xbd\xbd\xa3c0r)\x9f?\xae\x8d1-\x17\xdd\xf7\x04\xd2\xc3\xa4\xae@\xabi'\x0eI\x02\xc8\x9eE\xe1\xc4\xc5\xcdG\xf3\xa8\xd3\x86A\xe3\xf2\xe5UPd\x92\x80\x01\x91\x17\x95\x10\xdf\xd6\x9c\x8eQNc\xbcg\x8eH\xd1\x9d\xff\x854\x81w\xf2\xc7\xfd\x97\x94\x8a/9i\x90,\xc3\xc5\n\xd6<\x0bc\xd5\x8c\x0f\xe0\x0f\xec\xc4\xa7oo5\x06\xea5\xf2D\xe1\xd4\xc8\x01\x88\xd4#\x8bncT\x98\x1b\xf1\xddC\xdb\x0b\xb2\xbdn\x02m\x81\xdc{\x18\xe2\x1a]\xee"}
2022-03-28 13:19:41,885 [salt.minion      :1056][ERROR   ][15942] Error while bringing up minion for multi-master. Is master at salt-master-server.tf.local responding?

The salt-minion service status can be checked by running the command:

salt-minion:~ # systemctl status salt-minion

It will be shown as started, but it is not able to authenticate with the master, which will show part of the previous errors.

If the salt versions are checked, for both master and minion, a mismatch will be seen.
For example, on the minion:

salt-minion:~ # rpm -q salt-minion
salt-minion-3002.2-150300.53.10.1.x86_64

This is the first version that introduced the authentication problem, which will exist if the salt-master is not at least on that version. In the future, the problem can also exist in higher versions of salt-minion, as long as the version in the salt-master is not at least the one that corrected the error.
The salt-master version can be checked on master by running following command:

salt-master:~ # rpm -q salt-master
salt-master-3002.2-53.4.1.x86_64

Resolution

This can be ensured by installing latest available version of salt-master. After patching salt-master, the service will be restarted automatically. Once the operation has succeeded, the authentication errors in the salt minion will disappear (no restart needed there either as the service was already correctly started).

For example, as of today, April 6th 2022

SUSE Manager 4.1 / SLES 15 SP2 salt-master:

salt-master:~ # rpm -q salt-master
salt-master-3002.2-150200.58.1.x86_64

And SUSE Manager 4.2 / SLES 15 SP3 salt-master:

salt-master:~ # rpm -q salt-master
salt-master-3002.2-150300.53.10.1.x86_64

Even in the future, with different versions of both master and minion (as long as the master already includes the patch that fixed the authentication problem), problem could be solved. It is a good practice that for any server/client service, the server is running the same or higher version than any of the clients, in order to avoid issues like this one.

Cause

The problem could have been avoided by patching the server (salt master) first, and only after that, the clients (minions). Salt requires that the version on master is newer, or equal to the ones on the minions, otherwise any minions with a newer version will fail to start properly.

Status

Reported to Engineering

Additional Information

The latest version of salt addresses several security vulnerabilities (CVEs). Customer should refer to the release notes and/or changelog for more detailed information.

https://www.suse.com/releasenotes/x86_64/SUSE-MANAGER/4.2/index.html#_version_4_2_5_1

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

Document ID:000020625
Creation Date: 28-Mar-2022
Modified Date:28-Apr-2022
- SUSE Linux Enterprise Server
- SUSE Linux Enterprise Server for SAP Applications
- SUSE Manager Server
- SLES 12 Module: Advanced System Management Module
- SUSE Linux Enterprise Micro