Updating salt ssh clients start to fail with: IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $

This document (000021352) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Manager Server 4.3

Situation

After some time working normally, suddenly for example when attempting to apply updates to certain salt ssh clients, this fails with an error, excerpt:

Error calling Salt: com.suse.salt.netapi.exception.SaltException: com.google.gson.JsonSyntaxException: java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at path $

Resolution

Remove the locally deployed salt bundle on the problem client by deleting the "/var/tmp/venv-salt-minion/" directory. After this, any action taken / executed against the client should re-deploy the salt bundle to the client.

Also consider adjusting the configuration for the "systemd-tmpfiles-clean.timer" to not cleanup files under "/var/tmp/".

Cause

The "systemd-tmpfiles-clean.timer" was configured to remove files under "/var/tmp" after a certain amount of time, resulting in a corrupt salt bundle installation on the client.

Additional Information

With salt ssh minions, the salt bundle is deployed by the salt-ssh pre-flight script to "/var/tmp/venv-salt-minion/". When the systemd cleanup timer runs and removes files older than the configured amount of time, this can corrupt the locally deployed salt bundle.

In this case the Red Hat minion had the file "/usr/lib/tmpfiles.d/tmp.conf" with the below configuration, that removed files older than 30 days from the "/var/tmp" directory, excerpt:

q /var/tmp 1777 root root 30d

To disable time based cleanup of the "/var/tmp" directory, using the previous example, adjust the line "/usr/lib/tmpfiles.d/tmp.conf" for example as follows:

q /var/tmp 1777 root root -

NOTE: The configuration files used by the "systemd-tmpfiles-clean.timer" may differ based on distribution. For example on newer SLES based clients the configuration for "/var/tmp" is located in "/usr/lib/tmpfiles.d/fs-var-tmp.conf" and by default does not have a time based cleanup configured for this directory.

For more information see "man 5 tmpfiles.d".

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.