Configuration hints for kdump via SSH

SUSE Linux Enterprise Server 11 SP3
SUSE Linux Enterprise Server 11 SP4
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 12 SP1

Storing a kernel crash dump via SSH fails

Prequisite

An SSH connection between the crashed machine and the target, which provides the storage, needs to be possible on the network level.

Server identification

For security reasons, both machines must be able to verify the identity of the other end of the connection.

The kdump facility (SSH client) identifies the target machine (SSH server) by the target's host key. The SSH protocol allows many different algorithms to authenticate the other end. Which one is used depends on the server and client capabilities and preferences. SSH servers are usually configured for more than one algorithm. Each algorithm has its own host key. The default keys are generated during installation and stored as /etc/ssh/ssh_host_*_key on the server.

However, SLES11 SP3 default does not automatically generate a host key for an algorithm that can be used by kdump. Kdump relies on libssh2, which only implements DSA and RSA. For example, an RSA host key can be created by running "ssh-keygen -t rsa -b 4096 -N "" /etc/ssh/ssh_host_rsa_key" as the user root.

To verify the identity of the target server, an SSH client must have a copy of the public part of the target server's host key. The copy is kept in a known_hosts file on the client. Kdump will search the key in ~root/.ssh/known_hosts when the kdump initrd is generated (and copy it into the initrd).

Client authentication

The SSH client (on the crashed system) can be identified by the server by different methods. The kdump package implements two: public key authentication, or password (but not "keyboard-interactive"). When using public key authentication to authenticate to the server, kdump uses the root user's private keys, stored in ~/.ssh/id_dsa or ~/.ssh/id_rsa on the client. The public key (~/.ssh/id_*.pub) must be copied to the target user's ~/.ssh/authorized_keys on the server.

Note that there is a difference between password authentication and keyboard-interactive authentication, although there may be no difference in how they appear when connecting manually. While the keyboard-interactive authentication is enabled by default, the password authentication is not. It can be enabled with "PasswordAuthentication yes" in /etc/ssh/sshd_config on the server. There is no universal way to automate the keyboard-interactive authentication method; the server may send arbitrary strings and expect arbitrary replies from the client.

Since SLES11 SP4 (and SLES12), kdump has switched its backend from libssh2 to an external ssh command (from the OpenSSH package). After this change, kdump can use ECDSA keys. On the other hand, it no longer provides any way to use the password authentication.

File transfer

There are two different file transfer methods with SSH: sending raw data through the SSH stdin/stdout channels to a remote command, or using the SFTP subsystem. Before SLES11 SP4 (and SLES12), kdump implemented only the SFTP method, but confusingly called it "ssh://" in the KDUMP_SAVEDIR configuration. More recent versions distinguish between "ssh://" and "sftp://".

The "ssh://" protocol in SLES11 SP4 (and SLES12) is implemented by running several commands on the target server ("mkdir", "cd", "dd"). This does not work if the destination user does not have a usable shell for security reasons (e.g. if the shell is /bin/false). This may break when using an SP3 configuration for SP4. To get the old behvior, "ssh://" can be replaced with "sftp://".

For SFTP transport, the SFTP subsystem needs to be enabled on the SSH server ("Subsystem sftp" setting in /etc/ssh/sshd_config). This is enabled by default on SLES.