Running "crash" to analyze dump data on SLES11

Crash is a powerful tool that will analyze core dumps after crashing, to help you with troubleshooting or forensics analysis.

Coredump files after an incident are stored in:

/var/crash/$DATE

There is a README file in this path with basic info of the core dump file:

sles-beta:/var/crash/2013-03-27-10:32 # cat README.txt
Kernel crashdump
----------------

Crash time : 2013-03-27 10:31 (-0600)
Kernel version : 2.6.32.12-0.7-default
Host : linux
Dump level : 0
Dump format : compressed

sles-beta:/var/crash/2013-03-27-10:32 #

The way to run crash is:

crash vmlinux vmcore

NOTE: crash will store vmlinux compressed, we need to uncompress first:

# gzip -d vmlinux-2.6.32.12-0.7-default.gz

Then we run crash and we may have this error:

crash: vmlinux-2.6.32.12-0.7-default: no debugging data available
crash: vmlinux-2.6.32.12-0.7-default.debug: debuginfo file not found

crash: either install the appropriate kernel debuginfo package, or
copy vmlinux-2.6.32.12-0.7-default.debug to this machine

Notice that we said that we need a package, lets see:

sles-beta:/var/crash/2013-03-27-10:32 # rpm -qa | grep debug
sles-beta:/var/crash/2013-03-27-10:32 #

Indeed, we have no debug package, logic tells us that would be no problem installing it, see:

# zypper search *debug*

Apparently these are not the packages we need. What’s wrong?

Let’s see the repo list:

# zypper lr

That’s the problem, we have not activated the necessary repos, according to our distribution proceed to activate.

In this case it is SLES11 SP2:

# zypper mr --enable nu_novell_com:SLE11-SP2-Debuginfo-Core

# zypper mr --enable nu_novell_com:SLE11-SP2-Debuginfo-Updates

Then refresh references and zypper repos:

# zypper ref -s

# zypper refresh

And search again:

# zypper search debug

Now we list several packages related to the keyword debug, be more specific using the version of our kernel:

# uname -r

NOTE: If we are on the same server that generated the crash, we use that version of it, if we are doing the analysis from another server we need to use the exact version that was built with. This is seen in the README.txt.

sles-beta:/var/crash/2013-03-27-10:32 # grep "version" README.txt
Kernel version : 2.6.32.12-0.7-default
sles-beta:/var/crash/2013-03-27-10:32 #

Search for the correct Kernel debuginfo package and install it:

# zypper search -s kernel-*-debuginfo*

In this example it is: kernel-default-debuginfo-2.6.32.12-0.7.1, after install we are ready to run crash again.

crash vmlinux-2.6.32.12-0.7-default vmcore

Remember, we decompressed vmlinux at first.

Voilá!

In the first screen we have useful information, process name, pid, status, cpu, etc etc etc.

Now we can analyze the core dump using ‘backtrace‘, ‘files‘, ‘ps‘, ‘log‘ etc. And do the analysis as long and deep as desired.

Happy debugging!