How to collect a stacktrace from the Rancher server process in Rancher v2.x

During troubleshooting it may be useful to collect a stacktrace from a running Rancher instance, and this article details the steps for creating one.

1. Open two shell sessions and source an admin kubeconfig for the Rancher server cluster in both.

2. In one of the shell sessions, determine the name (e.g. rancher-7b9f4764f5-rs2jx) of the Rancher leader pod per the steps in the article here.

3. In the same shell, start streaming the Rancher leader pod's logs to file, replacing with the pod name identified in step 2.:

   kubectl logs -n cattle-system --tail=-1 -f <LEADER_POD_NAME> > rancher.log

4. In the other shell, send a SIGABRT signal to the rancher process in the leader pod, with the command below, to trigger a stacktrace, replacing with the pod name identified in step 2.:

   kubectl -n cattle-system exec <LEADER_POD_NAME> -- bash -c 'kill -SIGABRT  $(pgrep -x rancher)'

5. 1. When the stacktrace generation is complete the Rancher leader pod will restart and the kubectl logs command in step 3. will exit. You can then provide the rancher.log file, containing the trace, to Rancher Support.

Validating the success of the stacktrace

If you want to validate the stacktrace was successfully generated, you can confirm the presence of the SIGABRT signal and goroutine traces in the rancher.log file as below:

SIGABRT: abort
PC=0x461781 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0x7370308, 0x80, 0x0, 0x0, 0xc000000000, 0x7ffdd633d8f0, 0x435863, 0xc000446848, 0x7ffdd633d910, 0x40b04f, ...)
    /usr/local/go/src/runtime/sys_linux_amd64.s:535 +0x21
runtime.futexsleep(0x7370308, 0x7ffd00000000, 0xffffffffffffffff)
    /usr/local/go/src/runtime/os_linux.go:44 +0x46
runtime.notesleep(0x7370308)
    /usr/local/go/src/runtime/lock_futex.go:151 +0x9f
runtime.stoplockedm()
    /usr/local/go/src/runtime/proc.go:2068 +0x88
runtime.schedule()
    /usr/local/go/src/runtime/proc.go:2469 +0x485
runtime.park_m(0xc000be4780)
    /usr/local/go/src/runtime/proc.go:2610 +0x9d
runtime.mcall(0x0)
    /usr/local/go/src/runtime/asm_amd64.s:318 +0x5b

This should end with a section similar to the following after the goroutine traces:

rax    0xca
rbx    0x73701c0
rcx    0x461783
rdx    0x0
rdi    0x7370308
rsi    0x80
rbp    0x7ffdd633d8d8
rsp    0x7ffdd633d890
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc000dde7e0
r15    0x0
rip    0x461781
rflags 0x286
cs     0x33
fs     0x0
gs     0x0

Rancher Documentation on Troubleshooting the Rancher Server Kubernetes Cluster