SLES12 SP5 system panic in tcp_retransmit_skb()

SUSE Linux Enterprise Server 12 SP5 (SLES12 SP5)

A SLES12 SP5 system with kernel 4.12.14-122.12-default or older may suffer an intermittent panic. The console output or kernel log will show the following warning messages:

[172193.273976] WARNING: CPU: 38 PID: 0 at ../net/ipv4/tcp_timer.c:434 tcp_retransmit_timer+0x985/0x9b0 
[172193.273977] Modules linked in: loop mmfs26(OEX) mmfslinux(OEX) tracedev(OEX) binfmt_misc nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache tcp_diag inet_diag scsi_transport_iscsi bonding iscsi_ibft  
iscsi_boot_sysfs rdma_ucm(OEX) ib_ucm(OEX) rdma_cm(OEX) iw_cm(OEX) configfs ib_ipoib(OEX) ib_cm(OEX) ib_umad(OEX) mlx5_ib(OEX) mlx5_core(OEX) mlxfw(OEX) mpt3sas raid_class mptctl mptbase msr intel_rapl sb_edac x86_pkg_temp_thermal intel_
powerclamp iTCO_wdt coretemp iTCO_vendor_support kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel pcbc nls_iso8859_1 nls_cp437 vfat fat cdc_ether aesni_intel ipmi_ssif usbnet aes_x86_64 joydev crypto_simd mii ses glue_helper cryp
td igb ioatdma enclosure ch lpc_ich pcspkr scsi_transport_sas mfd_core dca wmi ipmi_si ipmi_devintf
[172193.274036]  ipmi_msghandler pcc_cpufreq button sunrpc mlx4_en(OEX) mlx4_ib(OEX) ptp pps_core ib_uverbs(OEX) ib_core(OEX) ext4 crc16 jbd2 mbcache sd_mod hid_generic usbhid mgag200 crc32c_intel i2c_algo_bit qla2xxx drm_kms_helper sysc
opyarea sysfillrect sysimgblt fb_sys_fops ttm xhci_pci nvme_fc mlx4_core(OEX) xhci_hcd ehci_pci nvme_fabrics devlink ehci_hcd drm nvme_core mlx_compat(OEX) scsi_transport_fc usbcore drm_panel_orientation_quirks megaraid_sas(OEX) sg dm_mu
ltipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod efivarfs autofs4
[172193.274075] Supported: Yes, External
[172193.274079] CPU: 38 PID: 0 Comm: swapper/38 Tainted: G           OE      4.12.14-122.12-default #1 SLE12-SP5
[172193.274080] Hardware name: LENOVO x3950 X6 -[6241AC4]-/00WA086, BIOS -[A9E152FUS-4.71]- 12/05/2019
[172193.274081] task: ffff8b0a8152c0c0 task.stack: ffffa5dab142c000
[172193.274084] RIP: 0010:tcp_retransmit_timer+0x985/0x9b0
[172193.274085] RSP: 0018:ffff8bc6bf483e48 EFLAGS: 00010246
[172193.274086] RAX: 0000000000000001 RBX: ffff8bc680c08000 RCX: 000000000000001f
[172193.274087] RDX: 000000281784d847 RSI: 0000000000000004 RDI: ffff8bc680c08000
[172193.274088] RBP: ffff8bc680c08158 R08: 0000000000000004 R09: ffffecc2a5232d5f
[172193.274088] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000100
[172193.274089] R13: ffffffff831ca400 R14: ffff8bc680c08000 R15: 0000000000000000
[172193.274090] FS:  0000000000000000(0000) GS:ffff8bc6bf480000(0000) knlGS:0000000000000000
[172193.274091] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[172193.274092] CR2: 00007fec7b9606e0 CR3: 000003c4c400a003 CR4: 00000000001606e0
[172193.274093] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[172193.274093] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[172193.274094] Call Trace:
[172193.274099]  <IRQ>
[172193.274104]  ? tcp_write_timer_handler+0x240/0x240
[172193.274105]  tcp_write_timer_handler+0xc6/0x240
[172193.274106]  tcp_write_timer+0x69/0x80
[172193.274114]  call_timer_fn+0x32/0x140
[172193.274116]  run_timer_softirq+0x1d6/0x420
[172193.274127]  ? timerqueue_add+0x54/0x80
[172193.274129]  ? enqueue_hrtimer+0x38/0x90
[172193.274133]  __do_softirq+0xce/0x28b
[172193.274141]  irq_exit+0xdb/0xf0
[172193.274147]  smp_apic_timer_interrupt+0x3f/0x60
[172193.274150]  apic_timer_interrupt+0x8f/0xa0
[172193.274151]  </IRQ>
[172193.274154] RIP: 0010:mwait_idle+0x7b/0x1a0
[172193.274154] RSP: 0018:ffffa5dab142fed8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[172193.274155] RAX: 0000000000000000 RBX: ffff8b0a8152c0c0 RCX: 0000000000000000
[172193.274156] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[172193.274156] RBP: 0000000000000026 R08: 0000000000000004 R09: ffffecc2a5232d5f
[172193.274157] R10: 00000001028fba98 R11: ffff8bc1268b8880 R12: ffff8b0a8152c0c0
[172193.274158] R13: ffff8b0a8152c0c0 R14: 0000000000000000 R15: 0000000000000000
[172193.274166]  do_idle+0x160/0x1f0
[172193.274168]  cpu_startup_entry+0x5d/0x70
[172193.274174]  start_secondary+0x1a5/0x200
[172193.274180]  secondary_startup_64+0xa5/0xb0
[172193.274182] Code: ff ff 31 d2 44 89 f6 48 89 df e8 07 f1 ff ff 84 c0 0f 84 ca f9 ff ff 31 f6 e9 c8 f9 ff ff 84 c0 0f 84 ae f9 ff ff e9 b6 f9 ff ff <0f> 0b 66 0f 1f 84 00 00 00 00 00 e9 c5 f7 ff ff 48 8b 7b 60 48 

which then will be followed by the following stack trace:

[172193.274207] BUG: unable to handle kernel NULL pointer dereference at 0000000000000030 
[172193.274213] IP: tcp_retransmit_skb+0x5c/0xc0
[172193.274216] PGD 0 P4D 0  
[172193.274218] Oops: 0000 [#1] SMP PTI
[172193.274220] CPU: 38 PID: 0 Comm: swapper/38 Tainted: G        W  OE      4.12.14-122.12-default #1 SLE12-SP5
[172193.274221] Hardware name: LENOVO x3950 X6 -[6241AC4]-/00WA086, BIOS -[A9E152FUS-4.71]- 12/05/2019
[172193.274222] task: ffff8b0a8152c0c0 task.stack: ffffa5dab142c000
[172193.274223] RIP: 0010:tcp_retransmit_skb+0x5c/0xc0
[172193.274224] RSP: 0018:ffff8bc6bf483e28 EFLAGS: 00010246
[172193.274225] RAX: 00000000fffffff5 RBX: ffff8bc680c08000 RCX: 0000000000000001
[172193.274226] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8bc680c08000
[172193.274227] RBP: 0000000000000000 R08: 0000000016c00000 R09: ffffecc2a5232d5f
[172193.274228] R10: 0000000000000004 R11: 0000000000000000 R12: 00000000fffffff5
[172193.274229] R13: ffffffff831ca400 R14: ffff8bc680c08000 R15: 0000000000000000
[172193.274230] FS:  0000000000000000(0000) GS:ffff8bc6bf480000(0000) knlGS:0000000000000000
[172193.274231] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[172193.274232] CR2: 0000000000000030 CR3: 000003c4c400a003 CR4: 00000000001606e0
[172193.274233] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[172193.274234] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[172193.274235] Call Trace:
[172193.274236]  <IRQ>
[172193.274238]  tcp_retransmit_timer+0x3ff/0x9b0
[172193.274240]  ? tcp_write_timer_handler+0x240/0x240
[172193.274241]  tcp_write_timer_handler+0xc6/0x240
[172193.274243]  tcp_write_timer+0x69/0x80
[172193.274244]  call_timer_fn+0x32/0x140
[172193.274246]  run_timer_softirq+0x1d6/0x420
[172193.274248]  ? timerqueue_add+0x54/0x80
[172193.274250]  ? enqueue_hrtimer+0x38/0x90
[172193.274251]  __do_softirq+0xce/0x28b
[172193.274253]  irq_exit+0xdb/0xf0
[172193.274255]  smp_apic_timer_interrupt+0x3f/0x60
[172193.274257]  apic_timer_interrupt+0x8f/0xa0
[172193.274259]  </IRQ>
[172193.274260] RIP: 0010:mwait_idle+0x7b/0x1a0
[172193.274261] RSP: 0018:ffffa5dab142fed8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[172193.274262] RAX: 0000000000000000 RBX: ffff8b0a8152c0c0 RCX: 0000000000000000
[172193.274263] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[172193.274264] RBP: 0000000000000026 R08: 0000000000000004 R09: ffffecc2a5232d5f
[172193.274265] R10: 00000001028fba98 R11: ffff8bc1268b8880 R12: ffff8b0a8152c0c0
[172193.274265] R13: ffff8b0a8152c0c0 R14: 0000000000000000 R15: 0000000000000000

The most important part in this stack trace is the following line:

​​​​​​​RIP: 0010:tcp_retransmit_skb+0x5c/0xc0
​​​​​​​

​​​​​​​The numbers in the brackets [..] may vary as they are just the uptime in seconds since reboot.

This is a known problem and has been fixed with kernel 4.12.14-122.17.1 and newer maintenance update kernels.