SLES12 SP3,4,5 or SLES15 SP1,2 VMware guests are crashing constantly

This document (7023505) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15 Service Pack 2 (SLES 15 SP2)
SUSE Linux Enterprise Server 15 Service Pack 1 (SLES 15 SP1)
SUSE Linux Enterprise Server 15 (SLES 15)
SUSE Linux Enterprise Server 12 Service Pack 5 (SLES 12 SP5)

SUSE Linux Enterprise Server 12 Service Pack 4 (SLES 12 SP4)
SUSE Linux Enterprise Server 12 Service Pack 3 (SLES 12 SP3)

Situation

After updating or installing a VMware guest with the latest kernel of the above mentioned SLES versions the VM is crashing on a regular basis. If a kernel dump has been captured the saved dmesg.txt in the crash directory will show the following output at the end.

[  244.099658] kernel BUG at ../drivers/net/vmxnet3/vmxnet3_drv.c:1441!
[  244.099683] invalid opcode: 0000 [#1] SMP 
[  244.099704] Modules linked in: af_packet iscsi_ibft iscsi_boot_sysfs vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel drbg ansi_cprng aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr mptctl vmw_vmci vmxnet3 i2c_piix4 shpchp ac fjes button processor ata_generic ext4 crc16 jbd2 mbcache sr_mod cdrom sd_mod ata_piix vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw ahci libahci drm mptspi scsi_transport_spi libata mptscsih mptbase floppy sg dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua scsi_mod autofs4
[  244.100066] Supported: Yes
[  244.100081] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.4.156-94.64-default #1
[  244.100108] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
[  244.100148] task: ffff880133170340 ti: ffff880133174000 task.ti: ffff880133174000
[  244.100176] RIP: 0010:[<ffffffffa039506d>]  [<ffffffffa039506d>] vmxnet3_rq_rx_complete+0xbdd/0xda0 [vmxnet3]
[  244.100217] RSP: 0018:ffff88013fd83e08  EFLAGS: 00010297
[  244.100238] RAX: 0000000000000040 RBX: ffff880036869128 RCX: ffff880036b7c700
[  244.100265] RDX: 0000000000000008 RSI: 0000000000000001 RDI: 0000000000000040
[  244.100291] RBP: ffff880036869040 R08: 0000000000000030 R09: 0000000000000000
[  244.100318] R10: ffff8800b3eb0f18 R11: 0000000000000000 R12: ffff880036b0ca10
[  244.100345] R13: ffff880036868840 R14: 00000000000000a1 R15: ffff880036a190d0
[  244.100372] FS:  0000000000000000(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
[  244.100402] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  244.100424] CR2: 00005654124d5000 CR3: 0000000036b30000 CR4: 0000000000360670
[  244.100494] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  244.100522] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  244.101280] Stack:
[  244.102075]  ffff880036869100 0000000036b28200 0000000000000000 0000000000000000
[  244.102862]  0000000000000002 ffffffff810c4f47 0027000100000040 ffffffff81f06b00
[  244.103636]  ffff880036869060 ffff880036868840 ffff880036869060 ffff880036869040
[  244.104403] Call Trace:
[  244.105183]  [<ffffffffa0395371>] vmxnet3_poll_rx_only+0x31/0xa0 [vmxnet3]
[  244.105967]  [<ffffffff81521fec>] net_rx_action+0x15c/0x370
[  244.106722]  [<ffffffff8108637c>] __do_softirq+0xec/0x300
[  244.107507]  [<ffffffff8108684a>] irq_exit+0xfa/0x110
[  244.108315]  [<ffffffff816201a1>] do_IRQ+0x51/0xe0
[  244.109078]  [<ffffffff8161d782>] common_interrupt+0xc2/0xc2
[  244.112340] DWARF2 unwinder stuck at ret_from_intr+0x0/0x1b
[  244.113071] 
[  244.113771] Leftover inexact backtrace:
               
[  244.115119]  <IRQ> 
[  244.115129]  <EOI> 
[  244.115765]  [<ffffffff81020e80>] ? idle_notifier_unregister+0x20/0x20
[  244.116428]  [<ffffffff81061272>] ? native_safe_halt+0x2/0x10
[  244.117071]  [<ffffffff81020e98>] ? default_idle+0x18/0xd0
[  244.117696]  [<ffffffff810c5e01>] ? cpu_startup_entry+0x2f1/0x390
[  244.118314]  [<ffffffff8104ed8a>] ? start_secondary+0x16a/0x190
[  244.118946] Code: 14 fc ff ff 45 29 c4 66 44 89 64 02 02 e9 53 fc ff ff 48 8b 05 a5 a3 a8 e1 e9 96 fd ff ff 66 83 78 10 08 0f 84 a0 fb ff ff 0f 0b <0f> 0b 45 84 c9 0f 85 be f7 ff ff 0f 0b 0f b6 95 50 01 00 00 c7 
[  244.120872] RIP  [<ffffffffa039506d>] vmxnet3_rq_rx_complete+0xbdd/0xda0 [vmxnet3]
[  244.121485]  RSP <ffff88013fd83e08>

Note: depending on what kernel is installed the line number in the following message may vary:
kernel BUG at ../drivers/net/vmxnet3/vmxnet3_drv.c:1441!

Resolution

VMware has released a knowledge base article at https://kb.vmware.com/s/article/2151480 which gives the following options:

Either upgrade to VMware ESXi 6.5 Update 1 or apply the following workaround:

Add the vmxnet3.rev.30 = FALSE parameter in the vmx file of virtual machine:
1. Power off the virtual machine.
2. Edit the vmx file and add the below parameter:
```
vmxnet3.rev.30 = FALSE
```
3. Power on the virtual machine.
If you do not want to power off the virtual machine, disable the receive data ring for each vmxnet3 vNIC on the VM by running this command:
```
ethtool -G ethX rx-mini 0
```
Note: Replace ethX with virtual machine interface name.

Note that even though the VMware article mentions that only Kernels greater 4.8 are affected this solution or workaround does apply to the SUSE Kernel too.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.