Enabling SCSI verbose logging and HBA verbose logging for analysis

This document (7007138) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 15
SUSE Linux Enterprise Server 12
SUSE Linux Enterprise Server 11
SUSE Linux Enterprise Server 10

Situation

A system is experiencing problems with connected SCSI devices. To analyze the issue more verbose output is needed.

Resolution

Please note: Enabling verbose SCSI logging will cause a huge amount of data written to log files. Depending on your system design it may happen that disk space gets filled up very quick.
As soon as all information was gathered, please revert the changes and disable verbose SCSI logging again.

There are 10 different categories of which logging verbosity can be set from 0 (off) to 7 (most verbose).

Categories Shift

1)  SCSI_LOG_ERROR                 0

2)  SCSI_LOG_TIMEOUT               3

3)  SCSI_LOG_SCAN                  6

4)  SCSI_LOG_MLQUEUE               9

5)  SCSI_LOG_MLCOMPLETE            12

6)  SCSI_LOG_LLQUEUE               15

7)  SCSI_LOG_LLCOMPLETE            18

8)  SCSI_LOG_HLQUEUE               21

9)  SCSI_LOG_HLCOMPLETE            24

10) SCSI_LOG_IOCTL 27

In order to set verbosity level of each category, select verbosity of each 0-7 and convert this number into binary according to this table:

0 000

1  001

2  010

3  011

4  100

5  101

6  110

7  111

Now take the selected verbosity of categories from 10) through 1) and calculate binary numbers together.

For example to raise verbosity:

Category IOCTL HLCOMPLETE HLQUEUE LLCOMPLETE LLQUEUE MLCOMPLETE MLQUEUE SCAN TIMEOUT ERROR

Decimal  3     0          0       0          0       0          0       3    6       1

Binary   011   000        000     000        000     000        000     011  110     001

Convert binary 011000000000000000000011110001 to hexadecimal by converting groups of four digits.

In this case:

01 1000 0000 0000 0000 0000 1111 0001

1   8    0    0    0    0    F    1

To set logging verbosity until next reboot, use command:

sysctl -w dev.scsi.logging_level=0x180000F1 or echo 0x180000F1 > /proc/sys/dev/scsi/logging_level

to set scsi logging which will be valid across reboots as well.

1. Insert dev.scsi.logging_level = 0x180000F1 into /etc/sysctl.conf or edit to desired log level

2. Run sysctl -p to make this change effective immediately

Instead of using hexadecimal numbers, it is also possible to enter the referring decimal numbers.
A useful setting of the logging level without being buried in logging details is ERROR=3, SCAN=3, MLQUEUE=2, MLCOMPLETE=2, which evaluates to:

echo 9411 > /proc/sys/dev/scsi/logging_level

The process to enable verbose logging on HBAs is as follows:

In case of Emulex cards, the kernel uses module called lpfc.ko:

1) Identify parameter to make verbose logging possible

modinfo lpfc | grep log

lpfc_log_verbose:Verbose logging bit-mask (int)

2) Add this parameter to module parameters. Edit /etc/modprobe.conf.local and add

options lpfc lpfc_log_verbose=1

For SLE10 and SLE11 the proper lpfc_log_verbose value is 1
For SLE12 and SLE15 the proper lpfc_log_verbose value is 0xf45cdf

3) Run mkinitrd, to apply this parameter during initrd phase of booting server

4) Reboot server, to see verbose logging of this module

In case of Qlogic cards, insert this line into /etc/modprobe.conf.local:

options qla2xxx ql2xextended_error_logging=1

To find which module storage the controller uses run:

hwinfo --storage-ctrl | grep Driver:

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.

Document ID:7007138
Creation Date: 03-Nov-2010
Modified Date:16-Nov-2023
- SUSE Linux Enterprise Desktop
- SUSE Linux Enterprise Point of Service
- SUSE Linux Enterprise Real Time
- SUSE Linux Enterprise Server