Fiber attached Multipathed volume mounts as read only

This document (7001766) is provided subject to the disclaimer at the end of this document.

Environment

SUSE Linux Enterprise Server 10 Service Pack 1

Situation

Kernel 2.6.16.46-0.12-smp
NETAPP Filer Storage (model unknown)
QLogic Corp. ISP2432-based 4Gb Fibre Channel

One lun is Fibre Channel attached to a NETAPP filer. An ext3 file system was created via LVM and mounted.

The file system appears to be fine and files can be touched/modified or removed without issues. A few minutes later, attempts to remove/modify/create files fail because the volume is now read-only.

The system log shows:

Oct 1 17:14:48 server1 kernel: EXT3-fs error (device dm-10): ext3_free_blocks_sb: bit already cleared for block 225281
Oct 1 17:14:48 server1 kernel: Aborting journal on device dm-10.
Oct 1 17:14:48 server1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
Oct 1 17:14:48 server1 kernel: EXT3-fs error (device dm-10) in ext3_truncate: Journal has aborted
Oct 1 17:14:48 server1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
Oct 1 17:14:48 server1 kernel: EXT3-fs error (device dm-10) in ext3_orphan_del: Journal has aborted
Oct 1 17:14:48 server1 kernel: EXT3-fs error (device dm-10) in ext3_reserve_inode_write: Journal has aborted
Oct 1 17:14:48 server1 kernel: EXT3-fs error (device dm-10) in ext3_delete_inode: Journal has aborted
Oct 1 17:14:56 server1 kernel: ext3_abort called.

Resolution

NETAPP has specific config for the multipath.conf.

Original multipath.conf

# /etc/multipath.conf
defaults {
user_friendly_names yes
}
devnode_blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^cciss!c[0-9]d[0-9]*"
}
devices {
device {
vendor "NETAPP "
product "LUN"
path_grouping_policy group_by_prio
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/opt/netapp/santools/mpath_prio_ontap /dev/%n"
features "1 queue_if_no_path"
path_checker readsector0
failback immediate
}
}

Netapp modified multipath.conf (changes in bold)

defaults {
user_friendly_names no
}
#
devnode_blacklist {
devnode"^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode"^cciss!c[0-9]d[0-9]*"
}

devices {
device {
vendor "NETAPP"
product "LUN"
path_grouping_policy group_by_prio
hardware_handler "0"
rr_weight uniform
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
prio_callout "/sbin/mpath_prio_ontap /dev/%n"
features "1 queue_if_no_path"
path_checker directio
failback immediate
rr_min_io 128
}
}

Additional Information

Related issues

Pre-service pack 1 installations can be affected by an issue with similar symptoms. That issue is covered in TID 3735930 - Filesystem goes read-only without the underlying storage reporting errors.

VMware guest installations can also be affected by an issue with similar symptoms. That issue is covered in TID 7005154 - Filesystem goes read-only in VMware.

Disclaimer

This Support Knowledgebase provides a valuable tool for SUSE customers and parties interested in our products and solutions to acquire information, ideas and learn from one another. Materials are provided for informational, personal or non-commercial use within your organization and are presented "AS IS" WITHOUT WARRANTY OF ANY KIND.