The holy cluster. Part 1 ISCSI Setup

Since I need to support quite some customers using High Availability Clustering I was quite often in the need to of having a test lab where I can try re-producing customer issues. Sometimes even trying to understand customer issues as well ;-).

So I thought hey there must be a complete guide A-Z on how to set things up. Yes there is very nice and informative documentation but not a complete guide from A-Z which also has all the tiny bit’s and pieces that are needed for such a test cluster setup like shared storage. How do I set it up? What can I use? Do I really need a SAN storage in the backend to get things running? So that’s the reason I wrote this blog post now.

The god answer to the above questions is: Quite easy setting things up. You can use e.g. your kvm server. NO you don’t need a real SAN storage in the backend. Even though it would be nice to have one wouldn’t it ;-). Anyway Don’t wanna hold you up! So let’s start with part one of the setup ISCSI LIO Server for shared storage.

1. Prerequisites:

I’ve listed my lab environment here so feel free to adjust it to your needs.

1 x SLES 12 SP2 Server running kvm + iscsi LIO server
Free disk space for an LVM Volume Group and for two logical lvm volumes
– one for a sbd 5 MB
– one for a share ~20GB (will be needed in Part 3 later)
Ideal would be if you also do have at least a DNS server

2 x SLES 12 SP2 Server’s as cluster nodes + iscsi initiator

For the rest of this guide please also note the following naming:
ISCSI LIO Server: skinner.simpsons.gov

Cluster Nodes: bart.simpsons.gov
maggy.simpsons.gov

2. ISCSI LVM preparations

I did create a new LVM Volume group named cluster via Yast:

Yast2 > System > Partinioner > Volume Management > Add >Volume Group

After the Volume Group has been created we need to add two Logical Volumes here one for the sbd device and one for the shared storage for the cluster.

To be able to identify them later on easier I named the first volume sbd and the second shared.

The sbd volume only needs to be 5 MB in size.

Please make sure to choose Raw Volume (unformatted)

Again do not format or mount the partition and just finish

Repeat the above for the shared volume as well and give it the rest of the space left in the volume group. When all is done the following should be shown when executing lvs on the console as root.

skinner:~ # lvs
LV  VG  Attr  LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
sbd    cluster -wi-a-----  8,00m
shared cluster -wi-a----- 20,49g

3. Configuring the ISCSI LIO Server with targetcli

Please make sure the following packages are installed:

zypper in yast2-iscsi-lio-server targetcli-fb open-iscsi

When the above packages are installed start the targetcli console to begin with the configuration.

skinner:~ # targetcli
targetcli shell version 2.1.fb43
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

Executing ls should show the following output:

/> ls
o- / ………………………………………………………………………….. […]
o- backstores ………………………………………………………………….. […]
| o- block ……………………………………………………….. [Storage Objects: 0]
| o- fileio ………………………………………………………. [Storage Objects: 0]
| o- pscsi ……………………………………………………….. [Storage Objects: 0]
| o- ramdisk ……………………………………………………… [Storage Objects: 0]
o- iscsi ………………………………………………………………… [Targets: 0]
o- loopback ……………………………………………………………… [Targets: 0]
o- vhost ………………………………………………………………… [Targets: 0]
o- xen_pvscsi ……………………………………………………………. [Targets: 0]
/>

Change into the backstores/block/ directory and create the corresponding block devices for the previously created logical lvm volumes:

/> cd backstores/block/

/backstores/block> create sbd /dev/cluster/sbd
/backstores/block> create shared /dev/cluster/shared

Afterwards the config should look like this:

/backstores/block> cd /
/> ls
o- / ………………………………………………………………………….. […]
o- backstores ………………………………………………………………….. […]
| o- block ……………………………………………………….. [Storage Objects: 2]
| | o- sbd …………………. [/dev/cluster/sbd (8.0MiB) write-thru deactivated]
| | o- shared ………. [/dev/cluster/shared (20.5GiB) write-thru deactivated]
| o- fileio ………………………………………………………. [Storage Objects: 0]
| o- pscsi ……………………………………………………….. [Storage Objects: 0]
| o- ramdisk ……………………………………………………… [Storage Objects: 0]
o- iscsi ………………………………………………………………… [Targets: 0]
o- loopback ……………………………………………………………… [Targets: 0]
o- vhost ………………………………………………………………… [Targets: 0]
o- xen_pvscsi ……………………………………………………………. [Targets: 0]
/>

Change into the iscsi directory to create the iscsi target:

/> cd iscsi
/iscsi> create
Created target iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
/iscsi>

Afterwards the config should look like this:

/iscsi> ls /
o- / ………………………………………………………………………….. […]
o- backstores ………………………………………………………………….. […]
| o- block ……………………………………………………….. [Storage Objects: 2]
| | o- sbd …………………. [/dev/cluster/sbd (8.0MiB) write-thru deactivated]
| | o- shared ………. [/dev/cluster/shared (20.5GiB) write-thru deactivated]
| o- fileio ………………………………………………………. [Storage Objects: 0]
| o- pscsi ……………………………………………………….. [Storage Objects: 0]
| o- ramdisk ……………………………………………………… [Storage Objects: 0]
o- iscsi ………………………………………………………………… [Targets: 1]
| o- iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d . [TPGs: 1]
|   o- tpg1 …………………………………………………….. [no-gen-acls, no-auth]
|     o- acls ………………………………………………………………. [ACLs: 0]
|     o- luns ………………………………………………………………. [LUNs: 0]
|     o- portals …………………………………………………………. [Portals: 1]
|       o- 0.0.0.0:3260 ………………………………………………………….. [OK]
o- loopback ……………………………………………………………… [Targets: 0]
o- vhost ………………………………………………………………… [Targets: 0]
o- xen_pvscsi ……………………………………………………………. [Targets: 0]
/iscsi>

Change into the target directory and then into the tpg1 and then into the luns directory to create the corresponding luns:

/iscsi> cd iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d/tpg1/

/iscsi/iqn.20...d1fbfce8/tpg1> cd luns
/iscsi/iqn.20...ce8/tpg1/luns> create /backstores/block/sbd
Created LUN 0.
/iscsi/iqn.20...ce8/tpg1/luns> create /backstores/block/shared
Created LUN 1.
/iscsi/iqn.20...ce8/tpg1/luns>

Afterwards the config should look like this:

/iscsi/iqn.20…ce8/tpg1/luns> ls /
o- / ………………………………………………………………………….. […]
o- backstores ………………………………………………………………….. […]
| o- block ……………………………………………………….. [Storage Objects: 2]
| | o- sbd …………………….. [/dev/cluster/sbd (8.0MiB) write-thru activated]
| | o- shared ………….. [/dev/cluster/shared (20.5GiB) write-thru activated]
| o- fileio ………………………………………………………. [Storage Objects: 0]
| o- pscsi ……………………………………………………….. [Storage Objects: 0]
| o- ramdisk ……………………………………………………… [Storage Objects: 0]
o- iscsi ………………………………………………………………… [Targets: 1]
| o- iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d . [TPGs: 1]
|   o- tpg1 …………………………………………………….. [no-gen-acls, no-auth]
|     o- acls ………………………………………………………………. [ACLs: 0]
|     o- luns ………………………………………………………………. [LUNs: 2]
|     | o- lun0 ………………………………………….. [block/sbd (/dev/cluster/sbd)]
|     | o- lun1 …………………………………. [block/shared (/dev/cluster/shared)]
|     o- portals …………………………………………………………. [Portals: 1]
|       o- 0.0.0.0:3260 ………………………………………………………….. [OK]
o- loopback ……………………………………………………………… [Targets: 0]
o- vhost ………………………………………………………………… [Targets: 0]
o- xen_pvscsi ……………………………………………………………. [Targets: 0]
/iscsi/iqn.20…ce8/tpg1/luns>

Get the default initiator name from another console:

skinner:~ # grep -v ^# /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1996-04.de.suse:01:a8c2aba2b371

Create the corresponding ACL’s modifying the initiator name for each of the cluster nodes as follows:

/iscsi/iqn.20...ce8/tpg1/acls> create iqn.1996-04.de.suse:bart.simpsons.gov
Created Node ACL for iqn.1996-04.de.suse:bart.simpsons.gov
Created mapped LUN 1.
Created mapped LUN 0.
/iscsi/iqn.20...ce8/tpg1/acls> create iqn.1996-04.de.suse:maggy.simpsons.gov
Created Node ACL for iqn.1996-04.de.suse:maggy.simpsons.gov
Created mapped LUN 1.
Created mapped LUN 0.

Afterwards the config should look like this:

/iscsi/iqn.20…ce8/tpg1/acls> ls /
o- / …………………………………………………………………………………………….. […]
o- backstores ……………………………………………………………………………… […]
| o- block ………………………………………………………………. [Storage Objects: 2]
| | o- sbd ………………. [/dev/cluster/sbd (8.0MiB) write-thru activated]
| | o- shared ………….. [/dev/cluster/shared (20.5GiB) write-thru activated]
| o- fileio ……………………………………………………………….. [Storage Objects: 0]
| o- pscsi ………………………………………………………………. [Storage Objects: 0]
| o- ramdisk ……………………………………………………………. [Storage Objects: 0]
o- iscsi ………………………………………………………………………….. [Targets: 1]
| o- iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d . [TPGs: 1]
|   o- tpg1 ………………………………………………………….. [no-gen-acls, no-auth]
|     o- acls ……………………………………………………………………………. [ACLs: 2]
|     | o- iqn.1996-04.de.suse:bart.simpsons.gov …………. [Mapped LUNs: 2]
|     | | o- mapped_lun0 ………………………………………… [lun0 block/sbd (rw)]
|     | | o- mapped_lun1 …………………………………… [lun1 block/shared (rw)]
|     | o- iqn.1996-04.de.suse:maggy.simpsons.gov …….. [Mapped LUNs: 2]
|     |   o- mapped_lun0 ………………………………………… [lun0 block/sbd (rw)]
|     |   o- mapped_lun1 ………………………………… [lun1 block/shared (rw)]
|     o- luns ………………………………………………………………………….. [LUNs: 2]
|     | o- lun0 ……………………………………….. [block/sbd (/dev/cluster/sbd)]
|     | o- lun1 ………………………………… [block/shared (/dev/cluster/shared)]
|     o- portals …………………………………………………………………….. [Portals: 1]
|       o- 0.0.0.0:3260 ……………………………………………………………… [OK]
o- loopback ……………………………………………………………………….. [Targets: 0]
o- vhost …………………………………………………………………………. [Targets: 0]
o- xen_pvscsi ……………………………………………………………………… [Targets: 0]
/iscsi/iqn.20…ce8/tpg1/acls>

Change into the portals directory to delete the default listener and to configure the new listener and port for the ISCSI LIO Server:

/iscsi/iqn.20...ce8/tpg1/acls> cd ../portals/
/iscsi/iqn.20.../tpg1/portals> ls
o- portals ..................................... [Portals: 1]
o- 0.0.0.0:3260 ........................................ [OK]
/iscsi/iqn.20.../tpg1/portals> delete 0.0.0.0 3260
Deleted network portal 0.0.0.0:3260
/iscsi/iqn.20.../tpg1/portals> create 192.168.2.235 3260
Using default IP port 3260
Created network portal 192.168.2.235:3260.

Afterwards the config should look like this:

/iscsi/iqn.20…/tpg1/portals> ls /
o- / ………………………………………………………………………………………………… […]
o- backstores …………………………………………………………………………………. […]
| o- block ……………………………………………………………….. [Storage Objects: 2]
| | o- sbd …………………….. [/dev/cluster/sbd (8.0MiB) write-thru activated]
| | o- shared …………. [/dev/cluster/shared (20.5GiB) write-thru activated]
| o- fileio ………………………………………………………………… [Storage Objects: 0]
| o- pscsi ……………………………………………………………….. [Storage Objects: 0]
| o- ramdisk ………………………………………………………….. [Storage Objects: 0]
o- iscsi …………………………………………………………………………….. [Targets: 1]
| o- iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d . [TPGs: 1]
|   o- tpg1 ……………………………………………………………. [no-gen-acls, no-auth]
|     o- acls ………………………………………………………………………………. [ACLs: 2]
|     | o- iqn.1996-04.de.suse:bart.simpsons.gov …………. [Mapped LUNs: 2]
|     | | o- mapped_lun0 ………………………………………… [lun0 block/sbd (rw)]
|     | | o- mapped_lun1 …………………………………. [lun1 block/shared (rw)]
|     | o- iqn.1996-04.de.suse:maggy.simpsons.gov …….. [Mapped LUNs: 2]
|     |   o- mapped_lun0 ……………………………………. [lun0 block/sbd (rw)]
|     |   o- mapped_lun1 …………………………………… [lun1 block/shared (rw)]
|     o- luns …………………………………………………………………………. [LUNs: 2]
|     | o- lun0 ……………………………………… [block/sbd (/dev/cluster/sbd)]
|     | o- lun1 …………………………………. [block/shared (/dev/cluster/shared)]
|     o- portals ……………………………………………………………………. [Portals: 1]
|       o- 192.168.2.235:3260 …………………………………………………….. [OK]
o- loopback …………………………………………………………………… [Targets: 0]
o- vhost ……………………………………………………………………………. [Targets: 0]
o- xen_pvscsi …………………………………………………………………… [Targets: 0]
/iscsi/iqn.20…/tpg1/portals>

For better handling we will now disable authentication (I do not recommend this for production environments!):

/iscsi/iqn.20.../tpg1/portals> cd ../
/iscsi/iqn.20...d1fbfce8/tpg1> set attribute authentication=0
Parameter authentication is now '0'.

Verify if the authentication is really turned off:

/iscsi/iqn.20...d1fbfce8/tpg1> get attribute

The result should look like this:

ATTRIBUTE CONFIG GROUP
======================
authentication=0
—————-
If set to 1, enforce authentication for this TPG.

cache_dynamic_acls=0
——————–
……

/iscsi/iqn.20…d1fbfce8/tpg1>

Last but not least change back to the / directory, save the configuration and exit the targetcli console.

/iscsi/iqn.20...d1fbfce8/tpg1> cd /
/> saveconfig
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json
/> exit

4. Enable needed services,configure initiators and test the configuration

Enable the targetcli, iscsid and target services for autostart on boot:

skinner:~ # systemctl enable targetcli.service

skinner:~ # systemctl enable iscsid.service

skinner:~ # systemctl enable target.service

Discover the iscsi target on the server

skinner:~ # iscsiadm -m discovery -t sendtargets -p skinner.simpsons.gov
192.168.2.235:3260,1 iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d

Edit the initiatorname.iscsi file to reflect the corresponding ACL names chosen above when creating the ACL’s for the two nodes:

skinner:~ # vi /etc/iscsi/initiatorname.iscsi
##
## /etc/iscsi/iscsi.initiatorname
##
## Default iSCSI Initiatorname.
##
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
#InitiatorName=iqn.1996-04.de.suse:01:a8c2aba2b371
InitiatorName=iqn.1996-04.de.suse:bart.simpsons.gov
InitiatorName=iqn.1996-04.de.suse:maggy.simpsons.gov

Afterwards restart the iscsid service to make the changes above effective:

skinner:~ # systemctl restart iscsid

Connect to the iscsi target on the server itself (also I would not recommend on a productions system but to test if everything works before configuring the clients it is just fine):

skinner:~ # iscsiadm -m node -p skinner.simpsons.gov --login
Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d, portal: 192.168.2.235,3260] (multiple)
Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d, portal: 192.168.2.235,3260] successful.

After successful login verify if the two luns are mapped once by path and once by id:

skinner:~ # ll /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root  9 Jul 10 17:55 ip-192.168.2.235:3260-iscsi-iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d-lun-0 -> ../../sdc
lrwxrwxrwx 1 root root  9 Jul 10 17:55 ip-192.168.2.235:3260-iscsi-iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d-lun-1 -> ../../sdd
....

skinner:~ # ll /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 Jul 10 17:55 scsi-36001405b17796fe35ea4f19917a2b53f -> ../../sdd
lrwxrwxrwx 1 root root  9 Jul 10 17:55 scsi-36001405c746d5e5b05c479eb99d12d97 -> ../../sdc
....

5. Configure the client’s to connect to the iscsi target

On the iscsi initiator/iscsi clients (bart.simpsons.gov and maggy.simpsons.gov) install and enable the following services:

zypper in open-iscsi yast2-iscsi-client

systemctl enable iscsid iscsi

systemctl start iscsid iscsi

Discover the iscsi target:

bart:~ # iscsiadm -m discovery -t sendtargets -p skinner.simpsons.gov
192.168.2.235:3260,1 iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d

Edit the initiatorname.iscsi file to reflect the corresponding initiator names defined for the nodes on the ISCSI LIO Server:

bart:~ # vi /etc/iscsi/initiatorname.iscsi
##
## /etc/iscsi/iscsi.initiatorname
##
## Default iSCSI Initiatorname.
##
## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1996-04.de.suse:bart.simpsons.gov

Afterwards restart the iscsid service to make the changes above effective:

bart:~ # systemctl restart iscsid

Connect to the iscsi target on the server:

bart:~ # iscsiadm -m node -p 192.168.2.235 --login
Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d, portal: 192.168.2.235,3260] (multiple)
Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.skinner.x8664:sn.1f9f4047108d, portal: 192.168.2.235,3260] successful.

Last but not least set startup mode to automatic for the corresponding targets for the ISCSI Initiator via YAST:

YAST > Network Services > iSCSI Initiator > Connected Targets > Edit > Startup > automatic

If all worked out fine and you did not have any typos. Hooooray you should have a running ISCSI LIO Server and two client nodes happily connected against the ISCSI targets.
In the second part (Holy Cluster Part 2 HAE Setup) I will concentrate on configuring and installing the HAE Cluster. Have fun!