RHEL iSCSI part deux

So as it turns out, the earlier post was not always the best approach to using iSCSI with RHEL. An important omission from the earlier post is that we are using Dell EqualLogic storage, and thus installed the EqualLogic Host Integration Tools (HIT). These tools help integrate iSCSI and multipathing into the server, changing the approach in the process. Below are the steps that we took to get our EqualLogic iSCSI storage online and working:

1. Create a new volume using the administration console of the SAN and ensure that a CHAP user has been created that has access to the volume


2. Install equallogic-host-tools on the server.


3. As root (or using sudo), ensure that ehcmd is started service ehcmd status


4. Edit /etc/iscsi/iscsi.conf by setting node startup as manual (as otherwise, iSCSI will try to establish too many redundant connections):
   
   #***************** # Startup settings #***************** #
   To request that the iscsi initd scripts startup a session set to "automatic". # node.startup = automatic # # To manually startup the session set to "manual". The default is automatic.
   node.startup = manual
   
   

   enable CHAP and setting the CHAP user and password

   
   
   # ************* # CHAP Settings # ************* #
   To enable CHAP authentication set node.session.auth.authmethod # to CHAP. The default is None.
   node.session.auth.authmethod = CHAP
   # To set a CHAP username and password for initiator # authentication by the target(s), uncomment the following lines:
   node.session.auth.username = user
   node.session.auth.password = password
   
   

   ...

   
   

   # To set a discovery session CHAP username and password for the initiator # authentication by the target(s), uncomment the following lines: discovery.sendtargets.auth.username = user discovery.sendtargets.auth.password = password



5. Run eqltune -v and follow any recommendations made by the tool


6. run ehcmcli -d to view the ehcmcli diagnostics information


7. Check to see which subnets EqualLogic host connection manager will manage. # rswcli -L. Exclude any subnets that you do not wish to use with multipath. In our case, we have two interfaces for data on one subnet and two interfaces for iSCSI on two other interfaces: rswcli -E -network 10.52.195.128 -mask 255.255.255.128 Run rswcli -L and ehcmcli -d to confirm that the correct adapters/interfaces are being used


8. Run Discovery:
   # iscsiadm -m discovery -p 10.10.5.50 -t st 10.10.5.50
   
   
   
   
   iqn.2001-05.com.equallogic:0-8a0906-a884a4502-fdefc658d2e4d064-oracle-data 10.10.5.50:3260,1
   
   
   iqn.2001-05.com.equallogic:0-8a0906-a884a4502-fdefc658d2e4d064-oracle-data 10.10.5.50:3260,1
   
   
   iqn.2001-05.com.equallogic:0-8a0906-a884a4502-fdefc658d2e4d064-oracle-data 10.10.5.50:3260,1
   
   
   iqn.2001-05.com.equallogic:0-8a0906-a884a4502-fdefc658d2e4d064-oracle-data 10.10.5.50:3260,1
   
   Multiple identical results will be returned as a result of the intefaces that the EqualLogic daemon creates.


9. Login to the target specifying the interface:
   # iscsiadm -m node -T iqn.2001-05.com.equallogic:0-8a0906-a884a4502-fdefc658d2e4d064-oracle-data -I eql.eth1_0 -l


10. Allow automatic logins using the node.startup property. Recall that this was set to manual in the /etc/iscsi/iscsi.conf file in the steps above, since we don't want to open sessions on all interfaces because the EqualLogic daemon creates multiple interfaces using the same adapters (eql.eth1_0, eql.eth1_1, etc). Instead, we allowed automatic startup on two interfaces, one for each adapter (eql.eth1_0 and eql.eth3_0):
    # iscsiadm –m node –T -I eql.eth1_0 –o update –n node.startup –v automatic
    
    
    # iscsiadm –m node –T -I eql.eth3_0 –o update –n node.startup –v automatic


11. Edit /etc/equallogic/eql.conf [MPIO Autologins] section to specify which adapters to use:
    
    # Autologin configuration [MPIO Autologins]
    
    
    # ehcmd will update any targets configured for autologin so the boot time logins
    # are only made through two iscsiadm ifaces using independent ethernet ports.
    # The ehcmd will create additional iSCSI sessions if necessary at a later point.
    # # If you wish to control which ethernet interfaces are used for autologin,
    # list the desired interfaces below, using a separate line for each.
    # Example: # Port = eth0
    # Port = eth1
    Port = eth1
    Port = eth3




12. Create a physical volume using the /dev/mapper/ name, not the SD device name or one of the derivatives created by the equallogic daemon (eql-0-8a0906-d11b93e09-93720a256664d91d-a, eql-0-8a0906-d11b93e09-93720a256664d91d-oracle-datap1, etc) # pvcreate /dev/mapper/eql-0-8a0906-d11b93e09-93720a256664d91d-oracle-data


13. Create the volume group on the newly created physical volume. # vgcreate SANVolGroup00 /dev/mapper/eql-0-8a0906-d11b93e09-93720a256664d91d-oracle-data


14. Create the logical volume. In our case, we created the volume the entire size of the SAN volume: # lvcreate --extents 100%VG --name ora_data2 SANVolGroup00


15. Format the file system as ext3: # mkfs.ext3 /dev/SANVolGroup00/ora_data2


16. Mount the new lv to a mount point: # mount /dev/SANVolGroup00/ora_data2 /oracle/ora_data2


17. Finally, edit /etc/fstab to mount the logical volume on startup: /dev/SANVolGroup00/ora_data2 /oracle/ora_data2 ext3 _netdev,defaults 0 0 Ensure that the mount option _netdev is used, so the volume does not mount until network interfaces are up

RHEL Mapper stealing Filesystem

Today, we've setup our RHEL server to use iSCSI with CHAP authentication to connect to our iSCSI array volumes. We found this tutorial very helpful, although instead of open-iscsi, we simply used RedHat's iscsi-initiator-utils package (probably the same thing).

One thing to note is that after successfully authenticating to the array via CHAP and partitioning the block (which was /dev/sdd) with LVM2, we could not successfully run the pvcreate /dev/sdd1 command to create a physical volume on which we can build our LVMs. When running pvcreate, we'd get the following:

#pvcreate /dev/sdd1
Can't open /dev/sdd1 exclusively. Mounted filesystem?

Although running 'mount' wouldn't show the drive mapped anywhere. As it turns out, after writing LVM2 to the partition table, /dev/mapper grabs onto the device which effectively "mounts" it within the mapper. In our case, it was listed in the mapper as three devices:

eql-0-8a0906-8a0b93e09-01120a256614d8a6-ora-data01p1
eql-0-8a0906-8a0b93e09-01120a256614d8a6-ora-data01
eql-0-8a0906-8a0b93e09-01120a256614d8a6_a

Note the long name referencing our Equillogic iSCSI SAN. We found two workarounds:

1. Run pvcreate using the /dev/mapper/ name, so:
   #pvcreate /dev/mapper/eql-0-8a0906-8a0b93e09-01120a256614d8a6-ora-data01p1
   This was less desirable, however, because of the long file name
   
2. Remove the device from the /dev/mapper and then run pvcreate:
   
   #dmsetup ls
   #dmsetup remove eql-0-8a0906-8a0b93e09-01120a256614d8a6-ora-data01p1
   #dmsetup remove eql-0-8a0906-8a0b93e09-01120a256614d8a6-ora-data01
   #dmsetup remove eql-0-8a0906-8a0b93e09-01120a256614d8a6_a
   #pvcreate /dev/sdd1
   Physical volume "/dev/sdd1" successfully created
   
   

That solved our problem, allowing us to create the physical volume and then create our LVMs on top of it. Thanks to this forum article for the tip.

Lesson Learned

After not eating lunch today and doing some quick configurations withing vCenter, I accidentally deleted a vDisk off of the wrong VM. Not good. Sure enough, when logging into the ESX host, the vDisk was gone, unrecoverable. I recall VMWare ESX 3.5 had a utility named vmfs-undelete for the purpose of recovering deleted vmdk files. No longer:

vmfs-undelete utility is not available for ESX/ESXi 4.0
ESX/ESXi 3.5 Update 3 included a utility called vmfs-undelete, which could be used to recover deleted .vmdk files. This utility is not available with ESX/ESXi 4.0.

Workaround: None. Deleted .vmdk files cannot be recovered.

[VMware ESX 4.0 Update 1 Release Notes]

Fortunately the data on the vDisk can be recreated by the developer in about an hour, so all told we're not set back too far. An important lesson learned though:

1. Always double check before permanently deleting
2. Don't have multiple configuration windows open at the same time
3. Eat lunch