Monday, April 21, 2014

Sun Storage Solaris Multipathing Software

MPxIO

Configuration files

For Solaris 10 and earlier
/kernel/drv/fp.confFiber channel port configuration file (For Fiber Channel Devices)
/kernel/drv/mpt.confmpt scsi/SAS HBA controller driver configuration file (For SAS Devices)
/kernel/drv/scsi_vhci.confMPXIO operation configuration file (Used to define loadbalancing and failback modes and configure 3rd party devices)
/kernel/drv/iscsi_confiSCSI MPxIO configuration file (For iSCSI devices)
For Solaris 11

In case of Solaris 11, files in the directory /kernel/drv/ and /etc/driver/drv are the sample default configuration files delivered with OS (not used by MPxIO). Do not edit these files. The location of the configuration files is /etc/driver/drv/. The 4 configuration files used in case of Solaris 11 are the same, performing the same task. Although the syntax may be different.

/etc/driver/drv/fp.conf
/etc/driver/drv/mpt.conf
/etc/driver/drv/scsi_vhci.conf
/etc/driver/drv/iscsi_conf
 
It is best practice to edit these files using the stmsboot utility, rather than editing them manually.

Enabling/Disabling MPxIO

The syntax of the command stmsboot used to enable/disable MPxIO is :
stmsboot [-D fp|mpt] [-d | -e | -u | -L | -l controller_number]

where:
-D fp|mpt -> to select the protocol where you want to enable/disable mpxio
-e -> to enable STMS on all HBA
-d -> to disable STMS on all HBA
-u -> to update vfstable and dump config after manual file configuration
-l or -L unchanged - see the manpage of stms for a full description.
Few examples of using stmsboot command are :
# stmsboot -eTo enable MPxIO or all devices (globally)
# stmsboot -D fp -eTo enable MPxIO for fiber channel devices
# stmsboot -dTo disable MPxIO for all devices (globally)

How to confirm MPxIO is enabled

To make sure that MPxIO is enabled for a LUN, simply run the format command and check for the /scsi_vhci in the logical path of the LUN.

# format
4. c6t60060E800561CF00000061CF00000002d0 
          /scsi_vhci/ssd@g60060e800561cf00000061cf00000002
Another way is to check the “mpxio-disable” parameter in the configuration files :

Solaris 7,8,9
# egrep "^mpxio-disable=" /kernel/drv/*.conf
/kernel/drv/scsi_vhci.conf:mpxio-disable="no";
 
Solaris 10
# egrep "^mpxio-disable=" /kernel/drv/*.conf
/kernel/drv/fp.conf:mpxio-disable="no";
 
Solaris 11
# egrep "^mpxio-disable=" /etc/driver/drv/*.conf
/etc/driver/drv/fp.conf:mpxio-disable="no";
For Solaris 10 and 11, MPxIO is enabled by default on X86 platforms.

Verifying that the scsi_vhci module is loaded

scsi_vhci is the virtual host controller interconnect driver used to manage path of storage LUN. If you find that the scsi_vhci module is not loaded, then most likely MPxIO is not enabled.
Solaris[TM] 7,8,9,10,11
# modinfo -c |grep scsi_vhci
 21          1 scsi_vhci                        LOADED/INSTALLED

Verifying Health of paths

We can check the multiple paths to the storage array LUNs by using mpathadm or luxadm command. The various states for the paths are :

ONLINE/primary
ONLINE/secondary
STANDBY/secondary
OFFLINE/*
UNAVAILABLE/*
 
mpathadm (supported for FC/SAS/iSCSI) [Works on Solaris 10, 11]
luxadm (supported on FC only) [Works on Solaris 7,8,9,10,11]

# mpathadm list LU
        /dev/rdsk/c9t60080E50001C0300000006464D8C9044d0s2
                Total Path Count: 2 
                Operational Path Count: 2
To check the health of the paths using mpathadm :
# mpathadm show LU  /dev/rdsk/c9t60080E50001C0300000006464D8C9044d0s2

Logical Unit:  /dev/rdsk/c9t60080E50001C0300000006464D8C9044d0s2
        mpath-support:  libmpscsi_vhci.so
        Vendor:  SUN     
        Product:  LCSM100_F
        Revision:  0777
        Name Type:  unknown type
        Name:  60080e50001c0300000006464d8c9044
        Asymmetric:  yes
        Current Load Balance:  round-robin
        Logical Unit Group ID:  NA
        Auto Failback:  on
        Auto Probing:  NA

        Paths:  
                Initiator Port Name:  210000e08b07922c
                Target Port Name:  203d0080e51c2cc4
                Override Path:  NA
                Path State:  OK
                Disabled:  no

                Initiator Port Name:  210000e08b07f32c
                Target Port Name:  203e0080e51c2cc4
                Override Path:  NA
                Path State:  OK
                Disabled:  no
.........
To check health of paths using luxadm :
# luxadm probe
                Found Fibre Channel device(s):
                  ......
                Logical Path:/dev/rdsk/c6t60060E800561CF00000061CF00000002d0s2
                Node WWN:203e0080e51c2cc4  Device Type:Disk device
                   ......
# luxadm display 203e0080e51c2cc4
DEVICE PROPERTIES for disk: 203e0080e51c2cc4
  Vendor:               SUN     
  Product ID:           LCSM100_F ------------> this is a 2540 or 2540-M2 array
  Revision:             0777
  Serial Num:           SV04720931      
  Unformatted capacity: 5120.000 MBytes
  Write Cache:          Enabled
  Read Cache:           Enabled
    Minimum prefetch:   0x3
    Maximum prefetch:   0x0
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c9t60080E50001C0300000006464D8C9044d0s2
  /devices/scsi_vhci/ssd@g60080e50001c0300000006464d8c9044:c,raw
   Controller           /devices/pci@1d,700000/SUNW,qlc@1/fp@0,0
    Device Address              203d0080e51c2cc4,0
    Host controller port WWN    210000e08b07922c
    Class                       primary
    State                       ONLINE
   Controller           /devices/pci@1c,600000/SUNW,qlc@1/fp@0,0
    Device Address              203e0080e51c2cc4,0
    Host controller port WWN    210000e08b07f32c
    Class                       secondary
    State                       ONLINE

Load Balancing policies

MPxIO also provides the load balancing capability to the I/O. There are basically 3 options to configure load balancing :
Round RobinSelects all paths in a round robin way.
Logical BlockI/Os within the range specified by range_size parameter would go on the same path. Improves performance on some storage arrays which have per path caching of data.
NoneNo load balancing – only one path is used.
The load balancing policy can be set in the configuration file /kernel/drv/scsi_vhci.conf for solaris 10 and earlier or in the /etc/driver/drv/scsi_vhci.conf file for solaris 11.
load-balance="round-robin";

Auto-failback policy

Auto-failback policy enables the re-enable the failed path after it has recovered from the failure. By default the auto-failback is enabled. There are 2 ways to enable the auto-failback :
Non-persistent way
By this method, the auto-failback option does not persist across reboots.
# mpathadm modify mpath-support --autofailback on libmpscsi_vhci.so
# mpathadm modify mpath-support -a on libmpscsi_vhci.so
Persistent way
For the changes to persist across reboots set the auto-failback parameter in the configuration file /kernel/drv/scsi_vhci.conf for solaris 10 and earlier or in the /etc/driver/drv/scsi_vhci.conf file for solaris 11. This requires a reconfiguration reboot.
auto-failback="enable";
Verifying auto-failback
To confirm that the auto-failback is enabled use :
# mpathadm show mpath-support libmpscsi_vhci.so | grep -i failback
           Supported Auto Failback Config:  1
           Auto Failback:  on  <--- 0="" current="" failback="" max="" polling="" pre="" rate="">


========================================================


bash-3.2# egrep "^mpxio-disable=" /kernel/drv/*.conf
/kernel/drv/fp.conf:mpxio-disable="yes"; => change to no/kernel/drv/iscsi.conf:mpxio-disable="no";
/kernel/drv/mpt.conf:mpxio-disable="yes"; => change to no
/kernel/drv/mpt_sas.conf:mpxio-disable="no";

bash-3.2# mpathadm list lu
        /dev/rdsk/c6t20000018620E6681d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c6t20000018620E42AFd0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c6t20000018620E400Bd0s2
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c6t20000018620E4155d0s2
                Total Path Count: 1
                Operational Path Count: 1
        /scsi_vhci/ses@g5080020000284028
                Total Path Count: 1
                Operational Path Count: 1
        /dev/rdsk/c6t600A0B800018B3CA000052F6534DB0C5d0s2
                Total Path Count: 2
                Operational Path Count: 2
        /dev/rdsk/c6t600A0B800018B3CA000052EF534DB005d0s2
                Total Path Count: 2
                Operational Path Count: 2

           

bash-3.2# fcinfo hba-port
HBA Port WWN: 210000144f415c2a
        OS Device Name: /dev/cfg/c1
        Manufacturer: QLogic Corp.
        Model: 2200
        Firmware Version: 02.01.145
        FCode/BIOS Version: ISP2200 FC-AL Host Adapter Driver: 1.15 04/03/22
        Serial Number: not available
        Driver Name: qlc
        Driver Version: 20120717-4.01
        Type: L-port
        State: online
        Supported Speeds: 1Gb
        Current Speed: 1Gb
        Node WWN: 200000144f415c2a
HBA Port WWN: 10000000c9c25df0
        OS Device Name: /dev/cfg/c4
        Manufacturer: Emulex
        Model: LP11000-S
        Firmware Version: 2.82a4 (B3D2.82A4)
        FCode/BIOS Version: Boot:5.02a1 Fcode:1.50a9
        Serial Number: 0999BT0-1121000JC2
        Driver Name: emlxs
        Driver Version: 2.80.8.0 (2012.09.17.15.45)
        Type: L-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 2Gb
        Node WWN: 20000000c9c25df0
HBA Port WWN: 10000000c9c25d6f
        OS Device Name: /dev/cfg/c5
        Manufacturer: Emulex
        Model: LP11000-S
        Firmware Version: 2.82a4 (B3D2.82A4)
        FCode/BIOS Version: Boot:5.02a1 Fcode:1.50a9
        Serial Number: 0999BT0-1121000J8Z
        Driver Name: emlxs
        Driver Version: 2.80.8.0 (2012.09.17.15.45)
        Type: L-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 2Gb
        Node WWN: 20000000c9c25d6f
       
       
       
bash-3.2# mpathadm list mpath-support
mpath-support:  libmpscsi_vhci.so


bash-3.2# mpathadm list initiator-port
Initiator Port:  210000144f415c2a
Initiator Port:  10000000c9c25df0
Initiator Port:  10000000c9c25d6f
Initiator Port:  iqn.1986-03.com.sun:01:00144f415c2a.5354dd03,4000002a00ff




bash-3.2# mpathadm show initiator-port 10000000c9c25df0
Initiator Port:  10000000c9c25df0
        Transport Type:  Fibre Channel
        OS Device File:  /devices/pci@9,600000/SUNW,emlxs@1/fp@0,0
bash-3.2# mpathadm show initiator-port 10000000c9c25d6f
Initiator Port:  10000000c9c25d6f
        Transport Type:  Fibre Channel
        OS Device File:  /devices/pci@9,600000/SUNW,emlxs@2/fp@0,0
       
       
bash-3.2# mpathadm show lu /dev/rdsk/c6t600A0B800018B3CA000052F6534DB0C5d0s2
Logical Unit:  /dev/rdsk/c6t600A0B800018B3CA000052F6534DB0C5d0s2
        mpath-support:  libmpscsi_vhci.so
        Vendor:  SUN    
        Product:  CSM100_R_FC    
        Revision:  0660
        Name Type:  unknown type
        Name:  600a0b800018b3ca000052f6534db0c5
        Asymmetric:  yes
        Current Load Balance:  round-robin
        Logical Unit Group ID:  NA
        Auto Failback:  on
        Auto Probing:  NA

        Paths: 
                Initiator Port Name:  10000000c9c25df0
                Target Port Name:  200400a0b81ffbef
                Override Path:  NA
                Path State:  OK
                Disabled:  no

                Initiator Port Name:  10000000c9c25d6f
                Target Port Name:  200500a0b81ffbef
                Override Path:  NA
                Path State:  OK
                Disabled:  no

        Target Port Groups: 
                ID:  6
                Explicit Failover:  yes
                Access State:  standby
                Target Ports:
                        Name:  200400a0b81ffbef
                        Relative ID:  0

                ID:  7
                Explicit Failover:  yes
                Access State:  active
                Target Ports:
                        Name:  200500a0b81ffbef
                        Relative ID:  0
                     
=================================
Test multipath

bash-3.2# fcinfo hba-port

HBA Port WWN: 10000000c9c25df0
        OS Device Name: /dev/cfg/c4
        Manufacturer: Emulex
        Model: LP11000-S
        Firmware Version: 2.82a4 (B3D2.82A4)
        FCode/BIOS Version: Boot:5.02a1 Fcode:1.50a9
        Serial Number: 0999BT0-1121000JC2
        Driver Name: emlxs
        Driver Version: 2.80.8.0 (2012.09.17.15.45)
        Type: L-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 2Gb
        Node WWN: 20000000c9c25df0
HBA Port WWN: 10000000c9c25d6f
        OS Device Name: /dev/cfg/c5
        Manufacturer: Emulex
        Model: LP11000-S
        Firmware Version: 2.82a4 (B3D2.82A4)
        FCode/BIOS Version: Boot:5.02a1 Fcode:1.50a9
        Serial Number: 0999BT0-1121000J8Z
        Driver Name: emlxs
        Driver Version: 2.80.8.0 (2012.09.17.15.45)
        Type: L-port
        State: online
        Supported Speeds: 1Gb 2Gb 4Gb
        Current Speed: 2Gb
        Node WWN: 20000000c9c25d6f

bash-3.2# luxadm -e forcelip /dev/cfg/c4


$ luxadm -e forcelip /dev/cfg/c4

bash-3.2# tail -f /var/adm/messages

[ID 349649 kern.info] [ 5.06C6]emlxs1: NOTICE: 730: Link reset. (Resetting link...)
[ID 349649 kern.info] [ 5.03DD]emlxs1: NOTICE: 710: Link down.
[ID 349649 kern.info] [ 5.0715]emlxs1: NOTICE: 730: Link reset.
[ID 349649 kern.info] [ 5.061B]emlxs1: NOTICE: 720: Link up. (2Gb, loop, initiator)


[ID 349649 kern.info] [ 5.06C6]emlxs0: NOTICE: 730: Link reset. (Resetting link...)
[ID 349649 kern.info] [ 5.03DD]emlxs0: NOTICE: 710: Link down.
[ID 349649 kern.info] [ 5.0715]emlxs0: NOTICE: 730: Link reset.
[ID 349649 kern.info] [ 5.061B]emlxs0: NOTICE: 720: Link up. (2Gb, loop, initiator)