VMware

vSphere 5 is here! But What Don't I Know? | Main | Guest OS Partition Alignment

08/26/2011

vSphere 5.0 Storage Features Part 12 - iSCSI Multipathing Enhancements

In 5.0, VMware has added a new UI interface to make it much easier to configure multipathing for Software iSCSI. The UI is a major change from 4.x. Previously users needed to use the command-line to get an optimal multi-path configuration with Software iSCSI.

The UI allows one to select different network interfaces for iSCSI use, check them for compliance and configure them with the Software iSCSI Adapter. This multipathing configuration is also referred to as iSCSI Port Binding.

Why should I use iSCSI Multipathing?

The primary use case of this feature is to create a multipath configuration with storage that only presents a single storage portal, such as the DELL EqualLogic and the HP/Lefthand. Without iSCSI multipathing, these type of storage would only have one path between the ESX host and each volume.  iSCSI multipathing allows us to multipath to this type of clustered storage.

Another benefit is the ability to use alternate VMkernel networks outside of the ESXi Management network. This means that if the management network suffers an outage, you continue to have iSCSI connectivity via the VMkernel ports participating in the iSCSI bindings.

Let's see how you go about setting this up. In this example, I have configured a Software iSCSI adapter, vmhba32.

No-devices

At present, no targets have been added, so no devices or paths have been discovered. Before implementing the iSCSI bindings, I need to create a number of additional VMkernel ports (vmk) for port binding to the Software iSCSI adapter.

Vmkernel-nw

As you can see from the above diagram, these vmnics are on trunked VLAN ports, allowing them to participate in multiple VLANs. For port binding to work correctly, the initiator must be able to reach the target directly on the same subnet - iSCSI port binding in vSphere 5.0 does not support routing. In this configuration, if I place my VMkernel ports on VLAN 74, they will be able to reach the iSCSI target without the need of a router. This is an important point, and needs further elaboration as it causes some confusion. If I do not implement port binding, and use a standard VMkernel port, then my initiator can reach the targets through a routed network. This is supported and works just fine. It is only when iSCSI binding is implemented that a direct, non-routed network between the initiators and targets is required, i.e. initiators and targets must be on the same subnet.

There is another important point to note when it comes to the configuration of iSCSI port bindings. On vSwitches which contain multiple vmnic uplinks, each VMkernel (vmk) port used for iSCSI bindings must be associated with a single vmnic uplink. The other uplink(s) on the vSwitch must be placed into an unused state. See below for an example of such a configuration:

Unused-vmnic

This is only a requirement when you have multiple vmnic uplinks on the same vSwitch. If you are using multiple vSwitches with their own vmnic uplinks, then this isn't an issue. Continuing with the network configuration,  we create a second VMkernel (vmk) port. I now have two vmk ports, labeled iscsi1 & iscsi2. These will be used for my iSCSI binding. Note below that one of the physical adapters, vmnic1, appeares disconnected from the vSwitch. This is because both of my VMkernel ports will be bound to vmnic0 only, so vmnic1 has been set to unused across the whole of the vSwitch.

Vmkernel-nw4Next, I return to the properties of my Software iSCSI adapter, and configure the bindings and iSCSI targets. There is now a new Network Configuration tab in the Software iSCSI Adapter properties window. This is where you add the VMkernel ports that will be used for binding to the iSCSI adapter. Click on the Software iSCSI adapater properties, then select the Network configuration tab, and you will see something similar to the screenshot shown below:

Bindings1

After selecting the VMkernel adapters for use with the Software iSCSI Adapter, the Port Group Policy tab will tell you whether or not these adapters are compliant or not for binding. If you have more than one active uplink on a vSwitch that has multiple vmnic uplinks, the vmk interfaces will not show up as compliant. Only one uplink should be active, all other uplinks should be placed into an unused state.

Bindings2
You then proceed to the Dynamic Discovery tab, where the iSCSI targets can now be added. Now, because we are using port binding, you must ensure that these targets are reachable by the Software iSCSI Adapter through a non-routeable network, i.e the storage controller ports are on the same subnet as the VMkernel NICs:

Dynamic-discovery
At this point, I have two vmkernel ports bound to the Software iSCSI Adapter, and 4 targets. These 4 targets are all going to the same storage array, so if I present a LUN out on all 4 targets, this should give me a total of 8 paths. Let's see what happens when I present a single LUN (ID 0):

Devices-8paths
So it does indeed look like I have 8 paths to that 1 device. Let's verify by looking at the paths view:

8paths
And if I take a look at a CLI multipath output for this device, I should see it presented on 8 different targets:

~ # esxcfg-mpath -l -d naa.6006016094602800c8e3e1c5d3c8e011 | grep "Target Identifier"
   Target Identifier: 00023d000002,iqn.1992-04.com.emc:cx.ckm00100900477.a2,t,1
   Target Identifier: 00023d000001,iqn.1992-04.com.emc:cx.ckm00100900477.a2,t,1
   Target Identifier: 00023d000002,iqn.1992-04.com.emc:cx.ckm00100900477.b3,t,4
   Target Identifier: 00023d000001,iqn.1992-04.com.emc:cx.ckm00100900477.b3,t,4
   Target Identifier: 00023d000002,iqn.1992-04.com.emc:cx.ckm00100900477.b2,t,3
   Target Identifier: 00023d000001,iqn.1992-04.com.emc:cx.ckm00100900477.b2,t,3
   Target Identifier: 00023d000002,iqn.1992-04.com.emc:cx.ckm00100900477.a3,t,2
   Target Identifier: 00023d000001,iqn.1992-04.com.emc:cx.ckm00100900477.a3,t,2

 
This new UI for iSCSI Bindings certainly makes configuring multipathing for the Software iSCSI Adapter so much easier. But do keep in mind the requirements to have a non-routable network between the initiator and target, and the fact that vmkernel ports must have only a single active vmnic uplink in vSwitches that have multiple vmnic uplinks.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: Twitter VMwareStorage

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Hi Chogan,

I've been deploying vSphere clusters running on Equallogic Storage for over 2 years, and DELL White Paper on configuring vSphere instructs on using multiple active adapters.

Your blog post instructions on leaving adapters unused intrigues me: how can vSphere reach over-gigabit uplinks (using gigabit nics, of course) ? Will it create a new connection on-the-fly, or the unused adapter will only be used as a backup nic ?

Hi Fernando, thanks for the comment.
Definitely follow the vendor instructions. The procedure here is simply to show off the new UI interface - the configuration that I used is not the optimal one. Using multiple uplinks, one for each VMkernel port, as you have been doing & as EQL recommends, is definitely the way to go.

No mention of jumbo frames, is the MTU setting still only via CLI?

Good stuff, Cormac.

Fernando, the adapters are unused *for that particular vmknic.* To use all the adapters, created as many vmknics as you have adapters, and set up the vmknics with one active uplink (adapter), each.

The paths will then be used however you configure them through your stroage setup.

A Much more optimal way to set this up is to make both NICs active on the vSwitch, and then set the vmknics to an Active/Unused config in their NIC Teaming tab, with vmk1 using vmnic0 as active, and vmk2 using vmnic1. Having multiple paths that traverse just the one nic doesn't make a lot of sense.

What when you use VD switches? 2 or more vmNICs will be linked to 1 dvSwitch so I get an error when enabling iSCSI saying that only 1 NIC can be attached for iSCSI?
What is the best practice there?

Ron, agreed. That is an optimal config that you describe.

Bart, for the different vmkernel ports that are used for iSCSI binding, they need to be changed so that only one uplink is active, and the other is unused.

Hi Chogan,

Does this mean we don't need to do the "esxcli swiscsi nic add -n vmk1 -d vmhba34" stuff we used to do in ESXi 4.1?

Thanks!


Chris

Hi Chris,
That is correct. This new UI means that you no longer need to use the CLI commands to setup iSCSI Multipathing/Port Binding.

"you must ensure that these targets are reachable by the Software iSCSI Adapter through a non-routeable network"

Is there a way to override this? I need to use iSCSI targets at remote, NAT-ed addresses. Discovery is working fine, but it resolves to the internal address of the device. If i add static iscsi device, it never reaches the target.

Hi Imo,
I'm afraid not. iSCSI adapters using VMkernel adapters are not able to connect to iSCSI ports on different subnets, even if those ports are discovered by the iSCSI adapters, even in vSphere 5.0.

Hi,

Thanks for the info. This might be a feature in future vSphere releases, because in my case the lack of this function will force me to use hyper-v server where routed/NAT-ed iscsi devices are working out of the box.

Hi Imo,
Just to clarify, the initiator and target only need to be on the same subnet to use binding/multipathing. If you do not use binding, the connection between initiator and target will be via a standard VMkernel port which 'can' be routed. I hope that is a solution for you - we'd hate to force you to use Hyper-V :-)

Hi,

I do not need MP.
If it should work, I will give it another try, but I already spent several hours without any luck. I used wireshark on the target and not a single iSCSI packet ever reached it from the vSphere host, while the same config worked without any configuration change if the target was on the same subnet...

HI Imo, please open a Service Request if you still can't get it to work. without binding, iSCSI should work perfectly well over routed networks.

Hi,

Sry, it was my fault, I followed the guides blindly and I created a vmkernel port group with a single NIC/PG... After removing it it's working.

Thanks
Imo

Glad to hear that Imo.

The second figure where the vSwitch0 is shown is wrong. You are stating that it must be done on the vSwitch, in my opionin you must do it on the VMKernel port group. There you need to put the nic's in unused and used dapater and bind it correctly. IT seems a bit confusing.

Btw, here is a url how i think it should be configured. Please correct me if i'm wrong :)

http://blog.vmpros.nl/2011/07/13/vmware-creating-iscsi-network-in-vsphere-esxi-5-0/

Thanks Ivan. The text is correct in so far as it refers to a VMkernel port, but the diagram clearly shows a vSwitch, so thanks for catching. I'll see if I can get it changed one of these days.

Post a comment

If you have a TypeKey or TypePad account, please Sign In.