In 5.0, VMware has added a new UI interface to make it much easier to configure multipathing for Software iSCSI. The UI is a major change from 4.x. Previously users needed to use the command-line to get an optimal multi-path configuration with Software iSCSI.
The UI allows one to select different network interfaces for iSCSI use, check them for compliance and configure them with the Software iSCSI Adapter. This multipathing configuration is also referred to as iSCSI Port Binding.
Why should I use iSCSI Multipathing?
The primary use case of this feature is to create a multipath configuration with storage that only presents a single storage portal, such as the DELL EqualLogic and the HP/Lefthand. Without iSCSI multipathing, these type of storage would only have one path between the ESX host and each volume. iSCSI multipathing allows us to multipath to this type of clustered storage.
Another benefit is the ability to use alternate VMkernel networks outside of the ESXi Management network. This means that if the management network suffers an outage, you continue to have iSCSI connectivity via the VMkernel ports participating in the iSCSI bindings.
Let's see how you go about setting this up. In this example, I have configured a Software iSCSI adapter, vmhba32.
At present, no targets have been added, so no devices or paths have been discovered. Before implementing the iSCSI bindings, I need to create a number of additional VMkernel ports (vmk) for port binding to the Software iSCSI adapter.
As you can see from the above diagram, these vmnics are on trunked VLAN ports, allowing them to participate in multiple VLANs. For port binding to work correctly, the initiator must be able to reach the target directly on the same subnet – iSCSI port binding in vSphere 5.0 does not support routing. In this configuration, if I place my VMkernel ports on VLAN 74, they will be able to reach the iSCSI target without the need of a router. This is an important point, and needs further elaboration as it causes some confusion. If I do not implement port binding, and use a standard VMkernel port, then my initiator can reach the targets through a routed network. This is supported and works just fine. It is only when iSCSI binding is implemented that a direct, non-routed network between the initiators and targets is required, i.e. initiators and targets must be on the same subnet.
There is another important point to note when it comes to the configuration of iSCSI port bindings. On vSwitches which contain multiple vmnic uplinks, each VMkernel (vmk) port used for iSCSI bindings must be associated with a single vmnic uplink. The other uplink(s) on the vSwitch must be placed into an unused state. See below for an example of such a configuration:
This is only a requirement when you have multiple vmnic uplinks on the same vSwitch. If you are using multiple vSwitches with their own vmnic uplinks, then this isn't an issue. Continuing with the network configuration, we create a second VMkernel (vmk) port. I now have two vmk ports, labeled iscsi1 & iscsi2. These will be used for my iSCSI binding. Note below that one of the physical adapters, vmnic1, appeares disconnected from the vSwitch. This is because both of my VMkernel ports will be bound to vmnic0 only, so vmnic1 has been set to unused across the whole of the vSwitch.
Next, I return to the properties of my Software iSCSI adapter, and configure the bindings and iSCSI targets. There is now a new Network Configuration tab in the Software iSCSI Adapter properties window. This is where you add the VMkernel ports that will be used for binding to the iSCSI adapter. Click on the Software iSCSI adapater properties, then select the Network configuration tab, and you will see something similar to the screenshot shown below:
After selecting the VMkernel adapters for use with the Software iSCSI Adapter, the Port Group Policy tab will tell you whether or not these adapters are compliant or not for binding. If you have more than one active uplink on a vSwitch that has multiple vmnic uplinks, the vmk interfaces will not show up as compliant. Only one uplink should be active, all other uplinks should be placed into an unused state.
You then proceed to the Dynamic Discovery tab, where the iSCSI targets can now be added. Now, because we are using port binding, you must ensure that these targets are reachable by the Software iSCSI Adapter through a non-routeable network, i.e the storage controller ports are on the same subnet as the VMkernel NICs:
At this point, I have two vmkernel ports bound to the Software iSCSI Adapter, and 4 targets. These 4 targets are all going to the same storage array, so if I present a LUN out on all 4 targets, this should give me a total of 8 paths. Let's see what happens when I present a single LUN (ID 0):
So it does indeed look like I have 8 paths to that 1 device. Let's verify by looking at the paths view:
And if I take a look at a CLI multipath output for this device, I should see it presented on 8 different targets:
~ # esxcfg-mpath -l -d naa.6006016094602800c8e3e1c5d3c8e011 | grep "Target Identifier"
Target Identifier: 00023d000002,iqn.1992-04.com.emc:cx.ckm00100900477.a2,t,1
Target Identifier: 00023d000001,iqn.1992-04.com.emc:cx.ckm00100900477.a2,t,1
Target Identifier: 00023d000002,iqn.1992-04.com.emc:cx.ckm00100900477.b3,t,4
Target Identifier: 00023d000001,iqn.1992-04.com.emc:cx.ckm00100900477.b3,t,4
Target Identifier: 00023d000002,iqn.1992-04.com.emc:cx.ckm00100900477.b2,t,3
Target Identifier: 00023d000001,iqn.1992-04.com.emc:cx.ckm00100900477.b2,t,3
Target Identifier: 00023d000002,iqn.1992-04.com.emc:cx.ckm00100900477.a3,t,2
Target Identifier: 00023d000001,iqn.1992-04.com.emc:cx.ckm00100900477.a3,t,2
This new UI for iSCSI Bindings certainly makes configuring multipathing for the Software iSCSI Adapter so much easier. But do keep in mind the requirements to have a non-routable network between the initiator and target, and the fact that vmkernel ports must have only a single active vmnic uplink in vSwitches that have multiple vmnic uplinks.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: VMwareStorage
Hi Chogan,
I’ve been deploying vSphere clusters running on Equallogic Storage for over 2 years, and DELL White Paper on configuring vSphere instructs on using multiple active adapters.
Your blog post instructions on leaving adapters unused intrigues me: how can vSphere reach over-gigabit uplinks (using gigabit nics, of course) ? Will it create a new connection on-the-fly, or the unused adapter will only be used as a backup nic ?
Hi Fernando, thanks for the comment.
Definitely follow the vendor instructions. The procedure here is simply to show off the new UI interface – the configuration that I used is not the optimal one. Using multiple uplinks, one for each VMkernel port, as you have been doing & as EQL recommends, is definitely the way to go.
No mention of jumbo frames, is the MTU setting still only via CLI?
Good stuff, Cormac.
Fernando, the adapters are unused *for that particular vmknic.* To use all the adapters, created as many vmknics as you have adapters, and set up the vmknics with one active uplink (adapter), each.
The paths will then be used however you configure them through your stroage setup.
A Much more optimal way to set this up is to make both NICs active on the vSwitch, and then set the vmknics to an Active/Unused config in their NIC Teaming tab, with vmk1 using vmnic0 as active, and vmk2 using vmnic1. Having multiple paths that traverse just the one nic doesn’t make a lot of sense.
What when you use VD switches? 2 or more vmNICs will be linked to 1 dvSwitch so I get an error when enabling iSCSI saying that only 1 NIC can be attached for iSCSI?
What is the best practice there?
Ron, agreed. That is an optimal config that you describe.
Bart, for the different vmkernel ports that are used for iSCSI binding, they need to be changed so that only one uplink is active, and the other is unused.
Hi Chogan,
Does this mean we don’t need to do the “esxcli swiscsi nic add -n vmk1 -d vmhba34″ stuff we used to do in ESXi 4.1?
Thanks!
Chris
Hi Chris,
That is correct. This new UI means that you no longer need to use the CLI commands to setup iSCSI Multipathing/Port Binding.
“you must ensure that these targets are reachable by the Software iSCSI Adapter through a non-routeable network”
Is there a way to override this? I need to use iSCSI targets at remote, NAT-ed addresses. Discovery is working fine, but it resolves to the internal address of the device. If i add static iscsi device, it never reaches the target.
Hi Imo,
I’m afraid not. iSCSI adapters using VMkernel adapters are not able to connect to iSCSI ports on different subnets, even if those ports are discovered by the iSCSI adapters, even in vSphere 5.0.
Hi,
Thanks for the info. This might be a feature in future vSphere releases, because in my case the lack of this function will force me to use hyper-v server where routed/NAT-ed iscsi devices are working out of the box.
Hi Imo,
Just to clarify, the initiator and target only need to be on the same subnet to use binding/multipathing. If you do not use binding, the connection between initiator and target will be via a standard VMkernel port which ‘can’ be routed. I hope that is a solution for you – we’d hate to force you to use Hyper-V
Hi,
I do not need MP.
If it should work, I will give it another try, but I already spent several hours without any luck. I used wireshark on the target and not a single iSCSI packet ever reached it from the vSphere host, while the same config worked without any configuration change if the target was on the same subnet…
HI Imo, please open a Service Request if you still can’t get it to work. without binding, iSCSI should work perfectly well over routed networks.
Hi,
Sry, it was my fault, I followed the guides blindly and I created a vmkernel port group with a single NIC/PG… After removing it it’s working.
Thanks
Imo
Glad to hear that Imo.
The second figure where the vSwitch0 is shown is wrong. You are stating that it must be done on the vSwitch, in my opionin you must do it on the VMKernel port group. There you need to put the nic’s in unused and used dapater and bind it correctly. IT seems a bit confusing.
Btw, here is a url how i think it should be configured. Please correct me if i’m wrong
http://blog.vmpros.nl/2011/07/13/vmware-creating-iscsi-network-in-vsphere-esxi-5-0/
Thanks Ivan. The text is correct in so far as it refers to a VMkernel port, but the diagram clearly shows a vSwitch, so thanks for catching. I’ll see if I can get it changed one of these days.
Hi, thanks for this info but I have a question around a discrepancy with the VMware documentation.
The following is taken from the vSphere Storage Guide ESXi5.0, p77
NOTE If you use separate vSphere switches, you must connect them to different IP subnets. Otherwise,
VMkernel adapters might experience connectivity problems and the host will fail to discover iSCSI LUNs.
Reading the guide it states that multipath iSCSI vmk ports must be on different networks, which seems to be at odds with this BLOG and most iSCSI configurations, and suggestions?
Hi Nathan,
The approach you mention is without iSCSI binding. In order to achieve some semblance of multipathing, you can put array controllers on different subnets, and then have the SW iSCSI initiator use different VMKernel ports to discover the LUNs by using its routing table.
iSCSI Binding provides you with a more elegant approach to implementing multipathing. And of course you do not need seperate vSwitches, nor do you need the VMkernel ports to be on different subnets
vmkernel ports must have only a single active vmnic uplink in vSwitches that have multiple vmnic uplinks –> I find this requirement incorrect and inefficient. In fact, we have deployed 2 systems of ESXi/iscsi SAN, each ESXi host having multiple NICs, using both software and hardware iscsi initiators. We created dedicated vSwitch for iscsi traffic, and these vSwitch have mutiple vmnics (all active) and multiple vmkernel ports –> no problem occurs when we make iscsi port binding, Port Group Policy shows Compliant. Besides that, we think ESXi will load-balance among these active vmnics on the vSwitch (this is default settings). Any other ideas welcomed.
Hi Minh,
In this configuration, I assume you see a single path to the iSCSI datastore from your ESX/ESXi host. Is that correct?
I don’t believe you will have load balancing in this case, nor any sort of multipathing.
The point here is that with this configuration, certain storage events will not allow your iSCSI traffic to failover to an alternate path. Using bindings to create multiple storage paths will give you more availability.
I would urge you to open a discussion with our support staff to discuss best practices for your iSCSI configuration.
Thanks for usefull article. Helped me a lot.
I’m having trouble adding new targets. It has worked just fine, but now I’m having trouble. The type of error I get in the ESXi events:
Login to iSCSI target iqn.2000-01.com.synology:ds1812.name2 on vmhba38 @ vmk1 failed. Target returned login error of: 0206.
Login to iSCSI target iqn.2000-01.com.synology:ds1812.name2 on vmhba38 @ vmk2 failed. Target returned login error of: 0206.
Login to iSCSI target iqn.2000-01.com.synology:ds1812.name2 on vmhba38 @ vmk2 failed. The iSCSI initiator could not establish a network connection to the target.
My setup is exactly the same as described exept one little detail: I have 2 extra dedicated NICS for this use, so I have enabled binding on both NICS, whereas you have only configured binding on one NIC. could that be my problem?
Regards Rasser
kb.vmware.com/kb/2001449 suggest that the 0206 error code implies that the array is out of resources. This might be caused if the array is over-provisioned by exporting too many targets or having too many initiators connecting to a single array.
Hi Cormac-
What if my storage array has ports on multiple subnets? For example, I have one array with two controllers. I’m using two storage ports on each controller. I would like to config to do multipathing and load balancing.
Controller One:
A0 – 10.1.100.1 – VLAN 1000
A1 – 10.1.101.1 – VLAN 1001
Controller Two:
B0 – 10.1.100.2 – VLAN 1000
B1 – 10.1.101.2 – VLAN 1001
I want to use four NICs on each ESXi host for iSCSI port binding. Should I do the following or do you recommend something else?
vSwitch1 for iSCSI:
vmk1: 10.1.100.10 – VLAN 1000 – assigned to vmnic3 only (using override failover order)
vmk2: 10.1.100.11 – VLAN 1000 – assigned to vmnic4 only (using override failover order)
vSwitch2 for iSCSI:
vmk3: 10.1.101.10 – VLAN 1001 – assigned to vmnic5 only (using override failover order)
vmk4: 10.1.101.11 – VLAN 1001 – assigned to vmnic6 only (using override failover order)
Create iSCSI software adapter and add the following VMkernel adapters under Network Configuration Tab under VMkernel Port Binding:
vmk1
vmk2
vmk3
vmk4
Will this work? I’ve been reading a ton of VMware documentation on iSCSI and it’s all over the place. I haven’t been able to find a real solid example that has a configuration like mine. Any help would be appreciated.
Thanks,
Scott
Also, this is in regards to ESXi 5.0.
I’m trying to do the same exact thing that you’re doing. Not having much luck. Did you ever figure it out?
Hi there, I discovered your website via Google at the same time as searching for a related topic, your site got here up, it seems great. I have added to favourites|added to bookmarks.