Let’s start by describing what this feature is. NPIV stands for N-Port ID Virtualization. It is an ANSI T11 standard that describes how a single Fibre Channel Physical HBA port can register with a fabric using several worldwide port names (WWPNs), what might be considered Virtual WWNs. This in turn means that since there a multiple Virtual HBAs per physical HBA, we can allow WWNs to be assigned to each VM.
At first glance, this might sound kind of cool. But I’m going to be blunt with you here – I still don’t see the value of this feature in its current form. The steps required to set this up are pretty complex and time consuming, and the only advantage I see from VMs having their own WWNs is possibly Quality of Service (QoS) measurement. With each VM having its own WWN, you could perceivably track Virtual Machine traffic in the fabric if you had the appropriate tools. Also, you may get more visiblity of VMs at the storage array level. But you have to use Raw Device Mappings (RDMs) mapped to the VM for NPIV, which means you do not get all the benefits associated with VMFS and VMDKs.
Having said that, I still see queries about how to configure NPIV from time to time. Therefore I decided to put the configuration steps into this post for reference. If there are readers who are considering using this feature (or have already implemented it), I’d really appreciate it if you would leave a comment and share with us the reason why you are using NPIV. Maybe I’m missing something obvious.
As you might see from the screenshots that I am using, the implementation is not being done on vSphere 5.0 but on a much earlier version of vSphere. However the steps haven’t changed one bit since we first brought out support for NPIV back in ESX 3.
Initial Configuration Check
First, you’ll need the following:
- Administrator access to your vSphere environment
- Administrator access to your FC Switch(es)
- Administrator access to your FC Array
Next, you’ll need to make sure that the HBA that you are using supports NPIV. You can find this information on the VMware HCL, but I should think that any HBAs purchased in the last 3 years will support NPIV out of the box. Best to double check though.
Same goes for the FC Switch. Earlier FC switches may not support NPIV either, so best to check that too.
Lastly, ensure that one or more LUNs have been presented from the array to your ESX host(s) over FC, and that these LUNs are visible. This will also prove that your FC zoning is done correctly.
Virtual Machine Configuration
Now we’re ready to being the configuration. Select the VM that will use NPIV and then go to the VM Properties, select the Options tab, and check the Fibre Channel NPIV setting:
As you can see, there are No WWNs currently assigned. Next, we need to map an RDM to the VM in order to enable NPIV. With the RDM mapped in Physical Compatability Mode (PassThru), return to the VM Properties Options tab and Fiber Channel NPIV & select the option to Generate new WWNs. After doing this step, a WWNN and 4 WWPNs should now be visible in the WWN Assignments view:
If you now examine the VMX file belonging to this VM, you will notice a number of new entries added:
wwn.node = “2833000c2900000b”
wwn.port = “2833000c2900000c,2833000c2900000d,2833000c2900000e,2833000c2900000f”
wwn.type = “vc”
That completes the VM configuration.
FC Switch Configuration
On the FC switch, you will now have to create zones from the VM’s NPIV WWPN to the Storage Array ports WWPN. However, the VM’s NPIV WWPNs are currently not active, so they do not appear in the nameserver on the FC switch. Therefore they will need to be added manually. Once they are manually added to the FC switch, the WWPNs can then be placed in zones with the WWPN of the Storage Array ports.
In this example, the WWNN & WWPNs of the VM have all been added to the same alias as the physical Qlogic HBA. This is by no means a best practice, but the advantage of doing it this way is that the NPIV WWNs will automatically be participating in the same zones as the physical WWN from the HBA.
Ideally, you would seperate the physical HBA WWPN and the VM’s NPIV WWPN into seperate alisaes. This would make management much easier. However, in this case, it was a quick way of allowing the NPIV WWNs be zoned to the same storage array as the physical QLogic HBA.
So at this point, the NPIV WWNN and WWPNs from the Virtual Machine are added to the FC switch and now are zoned to the FC ports of the storage array. That completes the FC Switch configuration.
Storage Array Configuration
Things now start to become interesting. This procedure can vary from array to array, but basically the objective is to assign disks from the array to the NPIV WWNs of the Virtual Machine. The examples included here are taken from an EMC Symmetrix array using SYMCLI, and from an EMC Clariion using the Navisphere UI.
a) EMC Symmetrix
In this example, the physical LUNs are known by device ids 0xF & 0×14. These device are masked to the NPIV WWN’s that were created in the Virtual Machine:
C:\….\bin>symmask -sid 1121 -wwn 2833000c2900000b -dir 15B -port 0 add devs F,14
The following devices are already assigned in at least one entry:
000F 0014
Would you like to continue (y/[n])?y
We now need to refresh the masking information on the Symmetrix:
C:\Program Files\EMC\SYMCLI\bin>symmask -sid 1121 refresh
Refresh Symmetrix FA directors with contents of SymMask database 000281601121 (y/[n]) ? y
Symmetrix FA directors updated with contents of SymMask Database 000281601121
Finally you can review the device masking on the Symmetrix and see the Dev F & 14 masked to the Physical HBA’s and the NPIV HBA’s of the Virtual Machine:
C:\Program Files\EMC\SYMCLI\bin>symmaskdb list database -sid 1121 -dir 15B -port 0
Symmetrix ID : 000281601121
Database Type : Type1
Last updated at : 03:44:50 PM on Thu Nov 29,2007
Director Identification : FA-15B
Director Port : 0
User-generated
Identifier Type Node Name Port Name Devices
—————- —– ——————————— ———
210000e08b949fef Fibre 210000e08b949fef 210000e08b949fef 000F
0014
210100e08bb49fef Fibre 210100e08bb49fef 210100e08bb49fef 000F
0014
2833000c2900000b Fibre 2833000c2900000b 2833000c2900000b 000F
0014
2833000c2900000c Fibre 2833000c2900000c 2833000c2900000c 000F
0014
2833000c2900000d Fibre 2833000c2900000d 2833000c2900000d 000F
0014
.
.
b) EMC Clariion
If on the other hand, you are doing this on an EMC CX array, you need to first open the Navisphere UI, select the object representing the array and select Connectivity Status:
We can see the 2 FC HBA’s logged in and Registered with the Storage Processors A & B of the Clariion. We now add the new initiators from the Virtual Machine, i.e. the NPIV WWNs. The Clariion expects that initiators are added in the format <Node WWN>:<Port WWN>. Note that in adding the initiator, we are automatically associating it with a host previously registered with the physical HBA. This has the added advantage of automatically adding the NPIV WWNs from the VM to whatever storage groups that the physical HBA is already a member of. This means that the LUN assignments wihin that storage group will automatically be presented to the NPIV WWNs too.
Once the NPIV WWNs of the VM have been added, all the initiators show up in the Connectivity Status as follows. Note that they are all associated with the same server name for reasons mentioned previously.
Note: You might find that you have to remove and re-add the host to the Storage Group for the new initiator groups to be picked up. Don’t know the reason, just adding this point in case you run into it.
Now all that remains to be done is to power on your VM. The RDM should now appear mapped to your VM, but using its own set of NPIV WWNs. The configuration is now completed.
Troubleshooting
You will want to read this section, believe me!
First, check that all the configuration tasks have complete correctly:
Configuring the Virtual Machine
- Add and RDM disk
- Create NPIV Node and Port Names
Configuring the Fibre Channel Switch
- Adding the WWN’s to the HBA Alias
- Checking the Zone and Zone Set information
Configuring the Storage Array:
- Create Initiator groups if necessary
- Present/Mask LUNs to the VM’s NPIV WWNs
If all of these tasks completed correctly, but the LUN is not being successfully presented, the following places should be checked for troubleshooting:
FC Switch – on the Brocade, a useful script will allow you to monitor whether or not the NPIV WWNs are logging in (X & Y are the ports where the physcial HBAs are logging into the switch:
while true; do portloginshow X; portloginshow Y; done
ESX host – Check /proc/scsi/qla2300/X and /var/log/vmkernel for messages pertaining to the NPIV initialization. X represents a HBA. The proc node will show both physical and virtual adapter information.
A useful, but quite old white paper on NPIV, can be found here.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage
Our production data(not the os)live on our SAN, not within virtual disks. This makes it much easier to go from physical to virtual and back again if so desired.
Thanks for the comment.
Since you are already using PT-RDMs (I assume), then that makes NPIV easier to implement.
But what benefit do you get by moving between physical HBA WWNs & NPIV WWNs in your environment?
We currently are exploring the idea of using NPIV for VMware guest as we have a complicated DR scenario that requires the LUN mgmt at the guest level. We have standalone hardware that performs all this functionality and segregates the duties of our admins.
NPIV was chosen as it allows our hosts to look like “standalone” servers and all the scripts do not need to be re-engineered.
Hi, I gess you could use NPIV to virtualize your backup server wich is using FC tapes. Using NPIV allow you to zone your tapes to your server as well as vmotion it. I’m not sure if this is an supported environment since backup servers are pretty much I/O consumer for an IT environment. Also, how the backup software will behave while you are issuing a vmotion task?
Cheers,
Davi
Hi Davi,
While the idea is very plausible, I don’t believe we support using NPIV outside of disk presentation. In the current implementation, you must first map an RDM to your VM, and then you can use NPIV for that disk. I’m not sure how you would go about this for tapes. Interesting idea though.
And yes, getting the tape drive mapped would be just one part of the puzzle. Ensuring that the backup software supports the config is another major part.
Is using FC tapes supported in vsphere? This may not be true?
Hi FY,
Support for FC tape drive connectivity to an ESX host (outside of NPIV) is an arrangement made directly with partners. For example, this KB article (kb.vmware.com/kb/1017757) describes how to setup HP StorageWorks D2D & VLS virtual tape libraries on ESX 4.x. Please carefully read the support statements in this KB if you plan on doing such a configuration.
Hi, I am proposing to use NPIV to virtualise our file server infrastructure. What are you thoughts on the matter? When provisioning storage in Vcenter and you configure the whole useable space it naturally gives the alarm errors which tend to be annoying – but the idea of having storage directly provisioned from the SAN rather than via vmdisks gives much better management of SAN related capacilities such as flash copies etc. and helps with backup over FC.
I am interested to hear what you think about this?
Hi Ali,
I would first evaluate whether RDMs could meet your needs. This will make the solution much simpler. If not, then you can pursue an NPIV solution, but be aware that the setup is much more complex.
HTH
Cormac
Hi Chogan,
> I don’t believe we support using NPIV outside of disk presentation
Do you know if this means that a virtual HBA in a VM can’t be told its port is a “target” instead of an “initiator”?
Thanks!
Eric
Hi Eric,
No, I do not believe that is possible.
Sorry
Cormac
Hi Cormac.
That kb article about HP StorageWorks (kb.vmware.com/kb/1017757) is no longer available.
Do HBA’s show up in device manager of the guest OS?
Hello d_
Not sure why the KB is no longer available, but to answer your question, no you will not see a virtual HBA in your VM. Therefore you do not have to worry about installing a HBA driver (e.g. Emulex/QLogic) in your Guest OS.
The NPIV configuration on a VM is done through the options screen shown above. There is no virtual hardware for NPIV.
Cormac
Would RDM/NPIV be the only way to do chargeback billing for storage without having to use vCenter Chargeback? I have a customer that wants to charge guests based on GB/tier.
With pRDM we can use array based snapshots but existing snapshots cannot be mounted back by storage proprietary tools like EMC Replication Manager. That’s because SCSI representation of FC disk. Would NPIV help us to mount array based snapshot back to the virtual machine? I had no chance to test it so far but it would be big benefit of NPIV if it is practically achievable.
Hi David,
There are various methods to allow a snapshot copy of a LUN to be presented to the same or different ESXi hosts (resignature or allowsnapshots). I may not be understanding your question correctly, so I would suggest raising a Service Request with our GSS support folks. Then you will be able to speak to one of our storage experts around the options available to you to address this.
Cormac
The first 3 images are not showing up.
fixed now – thanks for highlighting
What’s needed is a way for RDM disks to move with a VM during Vmotion. As far as I can determine, neither ESXi/vSphere nor PowerPath/VE can do this. Assuming the zoning is done right, I’d like to think that NPIV at least offers the possibility of keeping SCSI access for an RDM LUN tied to a VM’s host hardware, regardless of where Vmotion puts the VM.
So we can do this already – check out this post: http://blogs.vmware.com/vsphere/2012/02/migrating-rdms-and-a-question-for-rdm-users.html
I’m thinking more along this line:
“When a virtual machine is migrated from one host
to another, the VPORT is closed on the first host and opened on the destination host.”
http://pubs.vmware.com/vsphere-50/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-50-storage-guide.pdf
I have a test setup running with EMC’s PowerPath/VE, and after completing a migration of a VM, the VM’s Storage View continued to show the original host’s HBA’s; however, it eventually changed over to the current host’s HBA’s. This is exactly what’s wanted.
Default ESXi 5 multipath does the same thing. With a single VNX Storage Group containing all of the hosts in a VMware cluster, the RDM’s stay with the VM’s; i.e., they move to an HBA on the target host. This is the case with both VMware’s default multipath plug-in and with EMC’s PowerPath/VE plug-in. Reading about this stuff didn’t help, but now I’ve seen what I want to see and it’s good.
Is it possible to have a virtual HBA in a guest OS, with the full functionality as an physical HBA? We are setting up some SAN exercises and don’t have that much physical hosts for all our students. Creating NPIV ports and presenting them as a virtual FC HBA to a guet os seemed a good solution. The students need to display the WWN and configuring the zoning on a FC switch. Which will give them the possibility to mount the SAN storage volume in their VM. Is that possible with vmware and NPIV?
So you will not see a piece of virtual hardware representing a HBA in a VM with NPIV Phil.
It would be nice to have this, but that is not how NPIV is implemented at this time.
Right now, as the article states, LUNs are mapped to the ESXi hosts, then they are presented to the VMs as RDMs. From there, you can setup NPIV and have them mapped directly to the VM from the fabric.
So I’m not sure this meets your students requirements.
HI,Davi,great artical!
well , i wonder if the LUN mapped through NPIV to the virtual machine could be larger than 2TB ? for example 100TB?
is there any size limitation on this ?
It would be limited to the maximum size of an RDM, which is currently 64TB.
Hi, as you might now there are problems with VM’s running Windows2003/2008 and MSCS services. MSCS uses RDM’s and there is a SCSI-reservation that couses the ESX to have very long boot-times and rescan HBA-times.
There are some command to run but they do not work, prevent ESX to look at the RDM/LUNs at boot etc etc.
Would NPIV and pRDM be different in terms of SCSI-reservation problems ?
Not sure sins we still use RDM’s and that they are provisioned to both the ESX and the VM ( EMC VNX storagegroup )
Regards, Jonas
Hi
We study the implementation of a geocluster High Availability RedHat on physical servers. It consists of 2 nodes distributed on 2 sites. Every node reaches his own LUN with a replication SRDF between both sites. When a node becomes active, it switch the LUN from the mode WD to RW by using commands Symcli from the OS
We study the same architecture with virtual servers. My question looks like that of the Phil. Is it possible to switch the LUN from the mode WD to RW by using commands Symcli from the OS of the virtual machines via NPIV?
Thank you for your help
You can certainly do this. If you have an RDM mapped to a VM (be it a standard RDM or via NPIV), and you set the state to WD, it will appear as a read-only disk to the Guest OS.
Hi.
I’m not sure my question was clear enough.
Can I use a symcli command (shell script) from the inside of the guest OS VM (redhat) to switch the LUN state (WD or WR) ?
best regards
Yes – SYMCLI commands can be issues to a device from within a Guest OS.
2013-05-13T07:42:12.978Z cpu4:2661)ScsiNpiv: 1525: GetInfo for adapter vmhba3, [0x4100070cbd40], max_vports=127, vports_inuse=0, linktype=0, state=1, failreason=0, rv=0, sts=0
2013-05-13T08:09:02.683Z cpu4:4878)ScsiNpiv: 1718: Physical Path : adapter=vmhba3, channel=0, target=1, lun=3
2013-05-13T08:09:02.786Z cpu4:4878)WARNING: ScsiNpiv: 1348: Created vport for world 4879, vmhba3, rv 0
2013-05-13T08:09:02.786Z cpu4:4878)ScsiNpiv: 1149: NPIV vport rescan complete, [1:3] (0x4100070cf500) [0x410011c06b50] status=0xbad0003
2013-05-13T08:09:02.786Z cpu4:4878)WARNING: ScsiNpiv: 1806: Failed to Create vport for world 4879, vmhba3, rescan failed, status=bad0001
2013-05-13T08:09:02.992Z cpu4:4878)WARNING: ScsiNpiv: 1348: Created vport for world 4879, vmhba3, rv 0
2013-05-13T08:09:02.992Z cpu4:4878)ScsiNpiv: 1149: NPIV vport rescan complete, [1:3] (0x4100070cf500) [0x410011c07c10] status=0xbad0003
2013-05-13T08:09:02.992Z cpu4:4878)WARNING: ScsiNpiv: 1806: Failed to Create vport for world 4879, vmhba3, rescan failed, status=bad0001
2013-05-13T08:09:02.996Z cpu4:4878)ScsiNpiv: 1867: Vport Create status for world:4879 num_wwpn=2, num_vports=0, paths=1, errors=1
2013-05-13T08:09:03.883Z cpu1:4891)ScsiNpiv: 1718: Physical Path : adapter=vmhba3, channel=0, target=1, lun=2
2013-05-13T08:09:04.087Z cpu2:4891)WARNING: ScsiNpiv: 1348: Created vport for world 4892, vmhba3, rv 0
2013-05-13T08:09:04.088Z cpu2:4891)ScsiNpiv: 1149: NPIV vport rescan complete, [1:2] (0x4100070cedc0) [0x410011c06b50] status=0xbad0003
2013-05-13T08:09:04.088Z cpu2:4891)WARNING: ScsiNpiv: 1806: Failed to Create vport for world 4892, vmhba3, rescan failed, status=bad0001
2013-05-13T08:09:04.299Z cpu2:4891)WARNING: ScsiNpiv: 1348: Created vport for world 4892, vmhba3, rv 0
2013-05-13T08:09:04.300Z cpu2:4891)ScsiNpiv: 1149: NPIV vport rescan complete, [1:2] (0x4100070cedc0) [0x410011c07c10] status=0xbad0003
2013-05-13T08:09:04.300Z cpu2:4891)WARNING: ScsiNpiv: 1806: Failed to Create vport for world 4892, vmhba3, rescan failed, status=bad0001
2013-05-13T08:09:04.312Z cpu2:4891)ScsiNpiv: 1867: Vport Create status for world:4892 num_wwpn=2, num_vports=0, paths=1, errors=1
2013-05-13T08:18:54.332Z cpu1:2080)ScsiNpiv: 1525: GetInfo for adapter vmhba38, [0x4100070654c0], max_vports=0, vports_inuse=0, linktype=0, state=0, failreason=0, rv=-1, sts=bad0020
I configurationNPIV have error, configuration fails, please help