I know that many of you have reached out in the past about doing ptRDM to ptRDM migrations, especially in situations where you have a storage array refresh taking place in your environment. A ptRDM is a physical compatibility mode raw device mapping, to give it its full name. It is also known as a pass-thru RDM, in so far as we allows the virtual machine to issue ANY kind of I/O to the LUN. A whole bunch of comments requesting a feature like this appeared in the comments section of an RDM article I did last year. Obviously the ability to move those ptRDMs from one array to another array would be very desirable in those situations. I have taken this request for a ptRDM to ptRDM migration tool to our product managers and our engineering teams. This article will try to explain how it is impossible to create such a tool.
In SCSI, there is ‘state’ kept on the LUN. This ‘state’ on the LUN is tied directly to the initiator or Host Bus Adapter. The initiator is informed of the loss of any volatile state through SCSI Check Conditions, such as a Unit Attention. This is how the target informs the initiator on situations such as “what volatile state you previously changed for this LUN has been lost”. This is a trigger for the initiator to re-apply those changes. This could be anything from enabling write-thru caching or changing other operational features of the LUN. Since this ‘state’ is tied directly to the initiator and the initiator id is protocol specific (e.g. WWNN+WWPN for FC, iSCSI IQN name for iSCSI etc.) then this means that:
Now let’s take the primary use case of ptRDM, Microsoft Clustering. Loosing state would be an extremely bad thing to happen since one kind of ‘state’ tracked by a LUN is ‘who has the persistent reservation keys registered’ and ‘who holds a reservation’ (persistent or SCSI-2 type reservation). Not a good thing to loose track of.
You cannot vMotion a VM which uses ptRDMs in MSCS clusters. This is because you switch the VM to access the storage through another HBA/initiator, so state changes such as SCSI reservations might be lost.
You cannot Storage vMotion a VM which uses ptRDMs. Doing a Storage vMotion to another target for a live VM changes the target without informing the VM, and suddenly the LUN may now have a different model, vendor & INQUIRY string. This should never happen, not in physical hosts or virtual hosts.
The state is very much dependent on the array (and even firmware version), so for us to capture all that state correctly, we would first have to know what state we needed to retrieve and then reapply this for all of the different storage arrays on our HCL. We would also have to emulate around any quirks known for a target. In any operating system you will have workarounds for quirks with respect to different arrays (think of all the different settings needed on a storage array depending on the operating system which is connecting to it). The operating system will apply these workarounds based on INQUIRY information for example. But this is normally only done at boot time because in the physical world, devices don’t change. So if we were to do that in the virtual world, it would suddenly mean that even if we could move over all relevant state to the destination, then the quirks applied in the state for the old LUN are applied to the new LUN, even though they are not relevant for the new LUN. Conversely workarounds which should have been applied for the new LUN by the operating system have not been lost when the state from the old LUN is applied.
Of course there are other considerations too – what if the source LUN supported some fancy feature like snapshot, but the new LUN does not (or the SCSI command to initiate a snapshot was completely different)? It is impossible to do this correctly.
The bottom line is that this issue is an extremely tough one to crack because we cannot encapsulate all state from these external devices. My advice would be to move away from ptRDMs when possible. I understand that there are still situations in which ptRDMs are still necessary, such as the case of MSCS, but for those of you who use ptRDMs outside of MSCS, consider moving to a virtualized format, such as a virtual compatibility mode (non pass-thru) RDM or even VMDKs. The reason for this recommendation is the inherent lack of flexibility associated with ptRDMs, and lack of core vSphere features around these devices.
I believe that some customers use ptRDMs by default rather than “just when needed” because of – believed – simplicity, but reality is that they are limiting themselves feature-wise. Rather than expecting VMware to fix those limitations my recommendation is to avoid pRDMs when possible.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter:@VMwareStorage
