vSphere ESXi Microsoft vMotion vSAN vSphere vVOLs

Shared-Disk Clustering on vSphere: Getting Out of the “Multi-Writer Flag” Jam

In spite of our best efforts to amplify the negative implications of (and actively trying to discourage them from) doing so, it turns out that a lot of VMware Customers have been getting very “creative” and configuring their production Microsoft SQL Server instances for shared-disk clustering (aka Always On Failover Cluster Instance – “FCI”) on vSphere by leveraging what appears to be a “work-around” feature – the Multi-Writer Flag.

The Multi-writer flag option was created specifically to support the shared disk requirements of virtualized Oracle RAC instances on the vSphere platform, and although it has since been extended to support other use cases, it has never been recommended nor supported for use with Windows Server Failover Cluster (previously known as Microsoft Cluster Service (MSCS)). Unfortunately, there are publications out there demonstrating how to achieve this unsupported and dangerous configuration, and some Customers have followed these recommendations. These reccomendations not only expose Customers’ systems to the possibility of data corruption, it also leaves their workloads in unsupported state, meaning that, should these Customers require technical support from VMware, the unsupported configuration severely constrains how much assistance can be rendered to the Customers in their time of needs.

It is important to stress again:

The only supported ways to use VMDKs as shared-disks for Windows VMs in a vSphere environment is when the VMDKs are stored in a Clustered VMDK-enabled Datastore, as described in Clustered VMDK support for WSFC, or in the “Storage Configuration” section of this Article

The rest of this Post is focused on documenting how Customers who have mistakenly followed some of the erroneous guidance previously mentioned can correct the error and move their workloads into an optimal and supported configuration.

First, the warnings and caveats:

Before attempting any of the steps deocumented in this post, it is very important for the Operator to understand and accept that VMware does not warrant nor guarantee that these configuration willl not result in data loss or corruption. These steps, when carefully performed, have been known to achieve the stated objectives of:

  • Reconfiguring VMDK-backed shared disks for clustered Windows VMs which have been configured to use the multi-writer flag option
  • Relocating the VMDKs into one or more officially supported “Clustered VMDK”-backed datastore(s)
  • Presenting the VMDKs back to the VMs in a controlled manner to minimize or avoid the need for in-Guest or application-level reconfiguration

VMware strongly recommends that Customers performing these tasks ensure that they have a proven, repeatable roll-back plan to restore service and establish continuity in case of failure during the performance of these exercises.

It is assumed and expected that Customers have known good backup copies of all the data and configuration information of all the VMs which will be part of this reconfiguration exercise.

At a minimum, Customers should perform (or note) the following before starting these procedures:

  • The CURRENT configurations of the VMs, especially:
    • The Disks – which VMDKs corresponds to which volumes in the guest operating system
    • The names and location of the files backing EACH VMDK
    • The SCSI number and SCSI ID to which EACH Disk is connected. We will need to attach the disk to the SAME SCSI ID when we re-attach it
    • In Windows, the CURRENT Owner of the Disk Resources (verify this in WSFC)
  • If the WSFC Resource ownership is split among/between Nodes, FAILOVER ALL THE RESOURCES to one Node. This simplifies the reconfiguration process and is very helpful in avoiding confusions and mix-ups.
  • Power off all Passive Nodes BEFORE you power off the Active Node.
  • After you are done, you will need to power on the Active Node FIRST before powering on the other Nodes

Let’s begin by taking a look at our current configuration:

Figure 1- Current WSFC Configuration (Nodes)

 

Figure 2- Current WSFC Configuration (Disks)

 

Figure 3- Testing WSFC by failing disk resources over – in this case, we powered off the Active Node and watched the clustered resources come online on the Passive Node. This test is very important to establish the health of WSFC prior to making these changes.

 

Figure 4 – Current shared-disk configuration (depicting a common misconfiguration where shared disks belong to a 3rd, powered-off VM). How the disks are shared out is unimportant, so feel free to ignore this bit of information.

 

Figure 5 – Here is WSFC Node-1 sharing disks (With Multi-Writer Flag Enabled)

 

Figure 6 – Here is WSFC Node-2 sharing disks (With Multi-Writer Flag Enabled)

 

Figure 7- View of the shared disks (in Windows). Note that disks are ACTIVE and seen ONLY on one Node (FC2)

 

Figure 8 – View of the shared disks (in Windows) . We created test folders. When we are done with the reconfiguration, we will check for these folders, to demonstrate/verify zero (“0”) data loss after the changes. Ideally, we would test the workloads (e.g. Databases), but this is beyond our scope for this post.

 

Figure 9 – Shared disk view in Windows Disk Manager. Notich that Disk1 1 and 2 are ONLINE only on FC02. We have verified the health of our clustered resource (Disks).

 

Now, we are ready to perform disk reconfiguration in vCenter. We need to power off the VMs BEFORE proceeding with these steps

Figure 10 – In case you are wondering why we cannot just simply perform a storage vMotion operation on the disks and call it a day, the answer is that the process will fail because we’re sharing the disks.

 

Let’s demonstrate this anyway, for academic purposes.

Figure 11 – Let’s choose Storage vMotion

 

 

Figure 12 – Notice the compatibility warning???

 

Figure 13 – If we click on ‘Details’, we see why. The process will fail. We don’t want this.

 

End of academic curiosity exercise. Let’s move on.

Figure 14 – So, we DETACH (NOT DELETE) the disks from the WSFC Node VMs

 

Note: Please do NOT DELETE the disk(s). If you do, they are gone. We want to DETACH instead.

Figure 15 – We do the same on the second WSFC Node VM

 

Figure 16 – On the VM where the disks reside, we remove the ‘Multi-Writer’ flag from the disks

 

Figure 17 – Then, we set the SCSI Controller’s compatibility to ‘None’

 

Figure 18 – Now, we can migrate the disks

 

Figure 19 – Go through the steps
Figure 20 – Click on ‘Configure Per Disk’, select the shared disks, then click on ‘Configure’

 

Figure 21 – Select the target Clustered VMDK Datastore, Select the desired Storage Policy, set the disk format to ‘Thick Provision Eager Zeroed’, then click ‘Confirm’ and finish the configuration

 

Figure 22 – We still have the VM’s OS disk In Its original location, but the shared disks are now located in the Clustered VMDK Datastore

 

Figure 23 – Here are the shared disks In their new location

 

We are now ready to re-attach the disks to the WSFC Node VMs

Figure 24 – First, make sure that the Disks are configured as follows on the “Owner” VM

Notes:

  • Sharing: No sharing
  • Disk Mode: Independent Persistent
  • Note The Virtual Device Node SCSI ID
  • SCSI Bus Sharing: Physical
  • These Settings MUST be identical on the WSFC Nodes when we attach the Disks
Figure 25 – On the WSFC Node VM, add an existing Hard Disk

 

Figure 26 – Browse to the Clustered VMDK Datastore and select the file corresponding to the disk

 

Figure 27 – Repeat the steps for ALL shared disks

 

Figure 28 – Set the parameters as noted previously. These settings MUST be identical

 

Repeat the process for the other WSFC Node VMs

Figure 29 – Set the parameters as noted previously (you wrote them now before starting the exercise, right? :)). These settings MUST be identical

 

That’s it. Now, we power on the VM that used to be the ACTIVE node. Then, we log into Windows and wait for everything to start up. You can now power on the other Node VM(s)

Figure 30 – Let’s check out the disks in Windows Disk Manager

 

Figure 31 – Let’s check them in Failover Cluster Manager

 

Figure 32 – Let’s check for data loss. Are our folders still intact?

 

Figure 33 – All looks OK. Of course, at this point, we will be doing more than just checking for folders.

 

That’s it. This concludes our demonstration. We have reconfigured our misconfigured Windows Server Failover Cluster shared-disk configuration. We have removed its dependence on the unsupported “Multi-Writer Flag” band aid and moved the disks into the supported “Clustered VMDK” datastore option. We have done this without causing a major reconfiguration of Windows, WSFC or the clustered workloads.

Of course, we wished that we didn’t have to do all of this to begin with. However, we are glad that we are now in a happy (desired) place.

Related Articles