Some time back I wrote a blog post on enhancements to Storage vMotion in vSphere 5.0. One of these enhancements included the ability to Storage vMotion a Virtual Machine which had snapshots. Previously this was something we couldn't do, and was something a lot of customers were requesting, so this was a very welcome feature in vSphere 5.0. And Storage DRS with Virtual Machines that have snapshots also works, which is really cool.
However you need to be aware of one particular caveat with this operation in vSphere 5.0. There is a certain behaviour that Virtual Machines with snapshots have during a Storage vMotion operation in vSphere 5.0 which isn't very well documented. And this behaviour is also true for Storage DRS since it uses Storage vMotion for space utilization & load balancing operations.
workingDir parameter
To begin, many of you may be aware that the location of a virtual machine's snapshot redo log file is defined by the virtual machine .vmx file setting workingDir. By default, the workingDir property is the same as the virtual machine's home directory. However this could be set to another location/disk, especially if you wanted to create snapshots but did not have enough space on the volume where the VM base disk resided. This is described in detail in KB article 1002929.
New workingDir behaviour
In vSphere 5.0, the workingDir parameter still exists, but it will no longer be used as a pointer to a location for storing snapshot delta disks. The workingDir setting will now only be used to determine the location for the snapshot data (.vmsn) file. The delta disks are now always stored in the same home folder as the base disk. This is a significant change. Why are we changing the functionality of this setting in vSphere 5.0? Well, it really has to do with Storage DRS. This change means that snapshot delta disks now share the same performance, availability and storage consumption characteristics as parent disks, making Storage DRS work more accurately.
It should also be noted that if you do a Storage vMotion of a VM with snapshots and the VM has the workingDir parameter set, the workingDir setting will be removed from the .vmx & the .vmsn snapshot data file will be moved to the home folder of the VM on the destination datastore. You do get a warning in the migration wizard about this however (see below).
Can I override the new behaviour?
Yes you can, but there is a catch. If you really want to keep your snapshots on a different datastore to the base disk in vSphere 5.0, there is a new parameter which you must set along with the workingDir parameter. The new parameter, snapshot.redoNotWithParent = "TRUE", must be placed in the VM's .vmx file. This means that the workingDir parameter setting will now be used as the location for snapshot redo delta disks, as well as the .vmsn. However the same caveat described above applies in this case – the workingDir setting will be overridden by a Storage vMotion operation, and all snapshot deltas will get migrated to the same folder as the VM's base disk on the destination (so you better make sure you have enough space). Also, the workingDir setting will be removed from the .vmx file after the migration.
Therefore, if you use the snapshot.redoNotWithParent = "TRUE" parameter, you should refrain from doing Storage vMotion operations. The following warning will appear when a migration is attempted on a VM that has workingDir set in the .vmx file:
In fact, this warning also appears if you have just the workingDir setting in the .vmx file as mentioned earlier.
So, while Storage vMotion and Storage DRS both work fine with Virtual Machines that have snapshots, you should refrain from doing Storage vMotion operations with VMs that have the workingDir & snapshot.redoNotWithParent parameters set in their .vmx in vSphere 5.0.
And I would also recommend that VMs with these settings should not participate in Storage DRS either. If you want to use Storage DRS with VMs that have snapshots, then you should set aside enough space on the datastores for the VM's base disk and any snapshot delta disks which need to be kept in the VM's home folder.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: VMwareStorage
Adam Bokiniec
Hi Guys,
Thank you for this article but come-on, information like this where design of LUNs need to be re-designed need to be published in more that this article.
The change where VMs store the snapshots in the default home folder as the base disk is “Critical” to know about!!! Could you please have in mind for next version release to release a document that explains all changes in details where you specify significant changes and what to keep in mind, this particular changed caused our Exchange server with 1500 mailboxes downtime for two hours.
And when I then called support to investigate the reasons even the support engineer didn’t knew about this change! I love VMware but please better information in the future!
Regards
Adam
Chogan
Sorry to hear that this change impacted you in such a way Adam.
Let me reach out to our support guys, and see what can be done to raise the visibily around this change in behaviour.
Cormac
Mike
I’d like to second the alarm here, though I wasn’t impacted as strongly as Cormac. Centralizing snapshots onto high-performance datastores seems a natural method to organize and consolidate snapshot slackspace, and would seem analogous to the entire virtualization process.
Very inefficient!
Now all my datastores have to be expanded to account for potential explosion thanks to snapshots.