VMware Uptime Blog

I came across this blog last week regarding vSphere's changed blocked tracking (CBT) capability and was a bit concerned since it explicitly called out VMware Data Recovery (VDR) in the list of products affected.   As background, CBT is used by VMware products/capabilities like VDR and Storage vMotion to efficiently determine the amount of changed disk blocks.  I did some investigation on the topic to ascertain the extent of the problem - while I do agree that it exists, I am not convinced how often customers will see it in their production environment. 

In summary, you have to have the following ingredients:

• Using VDR (or any backup product that leverages CBT)
• Backing up a HW7 virtual machine

And then you have to do the following steps in sequence:

1. Perform a backup of the VM using VDR
2. Perform a manual snapshot of the VM
3. Perform another backup of the VM using VDR
4. Revert the manual snapshot
5. Perform another backup of the VM using VDR

What could end up happening is the snapshot revert in Step 4 causes the CBT change ID to decrement and become smaller, but CBT's assumption that change IDs will always increment and be a larger number.   This leads the incremental backup in Step 5 to not receive the correct list of changed blocks from CBT - the point that the blog was trying to convey. 

Can this be reproduced in a lab?  Absolutely.  

Does it happen frequently to matter to customers?  To really see this occur in the real world, there generally needs to be a few mini-steps between steps 4 and 5.  Things like more snapshot creates or deletes, multiple VM power off and on or VMotion operations.  These operations will generally decrement the CBT change ID - and increases the likelyhood of the mismatch to occur.

Should you be concerned?  You should at least be informed!   Some backup products already have workarounds implemented so the possibility of running into this scenario is eliminated.  The VMware team also created a KB article that summarizes the issues and potential workarounds.  As standard VMware practice, the KB will be updated as needed in the future with more information.