VMware

vSphere Storage Appliance (VSA) - A Very Nice Resilience Feature | Main | vSphere 5.0 Storage Features Part 7 - VMFS-5 & GPT

08/05/2011

2TB VMDKs on Upgraded VMFS-3 to VMFS-5. Really?

Yes! :-)

I've had this query a number of times now, so I decided to do a quick note about it. There seems to be some uncertainty around whether or not you can create larger (2TB - 512 byte) VMDKs on VMFS-3 filesystems that have been upgraded to VMFS-5. The answer is "yes you can". I think this uncertainty is arising because the upgraded VMFS-5 continues to use the previously defined file block size of the VMFS-3, and that block size had a bearing on the size of the VMDK that you could create, i.e.

  • 1MB file block size - max file size is 256GB
  • 2MB file block size - max file size is 512GB
  • 4MB file block size - max file size is 1TB
  • 8MB file block size - max file size is 2TB

And I guess folks are thinking that because we maintain this original file block size in the upgraded VMFS-5, this still has relevance to the maximum size of the VMDK. But this is not true. Let's run through a scenario:

Step 1 - Let's build a VMFS-3 with a 2MB file block size to begin with:

Vmfs-3

Note also that VMFS-3 volumes (single extent) are limited to 2TB.

Step 2 - Let's build a VM with the largest possible VMDK on this VMFS-3. The largest possible VMDK should be 512GB as limited by the block size.

Vmfs3-largest-vmdk

And it is. I can't increment this value any higher that 512GB. This implies that the maximum VMDK size on a VMFS-3 is limited by the file block size as mentioned previously. To get larger VMDKs, you need to build the VMFS-3 with larger block sizes. To get a 2TB (- 512 byte) VMDK, you need to use a 8MB file block size when creating the VMFS-3 datastore.

Step 3 - Let's upgrade this VMFS-3 to a VMFS-5. This is a very simple operation. Just select the Host > Configuration tab > Storage > Datastore, and if the datastore is VMFS-3, a link to Upgrade to VMFS-5 appears automatically. You simply click this to do the Upgrade.

Upgrade-vmfs3-to-vmfs5

After the upgrade, please note that the VMFS-5 continues to maintain some attributes of the VMFS-3, including the File Block Size of 2MB. Next we will see that even though these characteristics are maintained, you can still go ahead and create much larger VMDKs than before. Here are the datastore details after it has been upgraded to VMFS-5.

Upgraded-vmfs5

Step 4 - In order to build a much larger VMDK, I am first going to grow the VMFS-5 to a size larger than 2TB. Remember that VMFS-3 limited the maximum size of a single extent to 2TB. With an upgraded VMFS-5, you can have a much larger single extent datastore size. Simply select the Properties link of the datastore, and click on the Increase Size button & follow the wizard steps. In this example I have grown my single extent (upgraded) VMFS-5 to 3TB.

3tb-vmfs-5

Step 5 - This is the final step, and is really just to show you that an upgraded VMFS-3 to VMFS-5 datastore which has a 2MB file block size can handle the creation of very large (~2TB) VMDKs. Let's now build a new VM with the largest possible VMDK on this datastore. As you can clearly see, I can now select a 2TB VMDK for this new VM, using the same upgraded VMFS-5.

2tb-vmdk

Therefore, although the upgraded VMFS-5 continues to maintain many of the original characteristics of the VMFS-3, it can also avail of the new VMFS-5 features such as larger single extent volumes, and of course much larger VMDKs.

However, having said all that, and returning to something I alluded to in my original VMFS-5 host, if you have the luxury of deploying brand new volumes, and then Storage vMotion'ing your VMs to the newly created VMFS-5 datastores, then that would be the most preferred method of upgrading to VMFS-5.

Comments

Feed You can follow this conversation by subscribing to the comment feed for this post.

Thanks for the insight. Although 2TB+ VMDKs are not always that useful, I suppose this means legacy volumes with 1MB blocks will show similar flexibility, right? Would this hold true without expanding the LUN and simply migrating to VMFS5?

As an aside, would it be better to migrate to VMFS5 prior to svMotion'ing your VMDKs to newly created VMFS5 LUNs? In other words, would this have any benefit with respect to VAAI acceleration in the move where certain features would be previously disabled due to the differing block size of the target?

Cheers, and thanks again for the insight!

Hello Colin,
Thanks for the comment. To answer the first part, yes, all you need to do is upgrade the VMFS-3 to VMFS-5, and you have the 2TB+ VMDK capability - nothing else is required, even for 1MB file block sizes.

Your second comment is a very good one - there are indeed some considerations to take into account with file block sizes and Storage vMotion. This is discussed in a great blog posting by my colleage Duncan over on Yellow Bricks - http://www.yellow-bricks.com/2011/02/18/blocksize-impact/
The considerations he discusses are still valid.

What is the max size of a VMDK with vmfs 5?

The max size of a VMDK with VMFS-5 continues to be 2TB - 512 bytes.

Lame. :(

You can use physical compat (pass-thru) RDMs up to ~60TB in 5.0 if that's any help?

Is it possible to upgrate VMFS 3.33 datastore to VMFS5 on vsphere 4 environment?

Hi Bals, no it is not possible, and it would not make any sense to do so. In order to see VMFS-5 volumes, a host must be running ESXi 5. So upgrading to VMFS-5 can only be done when all hosts accessing that volume are running ESXi 5.0.

Thank You Chogan!

Colin, that was very interesting.

If step 4. (growing the datastore) was skipped, my guess is that the maximum size will remain 512 since the format will remain MBR. Do you agree?

I think that the format only changes to GPT once the size of the datastore is increased from 2TB, and the change to GPT format is what allows the 2TB -512 max size?

--Max

Hi Max,

Once the VMFS-3 has been upgraded to VMFS-5, the max VMDK size becomes 2TB - 512 bytes, no matter what the original block size.

It is only when the datastore (not the VMDK) is grown above the 2TB mark that the switch from MBR to GPT takes place.

Obviously, when you grow the VMDK to 2TB, you'll need an underlying datstore to handle the bigger size, so in all likelihood you will be growing both the datastore and VMDK.

However it is the datastore size changing to > 2TB which triggers the partition table format change.

Cormac

I'm not sure, but if I understand, if you have a .vmdk file in VMFS-3 format, with one of this size (depending on block size), then you have:

1MB file block size - max file size is 256GB
2MB file block size - max file size is 512GB
4MB file block size - max file size is 1TB
8MB file block size - max file size is 2TB

If you update VMFS-3 to VMFS-5, independently the block size that this VMDK file have in VMFS-3, now in VMFS-5 format this file can grown up to 2 Tb minus 512 bytes, no more than this size.

But if you want have a VMDK file over 2 Tb, you need to created new virtual disk in VMFS-5 format directly, with all benefits of this new file format.

This is true?

Hi Jose,

Everything is correct until your last statement. You cannot have a VMDK larger than 2TB, even on VMFS-5.

Cormac

Post a comment

If you have a TypeKey or TypePad account, please Sign In.