[Updated 29-April-2013] By default, prior to 5.0p5 & 5.1U1, an ESXi host has 80MB of VMFS heap at its disposal. This is defined in the advanced setting VMFS3.MaxHeapSizeMB. The main consumer of VMFS heap are the pointer blocks which are used to address file blocks in very large files/VMDKs on a VMFS filesystem. Therefore, the larger your VMDKs, the more VMFS heap you can consume. This is more true on VMFS-5, where double-indirect pointers exist to allow the unified 1MB file block size back a 2TB VMDK.
As a rule of thumb, we are conservatively estimating that a single ESXi host should have enough default heap space to address around 10TB of open files/VMDKs on a VMFS-5 volume.
If you change the advanced setting VMFS3.MaxHeapSizeMB to the maximum amount of heap (which is 256MB), we again conservatively estimate that about 30TB of open files/VMDKs can be addressed by that single ESXi host.
The point to keep in mind is that this is a per ESXi host setting – so each ESXi host can address this amount of open files/VMDKs per VMFS volume. Changes are coming to reduce heap consumption and allow a single ESXi host to address even larger VMDKs with less heap. However VMFS heap should be a factor that is taken into account when sizing deployments.
With the release on ESXi 5.0p5 in March 2013 & 5.1U1 in April 2013, both the default and the maximum size of the heap has been increased to 640MB. This means that the full 64TB of a VMFS volume may be addressed. Note that if you upgrade from a previous ESXi version, the maximum may remain at the previously configured setting and you may have to manually increase the size. For new installs, the new settings should be in place.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage
If you use an ESXi host with a large amount (say 24 * 3 TB disks) to serve a single VM which acts as a storage appliance, you may have greater than 30TB addressed by a single host, depending on RAID characteristics.That being the case, how should you address the VMFS3.maxHeapSizeMB calculation? Or is this inadvisable, or impossible?
well here it says 256MB equal 64TB
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004424
Hi Conor,
In this case, presenting those LUNs directly to a VM would necessitate the use of RDMs, which do not require VMFS heap. Therefore you do not need to take VMFS heap into consideration in that case.
Hi Karl,
Yes – my understanding is that this article is being reworked to include some of the guidelines in this blog post. Thanks for highlighting.
Cormac
Hi, this is great info! Could we clarify a bit what types of storage and disks are and are not affected by this? From the comments, it sounds like it does not affect RDMs…does that include both physical and virtual mode RDMs? Also, from the name of the option, can I correctly assume that it does not affect NFS storage?
Thanks!
-Loren
Correct on all counts Loren – this is applicable to VMFS only. It does not apply to vRDMs, pRDMs or NFS.
Has this limitation been addressed with ESXi 5.1?
Thanks,
Ron
Ron, the default VMFS heap in 5.1 is still 80MB as per KB 1004424. You can still increase the heap size to 256MB in 5.1.
I’ll take that as a “no”
. For us this means that we will need to change to using NFS on our new storage system (Nexsan NST5530). Of course there are other reasons for us to choose NFS over iSCSI, but this is a pretty good one. In our case we have virtualized all of our file services which collectively add up to about 44TB.
Out of curiosity, what happens if the ESXi host is presented with more open vmdk’s on vmfs-5 than it has RAM to manage? EG say the 64TB max of a single datastore/LUN?
Also, the KB should be updated to state that the vmdk limit of 25TB on ESXi 5.x is applicable to datastores formatted with vmfs-5. It seems to me that if the datastore is formatted with vmfs-3 and 8MB block size then the open vmdk limit could be 64TB since there is no memory penalty from the double indirect pointers imposed by vmfs-5. Correct?
Thanks,
Ron
If you have on a single ESXi host, and you require VMs to address 44TB of open VMDK simultaneously, then yes, NFS or RDMs would be advisable. Remember however that this is not a datastore limit but an ESXi host limit, So once you have two or more ESXi hosts addressing the VMFS datastore, the issue/restriction is mitigated. Regarding large VMFS-3 datastores, the issue is still there but not prevalent,
FYI there is a disscussion regarding this issue in the communities:
http://communities.vmware.com/message/2118608#2118608
It definitely seems that the issue is worked around by using larger block sizes. The 1MB block size of VMFS5 only has a max capacity of 25TB. But if you start with a VMFS3 with 8MB block and then upgrade then you can use much higher storage limits.
This article should be updated with the new maximums after the latest ESXi patches.