[Updated 29-April-2013] By default, prior to 5.0p5 & 5.1U1, an ESXi host has 80MB of VMFS heap at its disposal. This is defined in the advanced setting VMFS3.MaxHeapSizeMB. The main consumer of VMFS heap are the pointer blocks which are used to address file blocks in very large files/VMDKs on a VMFS filesystem. Therefore, the larger your VMDKs, the more VMFS heap you can consume. This is more true on VMFS-5, where double-indirect pointers exist to allow the unified 1MB file block size back a 2TB VMDK.
As a rule of thumb, we are conservatively estimating that a single ESXi host should have enough default heap space to address around 10TB of open files/VMDKs on a VMFS-5 volume.
If you change the advanced setting VMFS3.MaxHeapSizeMB to the maximum amount of heap (which is 256MB), we again conservatively estimate that about 30TB of open files/VMDKs can be addressed by that single ESXi host.
The point to keep in mind is that this is a per ESXi host setting – so each ESXi host can address this amount of open files/VMDKs per VMFS volume. Changes are coming to reduce heap consumption and allow a single ESXi host to address even larger VMDKs with less heap. However VMFS heap should be a factor that is taken into account when sizing deployments.
With the release on ESXi 5.0p5 in March 2013 & 5.1U1 in April 2013, both the default and the maximum size of the heap has been increased to 640MB. This means that the full 64TB of a VMFS volume may be addressed. Note that if you upgrade from a previous ESXi version, the maximum may remain at the previously configured setting and you may have to manually increase the size. For new installs, the new settings should be in place.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage
