The vSAN Witness Appliance was introduced as a free alternative to using a physical ESXi host as a vSAN Witness Host. This appliance is only used for housing vSAN Object Witness Components, and is not allowed to run any virtual machine workloads.
When deploying the vSAN Witness Appliance, there are 3 potential deployment options: Tiny, Normal, & Large. These deployment profiles all have 2 vCPUs, 1 vmdk for the ESXi installation, 1 vmdk for the vSAN cache device, and at least 1 vmdk for the vSAN capacity.
Typical Witness Appliance Requirements for 2 Node vSAN
When deploying 2 Node vSAN, a vSAN Witness Appliance is typically deployed with the “Tiny” profile.
A closer look at the Tiny profile shows the following requirements:
- vCPUs: 2
- vRAM: 8GB
- ESXi installation vmdk: 12GB (as of 6.5, previous were 8GB)
- vSAN Cache vmdk: 10GB (this is common across profiles)
- Does not ever need to be larger than 10GB
- Does not have to be backed by a flash device, but must be tagged as a flash device
- vSAN Capacity vmdk: 15GB
- Can be tagged as flash, but not required
- Management VMkernel Interface (vmk0)
- Must be reachable by vCenter for Management
- Can be tagged for vSAN Traffic if the WitnessPg is not.
- WitnessPg VMkernel Interface (vmk1)
- Cannot be on the same network at the Management Network (KB 2010877)
- If vSAN Traffic is tagged, must be able to communicate with vSAN Data Nodes using vSAN VMkernel Interface or alternate VMkernel Interface when using Witness Traffic Separation
- Not required if vSAN Traffic is tagged on the Management VMkernel Interface
Also, the vSAN Witness Appliance needs to run on a *supported vSphere 5.5 or higher environment.
*vSphere 5.5 End of General Support is September 19, 2018.
What are the requirements for multiple Witness Appliances?
When deploying multiple vSAN Witness Appliances, what are the requirements? This is common question that comes up when sizing for multiple 2 Node vSAN clusters.
Recent testing has been done specifically on the vSAN Witness Appliance when used in conjunction with 2 Node vSAN Clusters, and we have some updated underlying hardware guidance.
Supported Consolidation Ratios when using the vSAN Witness Appliance & 2 Node vSAN
Officially supported consolidation ratios for the vSAN Witness Appliance when used with 2 Node vSAN Clusters is as follows:
- CPU Consolidation – A vCPU to pCPU consolidation of 8:1 is supported
- Memory Consolidation/Overcommitment – A vRAM to pRAM consolidation of 8:1 is supported
- Capacity Utilization – Capacity utilization remains to be 1:1, but additional considerations should to be taken into account
- vSAN Witness Components are 16MB each
- A fully utilized “Tiny” vSAN Witness Appliance’s capacity disk can contain a maximum of 750 components
- Capacity utilization is going to be entirely dependent on the number of vSAN Witness Components.
The following illustration shows a single vSphere Host with multiple workloads (DNS, File Services, vCenter, etc) as well as 8 vSAN Witness Appliances.
While the host has 16GB of physical RAM, the 12 VMs/Appliances have been allocated 82GB of RAM. Also consider that vSphere itself requires some of the physical RAM.
How many resources does a vSAN Witness Appliance use?
Consider that the vSAN Witness Appliance is essentially an ESXi installation inside a VM. ESXi itself with vSAN does require a minimal amount of RAM.
Using the vSAN Memory Sizing KB (https://kb.vmware.com/kb/2113954), it can be seen that a “Tiny” vSAN Witness Appliance requires just over 6.2GB of RAM:
5426 MB + (1 * (636 MB + (8MB * 10))) + (1 * 70 MB) = 6212MB
The “Tiny” vSAN Witness Appliance is deployed with 8GB, which is sufficient for normal operation for vSAN 2 Node Clusters with up to 750 components.
The vSAN Witness provides quorum for vSAN 2 Node Clusters but has a small amount of utilization that is not directly associated with data reads or writes to vSAN objects.
Careful inspection of a vSAN Witness Appliance reveals that only a fraction of the provisioned resources are used during normal operations. Because the vSAN Witness only deals with vSAN object metadata, much of it’s resources remain largely idle. It isn’t until a failover occurs that the vSAN Witness requires additional resources.
Real world example of sizing a dedicated cluster for vSAN Witness Appliances
Pre-Consolidation Guidance
Using a 1:1 ratio for CPU, RAM, & Capacity for “Tiny” vSAN Witness Appliances, an example customer with 200 locations would require:
- 400 vCPUs (200 x 2 vCPU)
- 1.6TB of RAM (200 x 8GB RAM)
- 9TB Capacity (200 x 45GB)* if fully utilized
*45GB = 12GB (OS), 10GB (cache), 15GB (capacity), & 8GB (vSwap) - ~24TB RAW storage
That could equate to a cluster with:
- 10 Hosts
- Dual 20 core CPUs
- 192GB RAM
(Standard host configuration supporting 160GB for Appliances + ESXi/vSAN Memory requirements) - 2.4TB RAW Capacity
- Totals
- 400 cores
- 1.92TB RAM
- 24TB RAW Capacity (Mirroring + Slack)
As can be seen, these hosts are heavy on compute/memory and light on capacity. Remember that there is no requirement for the vSAN Witness Appliance to be backed by flash devices. With the cost of NL-SAS devices being relatively cheap, it is much more cost effective to have fewer hosts with more capacity.
Using the Supported Consolidation Guidance
Using the newly supported guidance of 8:1 for compute/memory, a smaller configuration would be appropriate.
- 400 vCPUs/8 = 50 vCPUs
- 1.6TB RAM/8 = 200GB RAM
- 2.4TB Capacity (Mirroring+Slack)
- 24TB RAW storage
This equates to a much smaller cluster:
- 3 Hosts
- Dual 10 core CPUs
- 96GB of RAM
(Standard host configuration supporting 67GB for Appliances + ESXi/vSAN Memory requirements) - 8TB RAW Capacity
- Totals
- 60 cores
- 288GB RAM
- 24TB RAW Capacity (Mirroring + Slack)
The above example, while utilizing the same amount of capacity, results in 1/3 fewer hosts and roughly 1/6th the amount of RAM when using the updated vSAN Witness Appliance consolidation guidance when used with 2 Node vSAN.
This directly translates to less infrastructure, at a lower overall cost, required to run multiple vSAN Witness Appliances.
Summary
The updated vSAN Witness Appliance support consolidation ratio of 8:1 for CPU and Memory provides better guidance for customers using 2 Node vSAN Clusters. This testing and validation was performed only for 2 Node vSAN Clusters using the “Tiny” profile. The 8:1 consolidation ratio is has not been tested or recommended for “Normal” or “Large” vSAN Witness Appliances when used with 2 Node or Stretched vSAN Clusters.
Look for more in-depth content around vSAN Witness Sizing with 2 Node vSAN in the coming weeks on StorageHub.
