I’ve received a number of questions from customers about using SAS expanders for external, or internal storage. I would like to clear up any confusion and discuss what is supported (and more importantly why!)
SAS expanders are supported only on VSAN ready nodes. Virtual SAN Ready Node is a validated server configuration in a tested, certified hardware form factor for Virtual SAN deployment. Ready node configurations can be found here. In this example we can see that 24 drives are supported on this Dell R730XD configuration.
SAS expander support is limited due to the significant amount of extended validation that is required to make sure they do not inhibit performance or avalability. A typical SAS physical connection has 4 SAS channels. Most internal HBA and controllers only have 2 SAS physical connections and in a directly connected configuration only support 8 drives. SAS expanders switch the connection, allowing up to 254 per connection. The SAS expander must work tightly with the raid controller (both are often made by the same manufacturer) and firmware and driver versions for both must be kept in “sync” to prevent issues. SAS expanders also support SATA tunneling protocol that allows a SATA drive to emulate a SCSI device.
The VSAN configuration guide (VCG) team works to identify and validate configurations. They work with the vendors to establish supported stable driver/firmware combinations. Flash devices, and mixed SATA/SAS environments put a lot of stress on the expanders and controllers and extensive testing is done to identify stable performant configurations.
So what if you want to “build your own” VSAN server today, and want to use more than 8 devices? The simple and easy way is to purchase a SAS pass-through HBA for each drive group. This has a number of advantages over SAS expanders.
- Dedicating a SAS HBA to each disk group breaks up internal fault domains, and reduces the impact of a failed controller.
- Dedicating a SAS HBA to each disk group increases the available queue depth and throughput for performance and faster rebuilds. We previously covered on the blog that this is a great way to linearly “scale up” performance inside each host. For all flash configurations this is especially important.
- Dedicating a SAS HBA to each disk group reduces the amount of firmware/drivers that must be tracked and accounted for in updates. The Virtual SAN health service can help with this task.
Another question that comes up is using external storage expansion. This is supported in limited cases of specific ready nodes. This is primarily for blade or modular systems that lack a number of internal drive bays. As with the SAS expanders, pay attention to the number of drives supported.
Examples of this include:
- HY-6: HP Bl460c Gen9 – A blade using a storage blade to add extra drive bays.
- AF-8: DELL FX2 FC630 – A modular server/blade hybrid solution with incredible density can leverage FD332 systems for expansion.
One thing to note in the case of the Dell FD332 is that it can be partitioned with half of the drives going to one compute module, and the other half going to another. This does require the Dual ROC (Raid on Chip) option be purchased.
VSAN offers extensive flexibility and performance across a huge number of server and storage device vendors. SAS expanders, and pass through HBA’s allow for flexibility in capacity, and performance when designing. The VSAN ready nodes act as a great simple quick place to start when designing a VSAN solution.
