The first NVMe drive is on the VSAN HCL and customers are already adopting it! Non-Volatile Memory Express (NVMe) is a specification for attaching and accessing flash storage devices to the PCI Express (PCIE). Intel has demonstrated that with NVMe VSAN clusters can easily scale into the millions of IOPS. Another Intel blog goes through model part #, capacity, form factor and tier information of the NVMe SSDs.
Traditional flash devices using AHCI or SCSI command set have only a single command queue that operates at a depth of up to 32 or 254 commands respectively. While great performance can be found operating at the deeper command depth, this does start to bring latency up. NVMe unlocks up to 65536 queues, and up to 65536 commands per queue allowing hugely parallel access to flash and memory storage devices. NVMe also promises significant CPU overhead reduction compared to traditional flash devices by utilizing a streamlined lightweight protocol. While AHCI can require as many as 9 register read/writes NVMe can process a command completion with only two.
NVMe drives are extremely fast. If you are initiating lots of cloning operations on a large cluster saturating 10Gbps links may be possible. When using NVMe you may want to consider 25/40Gbps connections and make sure you have adequate bandwidth between switches in the same cluster. The NVMe standard does not define hot plug, and relies on PCI-Express for hot plug support. Be sure to check with your server vendor if you will need this functionality.
VSAN is a software defined product. traditionally when new storage or drive interfaces come out you had to purchase an entirely new disk array to take advantage of it. Today if you already have servers that support NVMe over PCI-E you can simply install these cards and lower latencies and increase IOPS on your VSAN cluster. VSAN also provides a critical distributed mirroring functionality for NVMe devices, as currently RAID controllers do not exist for NVMe devices.
StorageReview.com released VMware Announces NVMe Support for VSAN article which details their recent review (benchmark and configuration testing) of Virtual SAN and, in effect, found it a tremendous performer.