Every week my top vSAN questions tend to concentrate around networking design. A lot of questions center on if it is ok to share physical interfaces for multiple purposes.
Do I need dedicated interfaces for vSAN?
In talking to customers a little more than half choose to put other traffic classes (Virtual machine, Management, vMotion etc) on the same interfaces as the interfaces used for vSAN. This configuration is fully supported and will perform properly if sized and configured correctly.
Why you would use shared interfaces?
Shared interfaces allow you to use fewer switch ports, fewer NIC ports, and less cabling. Lowering complexity, power usage, and cost in an environment. Consolidating and converging storage and traditional networking allows lower operational overhead also for monitoring, patching.
How do I prevent vMotion or other traffic from impacting vSAN traffic when sharing interfaces?
The majority of concerns from noisy neighbors can be controlled by using Network IO Control (NIOC). NIOC is included with the distributed switch license included with vSAN. For specific guidance see the vSAN Network Design Guide.
Are my switches “fast” enough to handle storage and regular traffic together?
Many modern 10Gbps networking has incredible low port to port latencies with deep packet buffers, and significantly higher packets per second rates. Top of Rack switching has come a long way over the years. GigE switches often had relatively low packets per second rating and small shared buffers. A older top of rack switch might only support 6 million packets per second, and transmit and receive buffers were both small and shared, and have port to port latencies as high as 50 microseconds. Modern 10Gbps switches have over 100x higher packets per second rating, dedicated port buffers, and port to port latencies up to 100x lower. VMware does recommend that real switches be used, over devices that lack port to port traffic capabilities. For more information on oversubscription considerations follow this link.
When should I use dedicated interfaces?
While the most extreme performance environments may still benefit from dedicated interfaces and switches the decision to converge, the choice is often done for operational reasons. If the virtual machine networking can not maintain high uptime (for operational, patching, or change control reasons) dedicating interfaces and switches to storage may still happen. The general trend is towards conference as networking teams become used to providing always on switching environments and the speed of ports advances. Also if your existing virtual machine networking is on legacy heavily oversubscribed devices designed for low throughput and largely idle enviroments (Such as Cisco FEX) then it may be suitable to consider separate switching for VMware vSAN traffic.
