There are numerous network discussions out there on large MTU or jumbo frames. Countless articles, videos, blogs, etc. have been written on jumbo frames. So why write yet another? It seems despite the copious amounts of collateral on jumbo frames, there is still continuous misuse, especially in the virtual environment. VMware Global Support Services continues to get calls on jumbo frame issues. I resolved an issue the other day where there were three different MTU (Maximum Transmission Unit) configured in the same environment! This article is not going to be a deep technical discussion but rather a conceptual overview to help understand why there are so many issues with jumbo frames.
Many documents recommend using jumbo frames to enhance network throughput and reduce CPU utilization. But I have yet to see a caveat, disclaimer or cautionary statement on implementation. What is that caveat; Only use jumbo frames in your virtual infrastructure if it is already setup and in use in your environment. If jumbo frames are not currently used in your network, the added value is not worth the additional complexity added to your infrastructure. If you intend to use jumbo frames or it becomes a requirement, make sure you AND your network administrator plan the implementation!
Why the caveat? Why is it such an ordeal to setup jumbo frames? It’s not really if you understand the concept of how jumbo frames work throughout in your environment. I recently did a webinar with Cody Hosterman and J Metz for SNIA.org on Virtualization and Storage Networking Best Practices and the subject of jumbo frame, unsurprisingly, came up. J had a great analogy using a basketball and hoops. Think of the basketball as the frame or packet size (MTU). Let’s say the ball has a diameter of 9-inches, and the first and last hoops are also 9-inches, no problem, correct? Incorrect, you see there are several hoops in the path, same goes for your network. It’s not just your host or even vmkernel and the target the frames or packets pass. What most often happens is the virtual admin configures the host or maybe a vmkernel with jumbo frames 9k MTU, but the rest of the network is unchanged and usually using the default 1500 MTU. Consequently, the large frames cannot pass through the smaller frame ports and you end up with dropped frames or packets. On the other hand, a small ball measuring 1.5-inches can easily fit through a 9-inch hole. I will delve into this later. Making sure all devices in the path support the intended frame size is the key to a reliable network and infrastructure. The illustration below represents the concept of large versus small frames passing through an environment.
End of the blog for those who do not have jumbo frames already configured in your environment.
For those who have jumbo frames configure in their environment, and completely understand it, let’s discuss how you may utilize them in your virtual environment. Jumbo frames let ESXi hosts send larger frames out onto the physical network. The network must support jumbo frames end-to-end that includes physical network adapters, physical switches, and storage devices. Your virtual network must also be configured to support jumbo frames, this includes virtual switches. The key to using jumbo frames is to make sure all the “hoops” can pass the largest ball (frame) you intend to use. You now have options of where you may use jumbo frames. You do not have to use jumbo frames everywhere and, in many cases, they are not needed. With the granularity VMware vSphere® networking offers, it is possible to have different MTU setting in your environment for different targets. For example, you may want to configure your storage connectivity to use jumbo frames while your virtual machines use the standard 1500 MTU. Remember, each vmkernel has its own configurable MTU. This allows for different services, vMotion, vSAN, vVols, iSCSI, NFS, etc. to have independent MTU. Make sure your vmkernels and storage targets are both using the same MTU. The illustration below shows a mixed MTU environment with vSAN traffic using 9000 and the witness traffic using the standard 1500. This is just one simple example of how mixed frame size or MTU may be utilized.
Click on the image to read more about this configuration.
The flexibility in configuration of virtual networking is usually where people get into trouble. Someone decides to “try” changing the default MTU but only changes one point or they do not change all related vmkernels or targets. This can cause issues such as vMotion failing, vSAN hosts become isolated, or external storage loses connectivity. Mismatched MTU settings cause dropped packets isolating or degrading host’s or target’s connectivity. Next thing you know, you are calling GSS to figure out why.
You need a thorough understanding of your network, frame sizes, and benefits before you implement. In docs.vmware.com there are over 600 articles about jumbo frames. Don’t assume you know all the details, do your research! Make sure your network supports the largest MTU you plan to use in your virtual environment. Don’t chase performance increases of a few percentages at the risk of reliability.
Remember, your number one priority is reliability!
Resources:
- Jumbo Frames in vSphere 6.7
- KB Articles on Jumbo Frames
- Connect on Twitter: @jbmassae
