Home > Blogs > Support Insider > Tag Archives: network

Tag Archives: network

New NSX KBs you need to know

Here is a list of currently trending KBs for NSX for vSphere

Netcpa issues in VMware NSX for vSphere 6.x (2137005)
Symptoms:

  • Virtual machines running on the same ESXi host fails to communicate with each other
  • Virtual machine fails to communicate with the NSX Edge Gateway (ESG)
  • Routing does not appear to be functioning despite having a defined route for the NSX Edge Gateway
  • Rebooting the NSX Edge does not resolve the issue
  • Running the esxcli network vswitch dvs vmware vxlan network list –vds-name=Name_VDS command on the ESXi host displays the VNIs as down

Network connectivity issues after upgrading or repreparing ESXi hosts in a VMware NSX/VCNS environment (2107951)
Symptoms:

  • Connecting virtual machines to a logical switch fails
  • Re-connecting the virtual machine to a logical switch fails

Oracle connections times out when forwarded through the VMware NSX for vSphere 6.1.x Edge (2126674)
Symptoms:

  • Oracle connections times out when forwarded through the VMware NSX for vSphere 6.1.x Edge
  • The NSX Edge is dropping packets and client/server keeps re-transmitting the same particular packets

MSRPC connections time out when forwarded through the VMware NSX for vSphere 6.1.x Edge (2137751)
Symptoms:

  • MSRPC connections time out when forwarded through the VMware NSX for vSphere 6.1.x Edge

Linux virtual machines with NFSv3 mounts experience an operating system hang after more than 15 minutes outage on the upstream datapath (2133815)
Symptoms:

  • Applications interacting with an NFSv3 mount experience a hang on hard mounted NFSv3 mounts or an error on soft mounted NFSv3 mounts after more than 15 minute upstream data path failure
  • The virtual machine performance increasingly degrades with the time it resides in an NFS hung state

VMware ESXi 6.0 Update 1 host fails with a purple diagnostic screen and reports the error: PANIC bora/vmkernel/main/dlmalloc.c:4923 – Usage error in dlmalloc (2135956)
Symptoms:

  • VMware ESXi 6.x host fails with a purple diagnostic screen
  • Netflow function is enabled on the ESXi host

Recommended minimum VMware Tools versions for Guest Instrospection in VMware NSX for vSphere 6.x (2139740)
Known issues:

  • Windows virtual machines using the vShield Endpoint TDI Manager or NSX Network Introspection Driver (vnetflt.sys) driver fails with a blue diagnostic screen

Understanding and troubleshooting Message Bus in VMware NSX for vSphere 6.x (2133897)
Symptoms:

  • This article can be referred to when it is suspected that a communication issue between the NSX Manager and the ESXi hosts is responsible for symptoms observed such as failure to publish configuration. Examples of such symptoms are:
    • Publishing firewall rules fails
    • Some of the ESXi hosts does not have the VDR/LIF information configured through the NSX Manager

Choosing a network adapter for your virtual machine

Network adapter choices depend on the version number and the guest operating system running on the virtual machine. This video discusses the different network adapter options available for virtual machines.

For additional information related to network adapters for your virtual machine, check out VMware Knowledge Base article: Choosing a network adapter for your virtual machine (1001805).

For more information on network types, see: Understanding networking types in hosted products (1006480).

New Network port diagram for vSphere 5.x

Over the past few weeks we have been working on constructing a brand new network diagram, depicting ports in use for vSphere 5.x

These diagrams have been very popular in the past and we hope you like this one too! We created Knowledgebase article: Network port diagram for vSphere 5.x (2054806) as a container for the pdf diagram. The pdf also lists all of the ports used in tabular format.

If you’d like to see more of these, tell us in the comments section below!

Network port diagram for vSphere 5

Alternate download location.

Note: This information provided is on a best effort basis. VMware will endeavor to update the diagram as new releases come out.

Networking options in VMware Workstation and Fusion

Networking in VMware Workstation and FusionWhether you just bought yourself a copy of VMware Fusion (for your Mac), Workstation (for Windows or linux) or are using Player, Ace, or even the old VMware Server product, you’ll be soon setting up your first virtual machine. Your new vm is going to want to talk to the outside world just like your physical machine, so it’s a good idea to understand some basic options available to you to ensure your new vm works right out of the box. There are three types of networking available to virtual machines. Each type has its own uses, behaviors and features. They are as follows:

  • Bridged networking
  • Host-only networking
  • Network Address Translation (NAT) networking

Using the wrong networking type or configuration settings may result in undesirable behavior, and frustration on your part, so lets understand these three variations and what they mean to you.

Bridged networking connects a virtual machine to a network using the host computer’s Ethernet adapter. If your host computer is on an Ethernet network ,this is often the easiest way to give your virtual machine access to that network. If you use bridged networking, your vm is a full participant in the network. It has access to other machines on the network and can be contacted by other machines on the network as if it were a physical computer on the network. This is a frequently used option.

Host-only networking creates a network that is completely contained within the host computer. Host-only networking provides a network connection between the virtual machine and the host computer. In this setup, your vm will not have access to the outside world, only the physical machine your are running it on. This approach can be useful if you need to set up an isolated virtual network.

Network Address Translation, or NAT for short, gives your virtual machine access to network resources using the host computer’s IP address. How is this different than bridged you ask? If you use NAT, your virtual machine does not have its own IP address on the external network. Instead, a separate private network is set up on the host computer. This method might be best if your virtual machines do not provide services but still need to access a network.

So, your not running a web or print server, or doing file sharing, and so forth. For further details on the differences and which is right for you, refer to KB article: Understanding networking types in hosted products (1006480) Once you have an idea of which method you need for your virtual machine, we have a video which shows you how to use the Virtual Network Editor.

Troubleshooting Network Teaming Problems with IP Hash

Last week, I shed some light on the challenges having multiple Vmkernel ports in the same subnet can cause. Today, I’ll be shifting gears back to troubleshooting common network teaming problems – more specifically with IP hash.

IP Hash load balancing is really nothing new, but unfortunately, it is very often misunderstood. To be able to effectively troubleshoot IP hash and etherchannel problems, it’s important to first understand how IP hash works and its inherent limitations.

From the vSphere 5.1 Networking Guide definition of IP Hash load balancing:

“Choose an uplink based on a hash of the source and destination IP addresses of each packet. For non-IP packets, whatever is at those offsets is used to compute the hash”

This may seem like an overly concise definition, but it does indeed summarize what the host is doing in a network team utilizing IP hash load balancing. To expand on this a bit – the host will take the source and destination IP address in a HEX format, perform an Xor operation on them, and then run the result through another calculation based on the number of uplinks in the team. The final result of the calculation is a number between 0 and the number of uplinks, minus one. So if you had four uplinks in an IP hash team, the result of each calculation would be a number between 0 and 3 – four possible outcomes that are each associated with one NIC in the team. This end result of the calculation is commonly referred to as the interface index number.

You may also be wondering how non-IP traffic would be put through this calculation since there wouldn’t be an IP header and thus it wouldn’t have a 32-bit source and destination IP address for this calculation. As per the definition from the networking guide: “For non-IP packets, whatever is at those offsets is used to compute the hash”. This essentially means that two 32 bit binary values will be taken from the frame/packet from where the IP addresses would usually be located if it was an IP packet. It really doesn’t matter if these two 32-bit values are not actually IP addresses. So long as there is binary data in these locations, the calculation can still be applied to balance this traffic out across the adapters in the team.

This behavior is very different from the default ‘Route based on the originating virtual port ID’ load balancing method. Instead of having a one-to-one mapping of virtual NICs to physical adapters in the host, any single virtual machine could potentially use any of the NICs in the team depending on the source and destination IP address.

It is this fact that makes IP Hash so desirable from a performance standpoint. It is the only load balancing type that could (in theory) allow any single virtual machine to utilize the bandwidth available on all network adapters in the team. If a virtual machine operates in a client/server type of role with a large number of independent machines accessing it (think hundreds of Outlook clients accessing an Exchange VM for example) the IP hash algorithm would provide a very good spread across all NICs in the team.

On the flipside, however, if you have a small number of IP addresses (or only one) doing most of the talking, you’ll find that the majority of the traffic may be favoring only one network adapter in the team. An example of this would be a database server that is accessed by only one application server. Since there is only one source and destination address pair, the hash calculation will always work out to the same index value and the host will only select one network adapter in the team to use.

That’s how it works from the host’s perspective, but how does the return traffic know which adapter to take? As you probably expected – it’s not just the host that has some math to do, the physical switch also needs to run packets through the same algorithm to select the correct physical switch port.

For this load balancing type to work correctly, the physical switch ports must be configured as an etherchannel bond (also called a port channel, or a trunk in the HP world). This essentially allows the bonding of multiple network adapters and causes the switch to perform the same type of hash calculation to ensure that it’s behaving in the exact same way for all traffic heading in the other direction. If the physical switch is not configured for an etherchannel, it’ll start getting confused very quickly as it’ll see the same MAC address sending traffic out of multiple ports resulting in what’s referred to as ‘MAC flapping’. I won’t go into too much detail into this scenario, but needless to say, all sorts of undesirable things start to happen when this occurs.

IP Hash and Etherchannel Limitations

Now that we have covered how IP hash works, let’s cover off briefly on the limitations of this type of teaming setup. The potential performance benefits of IP hash may not always outweigh the flexibility that is lost, or the other limitations that will be imposed.

I’d definitely recommend reading KB article: ESX/ESXi host requirements for link aggregation (1001938), which discusses this in detail, but to summarize a few key limitations:

ESX/ESXi supports IP hash teaming on a single physical switch only: This one can be a real deal-breaker for some. Because etherchannel bonding is usually only supported on a single switch, it may not be possible to distribute the uplinks across multiple physical switches for redundancy. There are some exceptions to this rule as some ‘stacked’ switches, or modular switches with a common backplane support etherchannel across physical switches or modules. Cisco’s VPC (virtual port channel) technology can also address this on supported switches. Consult with your hardware vendor for your options.

ESX/ESXi supports only 802.3ad link aggregation in ‘Static’ mode: This is also referred to as ‘Mode On’ in the Cisco world. This means that LACP (link aggregation control protocol) cannot be used. The only exception is with a vNetwork Distributed Switch in vSphere 5.1, and with the Cisco Nexus 1000V. If you are using vNetwork Standard Switches, you must use a static etherchannel.

Beacon Probing is not supported with IP hash: Only link status can be used as a failure detection method with an IP hash team. This may not be desirable in some situations.

An Example

Let’s walk through another fictitious example of a network teaming problem. Below is the problem definition we’ll be working off of today:

“I’m seeing very odd behavior from one of my hosts! Each virtual machine can’t ping different sets of devices on the physical network. If I try to ping a device, it may work from some virtual machines, but not from others. Connectivity is all over the place. I can’t make heads or tails of this! Help!”

Well this certainly sounds very odd – why are some devices accessible while others are not? And to make matters even more confusing, things that don’t work from one VM work fine from another and vice versa.

Let’s have a quick look at the vSwitch configuration on the host having a problem:

We’ve got three physical adapters associated with a simple virtual machine port group. Let’s have a look at the teaming configuration to determine the load balancing type employed:

And sure enough, Route based on IP hash is being employed as the load balancing method. With this very limited amount of information we can determine quite a bit about how the environment should be configured. We know the following:

  1. All three physical switch ports associated with vmnic1, vmnic2 and vmnic3 should be configured identically and should be part of a static etherchannel bond. Since this is a vNetwork Standard Switch, LACP should not be used.
  2. No VLAN tag is defined for the port group, so this tells us that we’re not employing VST (virtual switch tagging) and that the switch should not be configured for 802.1q VLAN trunking on the etherchannel bond. It should be configured for ‘access’ mode.
  3. The five virtual machines in the vSwitch will be balancing their traffic across all three network adapters based on source and destination IP addresses. The host will perform a mathematical calculation to determine which adapter to use.

To start our troubleshooting, we’ll first try to determine what works and what doesn’t. We’ll use two of the virtual machines, ubuntu1 and ubuntu2 for our testing today. We obtained a list of IP addresses and their ping results from our fictitious customer for these two virtual machines. Below is the result:

For ubuntu1:

10.1.1.1 Physical Router, default gateway Success
10.1.1.22 Physical Server, SQL database Success
10.1.1.23 Physical Server, SQL database Success
10.1.1.24 File Server on another ESXi host Fail
10.1.3.27 Physical Server, Primary Domain Controller Fail
10.1.3.28 Secondary Domain Controller on other ESXi host Success
10.1.1.101-106 All other VMs in the same port group Success

 

For ubuntu2:

10.1.1.1 Physical Router, default gateway Success
10.1.1.22 Physical Server, SQL database Success
10.1.1.23 Physical Server, SQL database Fail
10.1.1.24 File Server on another ESXi host Success
10.1.3.27 Physical Server, Primary Domain Controller Success
10.1.3.28 Secondary Domain Controller on other ESXi host Success
10.1.1.101-106 All other VMs in the same port group Success

 

Well that certainly seems odd doesn’t it? It’s pretty hard to find a pattern here and things really do seem to be “all over the place” as our fictitious customer claims. Neither of the VMs is having difficulty getting to the default gateway, but ubuntu1 can’t hit one of the VMs on another host and can’t hit a physical domain controller. Ubuntu2 can’t hit one of the physical SQL servers, but didn’t have any trouble reaching the other five machines we tried.

We can pretty much rule out the virtual machines in this situation as they can communicate with each other within the vSwitch just fine, and can also communicate with most of the devices upstream on the physical network as well. At this point, you may be suspecting that there is some kind of a problem with the teaming configuration, a physical switch problem or possibly something with one of the physical NICs on the host. All of these would be good guesses and would be in the correct direction here.

As I discussed in my blog post relating to NIC team troubleshooting last week, the esxtop utility can be very useful. Let’s have a look at the network view to see if it can lend any clues:

If you recall from my previous post when using ‘Route based on originating virtual port ID’, you’ll have a one-to-one mapping of virtual machines to physical network adapters when each VM has only a single virtual NIC. Not so with IP Hash. Instead, each virtual machine lists ‘all’ and the number of physical adapters it could be utilizing based on its calculation of the source and destination IP hash.

Well, that stinks. But we’re not quite ready to crawl to our network administration team in shame asking them to “check the switch”. Let’s take this a step further, get a better understanding of the problem and perhaps even impress them a bit in the process!

We can quite easily conclude that what we are dealing with here is an IP hash or an etherchannel problem. How do we know this? Very classic symptoms. Some source/destination IP combinations work just fine, while others don’t. Forget VLANs, forget subnet masks, routing – none of this matters in this situation. The same situation could present itself if you were communicating with something in the same network, or something 15 hops away on the other side of the world. The pattern that we are looking for here is actually the complete LACK of a pattern across the VMs. What we know is that we’re experiencing consistent difficulty reaching certain IP destinations, and it’s different from each VM we try in the vSwitch.

Unraveling the Mystery of IP Hash

Okay, so now what? Let’s think back to how IP hash works – it’s just a mathematical calculation. If the host can figure out which uplink to use, why can’t we do the same? Unfortunately, esxtop can’t help us, but if we have the source and destination IP addresses as well as the number of physical uplink adapters that are link-up, we can determine that path it will be taking.

I don’t want to scare anyone away here, but this does require some math. I’ll be the first to admit that I’m not the greatest with math, but if I can do it with the help of a calculator, then so can you. VMware has a great KB on how to do this calculation – KB: Troubleshooting IP-Hash outbound NIC selection (1007371), and I’d recommend giving it a read as well. I’ll walk you through this process for one of the source/destination address pairs and then we can see if a pattern finally emerges.

Let’s start with ubuntu1 and its default gateway. We know communication across this pair of addresses is working fine:

Source IP address: 10.1.1.101

Destination IP address: 10.1.1.1

Number of physical uplinks: 3

Let’s get started. The first thing we’ll need to do is to convert the IP addresses into HEX values for the calculation. There are some converters available that can do this in one simple step, but I’ll take the slightly longer road for educational purposes. First, let’s convert each of these IP addresses into their binary equivalents. This will need to be done for each of the four octets of the IP address separately. If you aren’t sharp with IP subnetting, or don’t really know how to convert from decimal values to binary, not to worry. Simply use the Windows Calculator in Programmer Mode and it can do the conversion for you.

Our IP address 10.1.1.101 – broken into its four independent octets – converts to binary as follows:

10

1

1

101

00001010

00000001

00000001

01100101

 

Pro-Tip: Remember, each of the octets of an IP address is expressed as an 8 bit value. Even if a binary value is calculated as fewer than 8 bits, you’ll need to append the 0’s at the beginning to ensure it’s always 8 bits in length. So a binary value of 1 for example, should be expressed as 00000001 for the purposes of our calculations today. Only the first octet can be expressed with less than 8 bits for this purpose because any leading zeros won’t change the total value of the number.

Next, we need to combine all four binary octets into a single 32-bit binary number. To do this, we simply append all of these octets together into the following:

00001010000000010000000101100101

Or if we drop the leading zeros at the beginning of the 32-bit value, we have:

1010000000010000000101100101

This number equates to 167,838,053 when converted to decimal format, but we’ll need the HEX value. To convert this, you can simply enter the long 32-bit binary number into Calculator in ‘Programmer’ mode, and then click the ‘Hex’ radio button. If you did it correctly, you should get the value:

0xA010165

Great, now we’ll do the same thing for our destination IP address, which is 10.1.1.1. After converting this to binary and then to HEX, we get the following value:

0xA010101

Pro-Tip: From here on out, I’ll always express hexadecimal values with a 0x prefix. This is so that there is no confusion between regular decimal numbers and hexadecimal numbers. You can usually identify a HEX value if it has an A, B, C, D, E or an F in it, but it could just as easily be made up of only numbers between 0 and 9. Our hexadecimal value of A010165 could also be represented as 0xA010165 – simply ensure your calculator is in HEX mode and drop the 0x when doing the upcoming calculations.

Now we’ve got everything we need to run this through the IP Hash calculation:

Source HEX address: 0xA010165

Destination HEX address: 0xA010101

Number of physical uplinks: 3

From this information, we run it through the following formula outlined in KB article: 1007371

(SourceHEX Xor DestHEX) Mod UplinkCount = InterfaceIndex

Even if you have no idea what Xor or Mod is, don’t worry – any scientific or programmer calculator can make quick work of this. In the Windows Calculator, simply go into programmer mode and the Xor and Mod functions are available. Don’t forget to ensure you are in HEX mode before entering in the values. Plugging our values into this equation, we get the following:

(0xA010165 Xor 0xA010101) Mod 3 = InterfaceIndex

Thinking back to high school math class, let’s calculate what’s in the brackets first, and we are left with the following:

0x64 Mod 3 = InterfaceIndex

And then we simply use Mod 3 of that value to calculate the interface index. Remember, because we have only three uplink adapters, a valid answer will be anything between 0, 1 or 2.

InterfaceIndex = 1

And there you have it. That wasn’t so bad was it? From our calculation we were able to determine that communication between 10.1.1.101 and 10.1.1.1 will be using interface index 1, which is the second uplink in the team as this value starts at 0.

A New View of the Situation

I went ahead and calculated the remaining interface index values for the other source and destination IP address combinations for ubuntu1 and ubuntu2. Below is the result:

From ubuntu1:

Source IP Destination IP Ping Success? Interface Index
10.1.1.101 10.1.1.1 Success 1
10.1.1.101 10.1.1.22 Success 1
10.1.1.101 10.1.1.23 Success 0
10.1.1.101 10.1.1.24 Fail 2
10.1.1.101 10.1.3.27 Fail 2
10.1.1.101 10.1.3.28 Success 0

 

From ubuntu2:

Source IP Destination IP Ping Success? Interface Index
10.1.1.102 10.1.1.1 Success 1
10.1.1.102 10.1.1.22 Success 1
10.1.1.102 10.1.1.23 Fail 2
10.1.1.102 10.1.1.24 Success 0
10.1.1.102 10.1.3.27 Success 1
10.1.1.102 10.1.3.28 Success 1

 

Well now, I’d say we are now seeing a very distinct pattern here. The physical adapter or physical switch port associated with interface index 2 is no doubt having some kind of a problem here. At this point, we can now hold our heads up high, walk down the hall to our network administration team and let them know that we’ve clearly got an etherchannel problem here. They may even be impressed when you tell them that you’ve narrowed it down to one of the three NICs in the team based on your IP Hash calculations!

After a quick look at the physical switch configuration, the network team provides you with the configuration of the port-channel and the three interfaces in question:

interface Port-channel8
switchport
switchport access vlan 10
switchport mode access
!
interface GigabitEthernet1/11
switchport
switchport access vlan 10
switchport mode access
channel-group 8 mode on
!
interface GigabitEthernet1/12
switchport
switchport access vlan 10
switchport mode access
channel-group 8 mode on
!
Interface GigabitEthernet1/13
switchport
switchport trunk allowed vlan 99-110
switchport mode trunk
switchport trunk encapsulation dot1q
switchport trunk native vlan 99

 

And there you have it. On the physical switch, port-channel 8 is made up of only two interfaces – not three. The third NIC in the team is associated with port GigabitEthernet1/13, which is an independent VLAN trunk port. Not only is it not a part of the etherchannel, its VLAN configuration is way off. Remember – IP Hash requires a consistent configuration on the physical switch. Uplink adapters cannot be removed or added to the vSwitch without ensuring that the physical switch ports associated with them are added to the etherchannel bond.

I hope you enjoyed this write-up and found it educational. I’d recommend checking out the following KB articles for more information on some of the topics I’ve discussed today:

Challenges with Multiple VMkernel Ports in the Same Subnet

Last week I took a look at some common network teaming problems and how to apply logical troubleshooting methodology to zero in on the problem. I’m hoping to write some follow-ups to dig a bit more into other load balancing types, but for now, I’ll be shifting the focus to host routing.

When I talk about host routing, I’m referring to just that – the host. Virtual machines route based on their guest configuration and based on what they can access on the physical network by means of their vSwitch and the upstream network configuration. Running virtual machines is certainly the primary focus of the vSphere suite of products, but there are numerous services used by the host that will require network connectivity. These include but are not limited to:

  • Management Networking and vCenter connectivity
  • vMotion
  • Fault Tolerance
  • Host storage connectivity (iSCSI, NFS)

ESXi differs in many ways from ESX – namely in the absence of the bolt-on Service Console that was used for management purposes. I won’t be going to too much detail to describe the differences between ESX and ESXi, as this has been covered at great length in many blog posts and technical documents out there. Rather, I will focus on host routing, how it differs between ESX and ESXi and how to avoid common pitfalls.

I’m going to begin by walking through a problem I’ve seen on more than one occasion here in Tech Support at VMware Support. Here is a fictitious problem statement that we’ll be operating off of today:

I’ve been running ESX 4.1 without issue, and I just recently deployed a new ESXi 4.1 host. The vSwitch and physical network configuration is identical, but I can’t mount my NFS datastore on the new ESXi host. I get a “Permission Denied” failure. I’ve triple-checked my NFS filer permissions and everything is configured exactly as it should be. Help!

You may be wondering just what NFS permission problems have to do with host routing? More often than not, it really is just a simple permissions problem on the NFS filer. For the purposes of our example today though, we’ll be assuming that we’ve already verified that the permissions are indeed correct.

We’ll be working with two hosts – one ESX and one ESXi – that are both version 4.1 Update 1. Let’s start by taking a look at the ESX host that has been working just fine. First, we’ll check the NFS mount in question to determine how it’s been mounted and the relevant connection information. As you can see, this is a read-only NFS mount called ISOs. The NFS server’s IP address is 192.168.2.55. We’re able to browse the NFS datastore just fine, and if I remove the mount and mount it again, it connects back without issue on this ESX host.

Now let’s take a quick look at the vSwitch configuration on this same host:

As you can see above, the setup is very simple – vSwitch1 is being used for NFS purposes. We have a single VMkernel port for NFS with an IP address of 192.168.2.121. The Service Console port is utilizing vSwitch0, with a separate physical uplink adapter.

Everything may appear in order here, but the most important point we’ll want to make note of is that the Service Console port appears to be in the same IP subnet as the NFS VMkernel port. This is not according to best practices. Storage networking should always be in a dedicated subnet associated with a non-routable VLAN or a dedicated physical switch. Best practices aside, there should be no reason why this NFS share can’t be mounted so long as the host networking and upstream physical network are configured correctly.

Now let’s take a look at the ESXi host that is having difficulty. The vSwitch is configured as follows:

At first glance, the configuration certainly does seem consistent. The same physical uplink adapter configuration was maintained and the same IP subnet is being used for management and NFS connectivity. To begin our troubleshooting, let’s try to mount the NFS share and check the VMkernel logging see the exact error:

Pro-Tip: In ESXi 4.x, the VMkernel logging is written to the /var/log/messages file, along with logging from a slew of other chatty services, including hostd and vpxa. To filter only VMkernel messages in ESXi 4.x into the ‘less’ viewer, you can use the following command:

# grep vmkernel /var/log/messages | less

As you can see above, this is indeed specifically being reported as a ‘permission denied’ error. The ESXi host is receiving an error 13 back from the NFS filer. There really doesn’t appear to be an obvious networking problem as we receive back the error 13 after our mount attempt and can ping 192.168.2.55 all day long without a single dropped packet. We can also ping the host without issue from the NFS filer as well.

So what exactly is going on here?

Looking back to our ESX host, we can recall that the Service Console and NFS VMkernel port were both in the 192.168.2.0/24 network. Our ESXi host doesn’t have a Service Console port. Instead, it uses a VMkernel port for management networking. Again, these two VMkernel ports are in the same 192.168.2.0/24 network. But how does the ESXi host know which VMkernel port to use for NFS client purposes? Short answer – it doesn’t.

The host selects only one VMkernel port to use when there are more than one in any directly connected network. Generally speaking, this will be the lowest numbered or first created VMkernel port. So if we have vmk0 and vmk1 in the same subnet, the host will usually elect vmk0 for all communications within that network.

Let’s test our theory by checking the VMkernel routing table. We do this by running the following two commands:

# esxcfg-route –l
# esxcfg-route -n

The -l option provides a simple routing table, and the -n (for neighbor list) gives you what is essentially an ARP cache for various MAC addresses, along with the VMkernel interface currently being used to reach it. As you can see above, the host’s routing table doesn’t even include the NFS VMkernel port, vmk1. Only the management kernel port is listed for the directly connected 192.168.2.0/24 network. So to put it simply, the host is passing all traffic to 192.168.2.55 – the NFS server – through vmk0. We’re getting an access denied message because the NFS server is configured to allow vmk1’s IP address – not vmk0’s management IP address.

We can even confirm this by doing a packet capture on vmk1 using the tcpdump-uw utility in ESXi. We used the following command to check for traffic destined to 192.168.2.55 (the NFS server) on vmk1:

# tcpdump-uw -i vmk1 dst 192.168.2.55

Based on our testing, we received no tcpdump-uw output at all for vmk1 when trying to mount the NFS share. If we repeat this with vmk0, we get a flurry of outgoing traffic destined to 192.168.2.55.

The next logical question you may ask is why did this occur in ESXi but not in ESX? The answer is quite simple – The host’s NFS client must use a VMkernel port for communication and on the ESX host, we have only one VMkernel port in the 192.168.2.0 network. In short, it has no other choice. The Service Console has an independent routing table and even if its IP address happens to be in the same IP subnet, the Service Console and VMkernel will make independent forwarding decisions based on their own tables.

As you can see below, we can view the independent VMkernel and Service Console routing tables on the ESX host using the esxcfg-route -l and route -n commands respectively.

You may have also noticed that there are two ping commands in ESX – ping and vmkping. As you have probably guessed, ping is used when you want to send ICMP echo requests out of a Service Console interface, and vmkping for a VMkernel interface. In ESXi, both commands still exist, but they both are used for VMkernel interfaces only and can be used interchangeably.

What about vMotion, iSCSI?

You may be wondering about vMotion, Fault Tolerance and iSCSI traffic via VMkernel ports. Can’t you just check off the appropriate checkbox in the VMkernel port properties to ensure the correct one is used for each service?

And you would be correct in your assertion – you can indeed select which VMkernel port to use for these services. Even if there are other VMkernel ports in the same subnet, the host will respect the choices you make via these checkboxes:

Although this certainly helps, having multiple VMkernel ports in the same subnet can still cause confusion – even with these options checked off.

Take for example vMotion. If a host has a VMkernel port for vMotion and another for Management in the same subnet, the host will forward vMotion traffic based on the VMkernel port checked off for vMotion. That’s great and all, but now let’s suppose that a network administrator accidentally reconfigured the wrong physical port on the switch during some maintenance and the vmnic used for vMotion in now communicating in the wrong upstream VLAN. Suddenly, live migrations are failing to and from this host.

The first logical troubleshooting step you may wish to do is to login to this host via SSH, and to try to ping the vMotion interfaces on other hosts. Surprisingly, you may find that you are able to ping all of the other vMotion interfaces without issue. This is because the ‘ping’ utility is not respecting your choice of vMotion interfaces – it is simply forwarding traffic based on the host’s routing table.

There are similar challenges with iSCSI. You can bind VMkernel interfaces for use with the software iSCSI initiator, and the host will indeed respect this binding. Again, ICMP does not. Some iSCSI storage arrays use ICMP ping echo requests/replies as heartbeats to ensure the host interfaces are up. If these echo replies are not being received from the expected interfaces it can lead to all sorts of problems. I won’t get into any more detail on this situation, as this alone could be a full-length blog post. In a nutshell, it can cause unexpected behavior should an interface associated with active iSCSI paths go down on the host.

What’s New in ESXi 5.1?

Most of what we looked at was relating to ESX/ESXi 4.1, but it applies as well to ESXi 5.0. Things do change slightly in ESXi 5.1 as some changes were introduced to both the behavior of ICMP as well as ping or vmkping utility available in the busybox shell.

First, I’d recommend taking a look at KB 2042189 that discusses the ICMP ping response behavior change in ESXi 5.1. In summary, ICMP echo replies are now only sent back via the interface they were received on. The host will no longer refer to its routing table to make these forwarding decisions for ICMP traffic. If you think back to our iSCSI example earlier, you can see how this would be ideal to address the heartbeat challenges relating to ICMP echo requests being sent from the SAN.

Second, we can now select which VMkernel interface to ping out of using the ‘ping’ or ‘vmkping’ command in ESXi 5.1 using the -I option. Previously we could do this only for IPv6.  This can help greatly if you find yourself in a situation with multiple VMkernel ports per subnet – especially when dealing with iSCSI where you want to test connectivity across multiple independent paths via two or more VMkernel ports.

What should I do?

How can you ensure that your traffic is utilizing the intended VMkernel interface? Unfortunately, there isn’t a special command or advanced setting that can be used to intervene in the host’s forwarding decisions in all scenarios. The best advice that can be given is to simply adhere to best practices. The relevant best practices are as follows:

  1. Have only one VMkernel port per IP subnet (the only exception here is for iSCSI multi-pathing, or multi-NIC vMotion in vSphere 5.x).
  2. A dedicated non-routable VLAN or dedicated physical switch for vMotion purposes.
  3. A dedicated non-routable VLAN or dedicated physical switch for IP Storage purposes.
  4. A dedicated non-routable VLAN or dedicated physical switch for Fault Tolerance purposes.

Not only will this ensure better segmentation of the various traffic types, it will also prevent the oddities I described earlier. Unfortunately, implementing these best practices in an existing environment may be much easier said than done. Based on our original NFS example, the configuration changes on the host would actually be quite simple – just a few IP address changes, but the physical infrastructure may be a different story altogether. You’d need to ensure that a VLAN was created, physical switches configured, along with readdressing of the NFS filer itself. Your network and storage administrators may not be too happy with you after you make a request like this, which is why it’s so important to configure your environment correctly from the beginning. Remember – VLANs are your friend.

Now let’s have a look at a different ESXi host – this one is configured to have services spread across various subnets/VLANs:

  • Management: vmk0 in the 192.168.2.0/24 subnet.
  • NFS: vmk1 in the 192.168.3.0/24 subnet.
  • vMotion: vmk2 in the 192.168.4.0/24 subnet.

On the physical switch, NFS and vMotion are configured for dedicated non-routable VLANs. On this host, the vSwitch configuration looks as follows:

As you can see above, we now have one VMkernel port per IP subnet and a neat and tidy host routing table:

Now each kernel port is associated with a separate ‘Local Subnet’. Everything else routes via the default gateway associated with the management VMkernel port as it should.

And there you have it – if you don’t give the kernel any other choice – only one choice – it’ll always do what is expected.

Some topical reading for you from the VMware Knowledge Base:

Working with packet filters in vCenter Heartbeat

vCenter Heartbeat provides rapid failover and failback on both physical and virtual platforms. It does this by monitoring VMware vCenter Server using the concept of active and passive server with same IP (Public IP) on both the servers.

Packet filter drivers are installed on a public network adapter during the installation of Heartbeat. Packet filter drivers make Public IP accessible from the active server by making it passthru and block it on the passive server. When switch-over or failover takes place it reverses the state of the packet filter.

Here is the packet filter enabled
You can check the state of your packet filter drivers using the command line. Navigate to the installation directory of vCenter Heartbeat and run nfpktfltr.exe gestate to see the status of drivers as shown:
Here it results passthru and this state of packets is on active server and public IP is accessible in the network. For passive servers the state of packet filter is filter and this means the Public IP is filtered for outside Network.
We can also install and uninstall packet filter drivers using the command line. Go to the install directory of vCenter Heartbeat and run nfpktfltr.exe uninstall
“<installdir>\VMware\VMware vCenter Server Heartbeat\r2\drivers\nfpktfltr\<HB version>”
For more information on installing and uninstalling the VMware vCenter Server Packet Filter Driver refer to KB article:
Installing and uninstalling the VMware vCenter Server Heartbeat packet filter driver (1009567).

You may also be interested in these related articles:

Top 10 KB Articles for Infrastructure

The following three lists of Knowledgebase articles comprise our top 10 articles that our Infrastructure Support Engineers are using to solve customer Support Requests during the month of January. When we say Infrastructure, it’s more of an internal term we use to group support teams. The Infrastructure guys cover issues including storage, networking, and system faults (crashes).

The first list is the overall top ten. The second and third lists are known infrastructure issues and how-to / informational KB articles that apply to ESXi 5.1 and vCenter Server.

Top 10 Infrastructure Articles (ESX/i 3.x/4.x/5.x, vCenter Server)

  1. Broadcom 5719/5720 NICs using tg3 driver become unresponsive and stop traffic in vSphere (2035701)
  2. Unmounting a LUN or Detaching a Datastore/Storage Device from multiple ESXi 5.x hosts (2004605)
  3. Installing async drivers on ESXi 5.x (2005205)
  4. Investigating virtual machine file locks on ESXi/ESX (10051)
  5. Using esxtop to identify storage performance issues (1008205)
  6. Recreating a missing virtual machine disk (VMDK) descriptor file (1002511)
  7. Commands to monitor snapshot deletion in ESX 2.5/3.x/4.x and ESXi 3.x.x/4.x.x/5.x (1007566)
  8. Committing snapshots when there are no snapshot entries in the Snapshot Manager (1002310)
  9. Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.0 (2004684)
  10. Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported (1012384)

Top 10 vSphere 5.1 Infrastructure known issues

  1. Broadcom 5719/5720 NICs using tg3 driver become unresponsive and stop traffic in vSphere (2035701)
  2. Unmounting a LUN or Detaching a Datastore/Storage Device from multiple ESXi 5.x hosts (2004605)
  3. Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported (1012384)
  4. ESXi/ESX hosts with visibility to RDM LUNs being used by MSCS nodes with RDMs may take a long time to boot or during LUN rescan (1016106)
  5. An ESXi/ESX host reports VMFS heap warnings when hosting virtual machines that collectively use 4 TB or 20 TB of virtual disk storage (1004424)
  6. ESXi 5.x host fails with a purple diagnostics screen when using Intel(R) Xeon(R) CPU E5-2### and E5-46## (2033780)
  7. ESX/ESXi hosts might experience read/write performance issues with certain storage arrays (1002598)
  8. vHBAs and other PCI devices may stop responding in ESXi 5.x and ESXi/ESX 4.1 when using Interrupt Remapping (1030265)
  9. ESXi 5.x hosts fail to mount VMFS5 volumes that are formatted with ATS-only capabilities (2006858)
  10. Cannot take a quiesced snapshot of Windows 2008 R2 virtual machine (1031298)

Top 10 vSphere 5.1 Informational/How To Articles

  1. Installing async drivers on ESXi 5.x (2005205)
  2. Investigating virtual machine file locks on ESXi/ESX (10051)
  3. Using esxtop to identify storage performance issues (1008205)
  4. Recreating a missing virtual machine disk (VMDK) descriptor file (1002511)
  5. Commands to monitor snapshot deletion in ESX 2.5/3.x/4.x and ESXi 3.x.x/4.x.x/5 .x (1007566)
  6. Permanent Device Loss (PDL) and All-Paths-Down (APD) in vSphere 5.0 (2004684)
  7. vSphere handling of LUNs detected as snapshot LUNs (1011387)
  8. Best practices for virtual machine snapshots in the VMware environment (1025279)
  9. Manually regenerating core dump files in VMware ESXi/ESX (1002769)
  10. Determining Network/Storage firmware and driver version in ESXi/ESX 4.x and 5.x (1027206)

Current Trending Issues and how to fix them

We’ve been doing some trending analysis to figure out the issues that our support folks are running into, beginning with System Management and Networking. Based on our analysis so far, here are the 5 top areas where our customers are encountering issues today. For each problem area, we provide the most frequent resolutions to those problems.

Top Issues in System Management

vCenter Server installation issues Troubleshooting a failed vCenter Server installation (1003890)
Problems with ESXi hosts (hardware issues) Common ESX/ESXi host configuration issues which can cause virtual machines to become unresponsive (1007813)
vCenter Server database problems corrupt or slow Troubleshooting vCenter Server database configuration issues (1003960)
Issues virtual machines, not able to manipulate them (ie. Power Operations, vMotion, etc.) Troubleshooting a virtual machine that has stopped responding (1007819)
Issues getting performance statistics from vCenter
Troubleshooting gaps in performance data or missing performance data in vCenter Server (1003878)

 

Top Issues in Networking

Network Datapath problems
Identifying Fibre Channel, iSCSI, and NFS storage issues on ESX/ESXi hosts (1003659)
Network Performance issues
Troubleshooting network performance issues in a vSphere environment (1004087)
Network Guest OS issues
Troubleshooting virtual machine network connection issues (1003893)
Networking Host Issues Identifying Fibre Channel, iSCSI, and NFS storage issues on ESX/ESXi hosts (1003659)
Network vSS issues ESX/ESXi hosts have intermittent or no network connectivity (1004109)

Poor network performance or high network latency on Windows vms

We recently updated one of our popular performance related KB articles, and let’s face it, we’ve never had a complaint from a customer about a vm performing too quickly!

This KB article tells us that there are three possible causes for this issue-

  1. Power plan
  2. High CPU ready time (also referred as “%RDY” or “%RDY time”)
  3. Receive Side Scaling (RSS)

What I saw in the KB under the Power plan heading of particular interest was the the line stating:

“On Windows 2008 and 2008 R2, the power plan is set to Balanced by default. Microsoft has observed and confirmed that changing the power plan from Balanced to High Performance may increase overall performance”.

The default is not going to be the best performing. The same is true for Windows 2012 Server as far as I know. Something to check. The KB is definitely worth a read.

Poor network performance or high network latency on Windows virtual machines (2008925)