VMware Networking Blog

Beaconing is one of those features that often confuses even the most experienced networking admin.

Shudong Zhou, one of our senior engineers, recently posted an entry on the internal blog explaining how it works and how you might use it. He gave me permission to cut and paste his entry. Here it is …

Beaconing is a software solution for detecting link failures downstream from the physical switch.ESX provides a simple and elegant teaming solution. All uplinks connected to a vswitch are assumed to connect to the same physical network (same broadcast domain) so they are all equivalent. Users can configure a list of active and standby uplinks for traffic to go out of the ESX host. If a link fails, the adapter driver detects it and marks the uplink as failed and stops using this uplink. Existing traffic will fail over to a standby uplink or redistributed to the remaining team members.

If a downstream link beyond the immediate physical port fails, the adapter driver obviously cannot detect it. This causes existing VMs using the uplink to lost network connectivity. The proper way to solve this problem is to enable Link State Tracking on the physical switch so that the adapter driver can see the failure. If the physical switch does not support Link State Tracking, beaconing provides a software alternative. Beaconing works as follows:

ESX periodically broadcast beacon packets out of all uplinks in a team. The physical switch is expected to forward all packets to other ports on the same broadcast domain. Hence, a team member is expected to see beacon packets from other team members. If an uplink fails to receive any beacon packets (actually missing 3 consecutive packets), we mark it bad. The failure can be due to the immediate link or a downstream link. With 3 or more uplinks in a team, we can pin point failures of a single uplink. With 2 uplinks in a team, we can detect downstream link failure, but we don't know which one is good and which bad.

ESX behavior when a beaconing failure is detected is as follows:

1. If two or more uplinks receive beacons from each other, those uplinks are considered good. We stop using uplinks which do not receive any beacon packets.
2. On ESX 3.5, if no uplink receives beacon packets, traffic is sent to all uplinks (shotgun mode). If a team has two uplinks, any link failure will result in all packets being sent to both uplinks.
3. On a future edition of ESX, we intend to make an additional improvement. If no uplink receives beacon packets, traffic is only sent to uplinks whose link status is “up”. If a team has two uplinks and one uplink experiences a failure in its immediate link, traffic will be sent out to the other uplink. This saves some CPU cycles.

When should one enable beaconing? When you are concerned that downstream link failures may impact availability and there is no Link State Tracking on the physical switch. Ideally, you should have 3 or more uplinks in the team (active + standby). But you can enable beaconing with 2 uplinks. Some customers don't like the shotgun mode on failure (see #2 above), that's a trade off you should make against some VM losing connection right away.

A few recent conversations with customers have quickly gravitated to the topic of 10 GigE. Some customers are making firms plans for deploying 10GigE in their ESX servers. The reasons vary, but top amongst these customers are:

1. Improving managability by reducing the cable and NIC sprawl. 2x 10GigE links versus 6, 8, 10 or more 1 GigE can simplify your infrastructure
2. Convergence of Fibre Channel and Network traffic using FCoE (Fibre Channel over Ethernet). The ability to do away with Fibre Channel HBAs and converge the FC SAN traffic with the ethernet network traffic is the TCO tipping point for some customers.

As you may be aware, FCoE is supported in ESX 3.5u2. Emulex and Qlogic 10 GigE FCoE CNAs (Channel Network Adapters) were qualified and added to our HCL just prior to VMworld in September. 

But, ...how does 10GigE perform on ESX 3.5? Our performance team published a paper this week on that very theme.   

Some of you may have caught the Cisco TechwiseTV episode yesterday on virtualization. The second segment of the program featured Dan Hersey from Cisco demonstrating the Nexus 1000V virtual switch running on a future generation of VMware ESX. Nexus 1000V is Cisco's third party switch implementation for VMware Virtual Infrastructure that was announced at VMworld in September. Anyone who has configured a Cisco switch (or just about any other switch for that matter!) should find the Nexus1000V command line interface very familiar! 

Cisco is offering $100k for the best application written to their AXP (Application Extension Platform). That's a lot of money!

The cool thing about this is that you can develop this on an AXP VM running on VMware Player on your PC. i.e. you  don't have to buy any Cisco hardware & software. Here is the press release.

At VMworld a few weeks back, we previewed the future of virtual networking with the vNetwork Distributed Switch and third party virtual switch support with the Cisco Nexus 1000V. I am delivering a webcast on October 16, titled "Next Generation Virtual Networking" where I will cover these announcements and give you some technical insight into what's coming. You can register here. 

The customer reaction to our networking announcements at VMworld has been fabulous. People really seemed to understand and see the relevance of our push toward greater manageability of the virtual network through our vNetwork Distributed Switch and how we are further embracing the "network team" and traditional network administration policies and boundaries through third party virtual switch support with the Cisco Nexus 1000V.

Last week I was guest on the VMware Communities Roundtable podcast where we discussed VDS and the Cisco Nexus 1000V. You can download the mp3 podcast here or stream/listen to it here (51 minutes).

Cisco, of course, has posted a great deal of information on the Nexus 1000V on their datacenter site. Be sure to clisk on the video where Doug Gourlay from Cisco interviews me in the Cisco studios about the new features. :-)

There is nothing like hands on or a demo to get the hang of a new concept or new feature. For those of you who didn't get some hands on time at VMworld with our vNetwork Distributed Switch (VDS), there is a great flash demo showing the configuration process of DVS.

While you are there, take a look at the newly posted information on our direction toward the "Virtual Datacenter OS" that will transform the data centers into an cloud of computing capacity.

And, if you are in the mood for a few more instructional videos on new features, take a look at the flash demos of Storage VMotion, Fault Tolerance, and Virtual Center.

Yesterday was a huge day for virtualization and virtual networking. Jacob Jensen, accompanied by Paul Fazzone from Cisco, started the day with a tech preview to

• vNetwork Distributed Switching, and
• 3rd Party virtual switch support with the Cisco Nexus 1000V

VDS is a major step forward in easing the administration burden of large virtual networks. VDS enables the configuration of "data center" level distributed virtual switches with corresponding distributed virtual port groups. The actual switching is still done locally on the individual vswitches, but the administration and control plane function is raised up into virtual center (VC). Now, when VMs migrate using VMotion, the actual port state will also migrate from vswitch to vswitch, maintaining consistency for statistics, security functions and other features.

The Cisco Nexus 1000V takes this a step further. A Virtual Supervisor Module (VSM) on a VM or separate appliance administers a number of Virtual Ethernet Modules (VEMs) that replace the vswitches on the ESX hosts. It resembles a modular switch with a supervisor (VSM) and linecards (VEMs). So what does this mean? It means a full cisco feature set with ACLs, QoS, TrustSec, Netflow v9, etc and of course a Cisco command line interface. To a network admin, it means he or she can administer and monitor the network to individual VMs in the same way and with the same control as was possible to physical servers before virtualization. This is a big deal and should be of particular interest to enterprises with distinct network and server/VI administration teams.

Anyway, back to the day's activities ...

Jacob's session was followed by a Platinum Keynote from Ed Bugnion (CTO for Cisco's Server Access & Virtualization BU and co-founder of VMware). Ed's session was enormous with standing room only. Ed's mention of the two year partnership between Cisco and VMware and the announcement of the Nexus1000V drew thunderous applause. It was great validation that this was the right thing to do.

I've quickly scoured a few blogs for mention of yesterday's announcements. Here are just a few from Colin McNamara, Data Center Knowledge, and Light Reading. And, of course, Omar Sultan's take on yesterday from a cisco perspective.

Anyway, what do you think? How will these technologies affect your environment? I welcome your comments.

VMworld is the biggest event on our calendar and it will not be short of exciting announcements in the world of virtualization and networking. If you're interested in networking and are going to VMworld here's my hitlist of "must attend" sessions (in chronological order):

1. Tuesday, Sep 16, 9:30am - TA2280 - Tech Preview: Virtual Switching Extensibility with VMware Infrastructure (Jacob Jensen, VMware & Paul Fazzone, Cisco) - Delfino 4102
2. Tuesday Sep 16, 11:00am - Cisco Keynote: Designing the next generation Data Center - Cisco and VMware (Ed Bugnion, VP/CTO, Cisco) -
3. Wednesday, Sep 17, 11:30am - TA3720 - Designing Data Centers with VM level Granularity (Paul Fazzone & Omar Sultan, Cisco) - Zeno 4609
4. Wednesday, Sep 17, 3:00pm - TA2275 - Tech Preview:  VMware Infrastructure Virtual Networking – Future Directions (Andrew Lambeth, VMware) - Delfino 4005

If you want to get an understanding of the concepts and best practices of virtual networking then attend my session (TA2441) on Thursday at 11:00am in Titan 2203 or the repeat at 3:30pm in Marcello 4405. Shameless plug.

One of our senior engineers, Srinivas Neginhal, is holding an advanced network configuration and troubleshooting session on Thursday Sep 18 at 2:00pm in Zeno 4708.

Note that Ed Bugnion, the Cisco keynote speaker (#2 in the list above), is a co-founder of VMware and ex-CTO. He is now the CTO of Cisco's Server Access and Virtualization BU. Ed is a great speaker and uniquely positioned to speak about the converging topics of networking and virtualization.

As a Platinum Sponsor of VMworld, some of my friends at Cisco are posting a few blog entries in anticipation of next week.   

Hope you all enjoy the conference! Should be fun. 

I was speaking with a customer recently about networking when the topic turned to the virtual nic or vnic bandwidth available between a VM and the virtual switch. The customer complained that the server guys  were performing "gymnastics" in trying to change the speed from its reported value of 100Mbps to a more palatable speed of 1Gbps.

Fortunately, one of our senior engineers was present at the meeting and was able to explain (more completely than me) that the bandwidth *reported* by the guest VM on the vnic interface is a function of the NIC driver on the guest VM (e.g. e1000, vlance, etc). The actual vnic is implemented in software, so is only limited by the processing capabilities of the host itself.

So, ...ignore what the guest reports as the NIC link speed ...(and bad on us for not making this more apparent!)

p.s. if you *do* want to set a limit on the traffic produced by a VM, you can use the "TX Rate Limiting" feature in the Port Group definition. Here you can set the maximum and average bandwidth as well as a "burst" size.