Home > Blogs > VMware vSphere Blog > Monthly Archives: December 2009

Monthly Archives: December 2009

Winner of Cycle 6 on the Cisco Nexus 1000V!

Congratulations to Jason Nash for winning this cycle of the vSphere blog contest.

Jason's entire entry can be seen below and also at:


Revisiting the Components of the Cisco Nexus 1000v

This is an update and bit of a rewrite of an earlier post I made here.  I wanted to add more detail and information now that I’ve worked with the Nexus 1000v more.

If you’ve read a lot of the Varrow blogs you’ll see information and talk about Cisco’s Nexus technology and products.  To be blunt, it can be confusing and a bit convoluted.  The hardware products, the Nexus 5000 and 7000, have been out for a little while now and we’re seeing more and more interest in those as companies see the need for high speed connectivity and the benefits of FCoE, especially now that FCoE is a standard.  But I think the Nexus 1000v is still a mystery to a lot of people.  There is a lot of “It’s really cool!” information out there but not a lot on how it really works.  One thing I’ve found is that the concept of the 1000v is very nebulous to many customers.  I can white board out how it works, the pieces, how they talk but it’s hard to “get it” without seeing it.  For that reason I created a nice demonstration video that is here.  If you’re new to the 1000v I highly suggest you check it out and see if it makes things click a little better for you.

Now let’s get in to some specifics.

The New Distributed Switch

Before getting in to the 1000v you first need to understand the new distributed switch in vSphere.  The Nexus 1000v uses this framework to provide much of its functionality.  In fact, from within vCenter the 1000v looks very much like the standard shipping distributed switch (dvSwitch).  But, while you can make changes to the dvSwitch in the vCenter GUI you can’t do that with the 1000v.  All changes to that must be done via the Nexus-OS command-line environment.  The idea behind the distributed switch is you now have one place to do the configuration and management of the network connectivity for your entire ESX cluster.  In the past you had to manually create vSwitches and Port Groups on every ESX server you brought up in the cluster.  With the distributed switch you configure your Port Groups just like you want in vCenter.  When a new ESX server moves in to the cluster and is joined to the dvSwitch (distributed virtual switch) it automatically sees the configuration in place.  It’s really great.  I’d say network configuration is the hardest part of the install for most VMware administrators and often a point of contention between the server admins and the network admins..maybe almost as bad as the contention between server guys and storage guys!

The benefit for organizations with separate teams, such as network and server admins, is that this moves “the line” back.  Before we virtualized everything the demarcation point between the server and the network was pretty simple, it was at the switch port.  Now that we’ve virtualized that has moved and sits somewhere in the ESX server so now a lot of that responsibility falls, correctly or not, on to the server admin.  They are th
e ones configuring networking and networking policies.  While the Nexus 1000v doesn’t move the line back to the physical switch port it gives the network team a virtual switch in the cluster that looks, feels, and acts like a hardware Cisco switch just like they are used to managing.

Components of the Nexus 1000v

When deploying the 1000v there are a few moving parts, not a lot but a couple.  The two primary pieces are the Virtual Supervisor Module (VSM) and the Virtual Ethernet Module (VEM). One thing that helps a lot of people is to imagine the 1000v like a multi-slot chassis switch such as a 6509 or a 4507R.  The VSMs are the supervisor management modules and the VEMs are the I/O port blades that provide connectivity.  The 1000v can have up to two VSMs and 64 VEMs which equates to a 66-slot chassis switch.  That’s a big switch!

So let’s look at the components in a bit more detail.  First is the VSM as it is the central management authority.

Virtual Supervisor Module

The VSM is a virtual version of a hardware supervisor module.   Some switches have redundant modules and the Nexus 1000v is no different.  The VSM runs as a virtual machine  on an ESX server in the cluster.  To provide fault tolerance you can run a second VSM in a standby role.  The secondary VSM will take over if the primary should fail.  Like a physical Supervisor Module there really isn’t any extra maintenance or management needed to run the second one.  Any configuration change on the primary is automatically replicated to the secondary.  So…I don’t see why you wouldn’t run two.  Also like many physical switches the supervisor modules do not have stateful failover, meaning they don’t share current information.  When the primary supervisor fails the secondary reboots, reads in the configuration, and starts working.

It’s important to note that the VSM is not in the data path.  That means it does not stop data flow through the cluster if the VSM should go down.  You won’t be able to make management changes but your VMs will continue to talk.  As of the latest version, 4.0(4)SV1(1),  you can now vMotion the VSMs around as long as it’s on an ESX server with a VEM installed that the VSM itself manages.  Just be sure to exclude it from DRS so you know when it moves!

Virtual Ethernet Module

This is where it gets really cool.  On each vSphere server in the cluster you install the Nexus 1000v VEM.  This is a piece of software that ties that server in to the distributed switch of the Nexus 1000v.  When you install and attach the VEM to the VSM for the cluster that ESX server’s VEM appears as a module on the switch.  So just like you log in to a Cisco chassis switch and do a “show module” you’ll do the same here.  Each ESX server will be its own module.  And that’s why it’s called a Virtual Ethernet Module.  There is an example of the “show module” output below.

How the Modules Communicate

So you have one or more VSMs installed as the brains of the switch and you also have some VEMs installed on ESX servers to act as the access port modules.  How do they talk to each other?  This is an important thing to understand as you need to get this before you start rolling out your VSMs.  Basically the Nexus 1000v uses a couple of VLANs as layer 2 communication channels.  These are the control and packet VLANs.  Their purpose is:

  • The Packet VLAN is used by protocols such as CDP, LACP, and IGMP.

The Control VLAN is used for the following:

  • VSM configuration commands to each VEM, and their responses.
  • VEM NetFlow exports are sent to the VSM, where they are then forwarded to a NetFlow Collector.
  • VEM notifications to the VSM, for example a VEM notifies the VSM of the attachment or detachment of ports to the distributed virtual switch (DVS).

Cisco recommends that the Control VLAN and Packet VLAN be separate VLANs; and that they also be on separate VLANs from those that carry data.  In the original version of the 1000v the VSMs and VEMs had to be on the same layer 2 network.  As of version 1.2 you can have layer 3 connectivity from the VSMs to the VEMs.  Just be sure all of the VEMs that are managed by these VSMs can talk via layer 3.  To put it simply, the VSM can be on another network but all the VEMs it manages must be together.  With Cisco releasing a hardware VSM appliance you’ll see more use cases for this functionality.  The vCenter server can be on a different layer 3 network than the VSM, that doesn’t matter.  Before you start deploying VSMs and VEMs you need to decide which VLANs you plan to use for packet and control and then create them on your physical switches that connect ESX servers.  To reiterate, the Control and Packet VLANs are for layer 2 communication.  This means that no IP addresses or default gateway needs to be assigned to these VLANs so just pick any unused VLAN numbers and get them going.

Deploying the Components

This particular post is not meant to be an in-depth guide to installing the 1000v.  Cisco offers their documentation here.  The point of this post is to give you a deeper understanding of how everything fits together so you better understand the function of the 1000v and how it integrates in to your environment.

Deploying the VSM is very simple.  It’s distributed as a ..ova/ovf file set and is easily added to the cluster via the vCenter client. Version 1.2 and above make installation even easier than the original version.  On the original you just imported the .ovf and booted the VSM.  Once booted you were presented the familiar “basic setup” Cisco walkthrough.  As of 1.2 you import the VSM appliance using a .ova file which walks you through a GUI wizard to configure those same items.  It just makes it simpler and faster than before.  In the wizard you’ll be asked for the packet and control VLANs as well as a “domain ID”.  This just defines the VSMs and VEMs that are working together.  Pick an unused number, I start with 1, and continue.  If you have multiple 1000v installations you’ll need to assign unique Domain IDs to each one.

Installing the VEMs isn’t much more difficult.  You have the option of doing it manually on each ESX server or using Update Manager.  The choice is yours.  Manually is easy as it’s a single command once you get the package file on the server.  There is no manual configuration needed on each VEM.  Everything will come from the VSM or vCenter.  If you scp the .vib file to the ESX server all you need to type to install it is:

esxupdate -b *.vib update

Once that is done you can check the installation by using the “vem status” command, such as:

[root@labclt-esx01 ~]# vem status
VEM modules are loaded
Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       32          10          32                1500    vmnic0
DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks
nexus1kv       256         51          256               vmnic1
VEM Agent (vemdpa) is running

That’s it.  Nothing else to do on each system.  Going forward I recommend updating the VEMs using Update Manager.  You can just add a repository in to Update Manager to get patches for the VEMs.  Makes it very easy.  The VSMs take a bit more work to update…but we’ll cover that in another post soon.

The final piece is some configuration in vCenter.  You have to install the Nexus 1000v plugin as well as perform some configuration on the VSM to connect it to vCenter.  This is easily done by opening a web browser and pointing it to the IP of the VSM you installed.  From there you can install the plugin, which is just a simple XML file to allow communication between the VSMs and vCenter.  When that happens your new distributed switch appears and the real fun begins….

Beyond Installation

Assuming everything went well and all the pieces are talking you are ready for configuration.  The first step is to bring each ESX host in the cluster and assign some unused NICs to uplink ports. You can kind of think of uplinks like vSwitches in the normal vSwitch that we are all used to.  Basically, an uplink defines which traffic goes over which physical NICs.  These are how you separate traffic.  So if you want vMotion over one set of NICs and VM Network traffic over another set you’d create two uplinks and assign the appropriate NICs to each.  Here is an example config for an uplink:

port-profile type ethernet system-uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 20-30
no shutdown
system vlan 10-12
state enabled

This is a very simple uplink called system-uplink.  It is allowed to carry all VLANs so all traffic in all port-groups will flow over any NICs assigned to this uplink.  Notice the “system vlan” option.  This specifies that the special VLANs such as Control and Packet can flow over this uplink.   You need to do this if you’re going to move those to the dvSwitch, which you’ll probably eventually do.  One thing worth mentioning is the “switchport trunk allowed vlan” command.  This tells the switch which VLANs can ride of this uplink.  This is how you relate port-groups to uplinks.  So a port-group used for VM traffic on VLAN 21 will ride over this uplink and these NICs.  If another port-group is used for VM traffic or a service console on VLAN 40 it will not go over this uplink and you’d need to define another uplink that could service that VLAN.  If you assign more than one they must be in a channel group.  The Nexus 1000v does not run spanning tree and therefore doesn’t like loops and will disable the connections if it sees duplicates.

If you’re using 10Gb CNA adapters you may only have two ports on the server which means, usually, everything will ride over a single uplink with redundant connections.  If you’re concerned about bandwidth problems due to heavy traffic features such as vMotion or FT you can use QoS to alleviate that.  If you’re in a legacy type configuration with 6 or 8 Gb connections you’ll probably have multiple uplinks just like you had multiple vSwitches before the move to the dvSwitch.  Some of the best practices we’ve had for a while still apply here, they may just be implemented a little differently.

Now that all hosts are attached to the distributed switch you can start creating port groups.  Creating port groups is done from the VSM via the Cisco NX-OS command-line.  NX-OS is the Nexus OS and is very similar to IOS.  Anyone used to IOS should feel at home and will find many nice enhancements.  Within NX-OS you create Port Profiles, which turn in to port groups within vCenter and the distributed switch.  Below is an example configuration for a Port Profile used to let VMs talk to the network.

 port-profile VM_VLAN300
vmware port-group
switchport mode access
switchport access vlan 300
no shutdown
state enabled

In many ways it is similar to a port configuration on a physical switch.  This creates a port group in vCenter named VM_VLAN300 that lets VMs talk to other devices on VLAN 300.  Below is a screenshot showing this port group available in the settings for a VM.

Screen shot 2009-07-17 at 7.35.11 PM

Behind the scenes the VMs and port groups act like Cisco devices.  As mentioned before, each ESX server is like a switch module.  For example:

vc4labsw# show mod
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
3    248    Virtual Ethernet Module          NA                 ok
4    248    Virtual Ethernet Module          NA                 ok
Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1       NA                                    NA
3       33393935-3234-5553-4539-32344e36464b  vc4lab1.varrow.com
4       33393935-3234-5553-4539-32344e36464c  vc4lab2.varrow.com

From that you can see there are two VEMs installed.  One on vc4lab1 and the other on vc4lab2, my two vSphere servers in this lab.  When a new connection is made to the dvswitch, whether it’s a VM or something like a Service Console, that connection is given a Virtual Ethernet interface or port assignment.  The connection will always keep the same port number no matter where it lives in the cluster.  Example again:

vc4labsw# show int bri
Interface     VLAN   Type Mode   Status  Reason                   MTU
Veth1         300    virt access up      none                     1500
Veth2         300    virt access up      none                     1500
Veth3         1      virt access up      none                     1500
Veth4         1      virt access up      none                     1500
Veth5         400    virt access up      none                     1500
Veth6         400    virt access up      none                     1500
vc4labsw# show int ve3
Vethernet3 is up
Port description is JN_Win2k8, Network Adapter 1
Hardware is Virtual, address is 0050.568a.31cf
Owner is VM "JN_Win2k8", adapter is Network Adapter 1
Active on module 3
VMware DVS port 352
Port-Profile is VM_VLAN1
Port mode is access
13616 Input Packets 13220 Unicast Packets
23 Multicast Packets 373 Broadcast Packets
1039898 Bytes
198323 Output Packets 87263 Unicast Packets
13016 Multicast Packets 98044 Broadcast Packets 19 Flood Packets
80092259 Bytes
8 Input Packet Drops 0 Output Packet Drops

In the example I performed a “show int brief” to get a concise list.  Notice it shows the interface and VLAN but not the type of connection.  If I narrow it down and look at ve3, Virtual Ethernet 3, I see that this is my Windows 2008 server named JN_Win2K8.  Notice the information displayed.  Very similar to a port on a physical switch.  No matter where this machine goes those statistics and information follow it.


There isa lot of good information out there on the Nexus 1000v now.  Cisco has a very good community site here.  If you’re starting out with the new virtual switch hopefully this post was useful and filled in some of the gaps.  Take a look at the video referenced above as well as the documentation from Cisco.  Cisco has very good documentation and install guides that will walk you through a normal deployment.  The key is to extrapolate what you need from those documents and fit it in to your environment.

Start of Cycle 7 – Tier 1 Applications

Greetings and happy holidays! This week starts our focus on virtualizing tier 1 applications. While some users have only used VMware for virtualizing tier 2 and 3 apps, now is the time to change that thinking. The release of vSphere 4.0 totally changes the conventional views many of us have had when evaluating apps for the virtualization platform. Please take a moment to write a blog entry about your thoughts on virtualizing tier 1 apps or any testing results you have seen or experienced. You have up until Jan. 8th to write a post to be entered in this last cycle of the vSphere blog contest.

See more reasons why VMware is the best platform for enterprise applications at:


ESXi 4.0 supports Jumbo Frames

Some people out there, such as in this post, have noticed a statement in the ESXi 4.0 Configuration Guide which says (on page 54):

"Jumbo frames are not supported for VMkernel networking interfaces in ESXi"

I am happy to say that this is merely an error in the documentation.  In fact, ESXi 4.0 DOES support Jumbo Frames on VMkernel networking interfaces.  The correction will hopefully appear in a new release of the documentation, but in the meantime, go ahead and configure Jumbo frames for your ESXi 4.0 hosts.

Cisco Nexus 1000V R1.2 for vSphere 4 Released

Just on a week ago, Cisco released the Nexus 1000V R1.2 virtual switch for vSphere 4. This introduced a bunch of new features mainly around setup/installation and security. The full list of new features are as follows:

  • GUI setup following software install
  • Layer 3 control between VSM and VEMs
  • Virtual Service Domains for classifying and separating traffic for network services
  • iSCSI Multipath—supporting multipath feature introduced in vSphere 4
  • XML API for developing client apps for managing/monitoring the Nexus 1000V
  • DHCP Snooping for validating DHCP messages and filtering invalid responses
  • Dynamic ARP Inspection for validating ARP requests and responses
  • IP Source Guard for filtering traffic on interfaces to valid MAC and IP addresses 
  • MAC Pinning for assigning Ethernet port members to particular port channel subgroup (where upstream switches do not support port channels)
  • Static Pinning

This is quite a list of new features for a dot release. I know of a couple of customers who will warmly welcome the new L3 control feature.

You can find a write up of these features in the Cisco Nexus 1000V Release Notes.  You can find the compatibility information here.

We have also updated the VMware vNetwork Distributed Switch and Nexus 1000V feature matrix. It is posted here on the Cisco website and will be available at the vmware.com/go/networking site in the next few days.

How Do I Get Started with the Nexus?

I would recommend the following Cisco videos to help with the installation and configuration of the Nexus 1000V.


Further Detail on the Nexus 1000V

We started with the basics about the Nexus in previous posts.

(See http://blogs.vmware.com/vsphere/2009/12/tell-me-more-about-the-cisco-nexus-1000v.html)

Now, let's get into further details.

Cisco Nexus™ 1000V is the result of a Cisco and VMware collaboration building on the VMware vNetwork third-party vSwitch API of VMware vDS and the industry-leading switching technology of the Cisco Nexus Family of switches. Featuring the Cisco® NX-OS Software data center operating system, the Cisco Nexus 1000V Series extends the virtual networking feature set to a level consistent with physical Cisco switches and brings advanced data center networking, security, and operating capabilities to the vSphere environment. It provides end-to-end physical and virtual network provisioning, monitoring, and administration with virtual machine–level granularity using common and existing network tools and interfaces. The Cisco Nexus 1000V Series transparently integrates with VMware vCenter Server to provide a consistent virtual machine provisioning workflow while offering features well suited for data center-class applications, VMware View, and other mission-critical virtual machine deployments. 

CapacityIQ 1.0.1 Release Supports vSphere! Available Now!

I know many of you have been waiting for this to happen.

Release notes - http://www.vmware.com/support/ciq100/doc/releasenotes_ciq101.html

Continuing vSphere Blog Contest on the Nexus 1000V until 12/18!

Our contest for the Nexus 1000V cycle remains open for one additional week. Get your entries in today!

VMware vCenter Site Recovery Manager 4.0 Performance and Best Practices White Paper Posted

VMware vCenter Site Recovery Manager 4.0 is a component of VMware Infrastructure that ensures a speedy and successful datacenter recovery by automating the recovery process and eliminating the complexity of managing and testing recovery plans. A new white paper, VMware vCenter Site Recovery Manager 4.0 Performance and Best Practices, is now available at http://www.vmware.com/files/pdf/VMware-vCenter-SRM-WP-EN.pdf

You can also find it under the technical resources section at http://www.vmware.com/products/site-recovery-manager/resource.html.

In this paper we discuss VMware vCenter Site Recovery Manager 4.0 performance, various dimensions on which the recovery time depends and certain performance tips/considerations on architecting recovery plans to minimize recovery time – few of which are mentioned as follows :
-> Fewer but larger NFS datastores can help speedup a recovery

-> Specifying a non replicated datastore for swap files will eliminate certain remote calls during a recovery and will help improve recovery performance.

-> Even placement of shadow virtual machines across all hosts on the recovery site will ensure maximum parallelism during a recovery.

and so on…

You will also find Database sizing guides under the Tools section which will help you in estimating the Site Recovery Manager Database growth as a function of the number of protection groups, recovery plans and more .

Week 2 – Cisco Nexus 1000V – Let’s See Those Blog Entries

Hey folks! We are excited to start week 2 of our focus on the Nexus 1000V. Get those blog entries in and grab your chance at our $100 prize. More to come on the Nexus 1000V shortly.