Home > Blogs > The Network Virtualization Blog > Category Archives: Network Virtualization

Category Archives: Network Virtualization

Deploying VMware NSX on Cisco Nexus 9000 & Cisco UCS Infrastructure

As VMware NSX gains broader adoption, we have heard many customer requests for guidance to help them run NSX on top of the latest Cisco infrastructure, namely Cisco UCS and Nexus 9000 series switches.

With customers choosing the benefits of VMware NSX along with the Software Defined Data Center (SDDC), the underlying hardware (Ethernet fabric, x86 compute, etc) provides reliable, resilient capacity, but the configuration, state and advanced features move to faster, more flexible software. The requests were for deploying NSX with Cisco infrastructure running in a standard IP-based fabric with the Nexus 9000’s in standalone mode (NX-OS Mode), as opposed to the proprietary ACI Mode. As with any IP fabric, VMware NSX works great with Nexus 9000 as the underlay. The combination of VMware NSX and Nexus 9000 in standalone mode enables the benefits customers have chosen to embrace with the SDDC.

We had previously put out a design guide on deploying VMware NSX with Cisco UCS and Nexus 7000 to help deploy NSX in current environments. Today we are putting out a new reference design for deploying VMware NSX with Cisco UCS and Nexus 9000 infrastructure, providing an easy path to the SDDC while incorporating the latest Cisco hardware.

The reference architecture along with the VMware NSX for vSphere Network Virtualization Design Guide provides guidance for network virtualization architects interested in deploying VMware NSX for vSphere for network virtualization with Cisco UCS blade servers and Cisco Nexus 9000 Series switches. It discusses the fundamental building blocks of NSX with VMware ESXi, recommended configurations with Cisco UCS and connectivity of Cisco UCS to Nexus 9000 switches.

VMware NSX on Nexus 9000

click to enlarge

VMware sees these requests as a clear indication that customers have voted clearly for the software-defined data center. Along the way, we have had many customers adopt and deploy NSX to virtualize their networks, such as Columbia Sportswear, WestJet, IlliniCloud, Synergent, JOIN Experience, TradeStation, USDA, NTT communications, PayPal, eBay, McKesson, Medtronic…I think you get the picture.

And whether the underlying network is old Cisco, new Cisco, or no Cisco, we will continue to help with valuable resources such as this to help customers succeed.

Nikhil

VCDX-NV Interview: Ron Flax On The Importance Of Network Virtualization

Ron Flax is the Vice President of August Schell, a reseller of VMware products and IT services company that specializes in delivering services to commercial accounts and the federal government, particularly intelligence and U.S. Department of Defense. RonFlaxRon is a VCDX-NV certified network virtualization professional and a VMware vExpert. We spoke with Ron about network virtualization and the NSX career path.

***

The most exciting thing about network virtualization, I think, is the transformative nature of this technology. Networks have been built the same way for the last 20 to 25 years. Nothing has really changed. A lot of new features have been built, a lot of different technologies have come around networks, but the fundamental nature of how networks are built has not changed. But VMware NSX, because it’s a software-based product, has completely altered everything. It enables a much more agile approach to networks: the ability to automate the stand-up and tear-down of networks; the ability to produce firewalling literally at the virtual network interface. And because things are done at software speed, you can now make changes to the features and functions of networking products at software speed. You no longer have to deal with silicon speed. It’s very, very exciting. With a software-based approach, you can just do so much more in such a small amount of time.

What we’re hearing from customers, at this point, is that they’re very interested to learn more. They’re at a phase where they’re ready to get their hands dirty, and they really want to understand it better. What’s driving a lot of adoption today is security, it is our foot in the door. When you speak with customers about the security aspects, the micro-segmentation capabilities, you may not even have to get to a virtual network discussion. Once you get the security aspect deployed, customers will see it in action and then a few weeks later will say, ‘Hey, you know, can you show me how the new router works?’ or ‘Can you show me how other features of NSX work?’ That’s when you can start to broaden your approach. So these compelling security stories like micro-segmentation or distributed firewalling get you in and get the deployment started, but ultimately it’s the flexibility of being able to deliver networks at speed, in an agile way, through software, through automation, that’s the home run.

I also think clients are excited about being able to deliver services more quickly to their business units. In the space I work in, the U.S. Federal Government, the workforce is typically segmented into a server team, storage team, network team, maybe a virtualization team. They haven’t gotten yet to the point where they have a cloud team, so it’s all kind of meshed together. What tends to happen in these siloed environments is the business, or the end user, is waiting on one of these factions to get their job done before they can deliver services. In a lot of cases it’s become the network team that acts as the long pole in the tent and gets things organized for getting a solution built. If they are the log jam, well…

With network virtualization it’s possible—it’s quite easy, in fact—to bring that capability to the virtualization guy, the server guy, the storage guy, or even the end user if you deliver this as a full Software-Defined Data Center or SDDC. Essentially you create a self-service interface, where the end user can actually build and create their networks for themselves. They no longer have to wait for the storage team to have enough storage, the network team to create the networks etc. They can do it themselves. So that’s a big “aha” moment for a lot of customers, They realize: ”we actually can deliver something secure, that works, and that’s isolated to the business in a reasonable amount of time.”

Seeing this transition made me realize that getting my VCDX-NV was a great opportunity. I just felt like if we were going to be in this market space, if we were going to be considered NSX experts, we had to have at least one person, if not many people, who were officially qualified by VMware. The experience was great. VMware went out of their way to really make a strong impression on us, and to invest in every candidate, to make it so that as many of us as possible would succeed and get through the process. I’m not going to say it wasn’t hard! The process is what it should be. It definitely will test you. But if you’re a network engineer, you’re going to want to learn as much as you can about networks. Certainly if you’re a CCIE and you have those skills, and you’ve passed certification for the physical network and all of the related design concepts. I would strongly advise you to get some form of NSX certification with VMware, even if it’s not the full VCDX-NV. The more you know, the more it’s going to help you. You still need to understand the underpinnings, the physical network, but you have that already, so take advantage. Learning about the software aspects of network virtualization can be instrumental in your job growth, your advancement. It’s going to help you in your career.

At the end of the day, this is technology. Technology changes very rapidly. Anybody who’s been around the technology world knows things change at a very, very quick pace. You can’t rest on your laurels. You have to retool yourself. You have to always retool yourself.

VCDX-NV Interview: Chris Miller Talks VMware NSX Certification

Chris Miller is the principal architect for AdvizeX in Columbus OH. He runs the NSX program from a technical and marketing perspective, including Chris Miller-AdvizeXenterprise pre-sales support and go-to-market strategies.

*** 

I started my career as a traditional Cisco networking guy. I spent 10 to 15 years as a network architect. But I’d been tracking what was going on in the community, with Open Flow and some of the other technologies. When I saw what VMware was doing, it got me pretty excited. I thought, ’It’s pretty revolutionary what’s going on here.’ I immediately jumped on the opportunity to take part in NSX.

In terms of enterprise customers, we weren’t initially seeing a lot of adoption in the market. Then VMware announced the Nicira acquisition, and Cisco announced what they were going to do with ACI, and heads started turning. I realized, you know, here are two of our largest partners putting their investment dollars behind this technology. And then, when I saw what NSX could do, and the benefits it could bring, it was very clear to me that this was the next wave.

What excites me most about network virtualization is that you essentially don’t have to worry about change control as much anymore. Now I can start building my services application to application. Everything is independent. I don’t have to get on the phone with folks and explain everything that I’m doing for every little change. It’s amazing. I am also excited about what this does for the private cloud. I think that the pieces that we’re missing for private cloud are primarily network and storage. We’ve had the compute for some time. This gives us a way to extract the networking pieces with NSX and the storage pieces with VMware. Now we can be hardware independent. Companies have been trying to look like Google and act like Google for years now; I think this is the technology that will finally enable them to do it.  So that is what is exciting, there is a there’s a whole new set of things for us to work on now – like private cloud.

Despite all this possibility, there are still people who aren’t convinced this is going to happen. Whether we like it or not, the industry’s changing. Networking’s changing. Even if you never did any network virtualization, you’re going to have to figure out how to integrate with the cloud—and a key component of that is the network. So us networking guys are going to have to change our skill sets, and we’re going to have to start thinking from a more converged perspective, from a cloud unintelligible perspective. By pursuing the advanced certification, you’re tooling up to understand that, and to be able to deal with what’s coming. So, to anyone who says he or she doesn’t really need to know about network virtualization, I’d say, “Ask mainframe guys how they feel about not needing to know S86.” It’s the same concept.

And getting certified now will have it’s advantages. Look at the CCIE, for example. Companies are seeking the low numbers, right? People will put ‘CCIE-50’ on their resumes. There’s a lot of prestige around that. Five years out, it’s going to be the same for VCDX-NV. So I’d say, if you can get in early, you’re getting in on a cutting-edge new technology; you’re getting a highly sought-after, well-respected certification before anybody else. Worst-case scenario? It builds your resume. Best case? It helps you tool up for the future. You’re either going to adopt, or you’re going to get left behind.

 

Introducing New VCE VxBlock Systems with Integrated VMware NSX

Last month, we outlined VMware’s vision for helping customers achieve one cloud for any application and any device. We believe the prevailing model for cloud adoption will be the hybrid cloud, and the best architecture for achieving the hybrid cloud is through a software-defined data center architecture. The fastest path to building reliable infrastructure for the hybrid cloud is through the use of converged infrastructure systems, and no company has been more successful at delivering on the promise of converged infrastructure than our partner VCE.

Now, the ability to procure and deploy the VMware NSX network virtualization platform with VCE converged infrastructure is about to get whole lot easier.

Today, VCE launched VCE VxBlock Systems, a new family of converged infrastructure systems that will factory-integrate VMware NSX for software-defined data center deployments. The new VxBlock Systems will include VCE pre-integration, pre-testing and pre-validation of VMware NSX, with seamless component-level updates, ongoing lifecycle assurance, and unified single-call support from VCE.

As I wrote previously, VMware NSX already runs great on existing Vblock Systems. Customers today are deploying VMware NSX with their existing Vblocks, and customers will be able to extend VMware NSX environments across their entire VCE converged infrastructure environment as they move to the new VxBlock Systems.

This solution will be a powerful building block for the software-defined data center, delivering unparalleled IT agility through automation, and unparalleled security through micro-segmentation.

Agility through IT Automation

  • Reduce time to provision multi-tier networking and security services from weeks to minutes.
  • Achieve faster development, testing and deployment of new applications by aligning network and security provisioning with compute and storage provisioning.
  • Streamline IT operations through programmatic creation, provisioning, snapshotting, deleting and restoration of complex software-based networks.
  • Build advanced workflows through cloud management platforms to automate provisioning of networking and security, including switching, routing, firewalling, and load balancing without manually reconfiguring physical network devices.

Unparalleled Security

  • Use micro-segmentation and isolation capabilities of VMware NSX to build security directly into the data center infrastructure.
  • Insert advanced partner services from leading security vendors to improve threat protection, reduce risk and help address their compliance requirements.
  • Achieve better security inside the data center through fine-grained policies that enable firewall controls and advanced security down to the level of the virtual NIC.
  • Create dynamic security policies that are automatically applied when a virtual machine spins up, are moved when a virtual machine is migrated and are removed when a virtual machine is de-provisioned

VMware NSX is the ideal platform for virtualizing the network running on top of VCE converged infrastructure.

Hatem

Network Virtualization at VMware Partner Exchange 2015

VMware Partner Exchange (PEX) is your one-stop shop when it comes to learning about network virtualization and the technology extends VMware’s vision of the software-defined data center. At this year’s event, we are offering both an executive track and a technical track to help partners build their businesses and advance their knowledge, as you take customers on the path to Virtualizing the Network.

PEX Image

If you are a partner that is new to network virtualization, we have a program/learning path where you can send two people to PEX and to achieve their network virtualization competency by attending the 3-Day NSX Install, Configure and Manage Boot Camp prior to the start of the conference. Participants can then attend the free instructor-led VSP-NV and VTSP-NV boot camps during the conference. Continue reading

OVS Fall 2014 Conference: Observations and Takeaways

Last week we hosted the Open vSwitch 2014 Fall Conference, which was another great opportunity to demonstrate our continued investment in leading open source technologies. To get a sense of the energy and enthusiasm at the event, take a quick view of this video we captured with attendees.

I’ve been thinking about the key takeaways from everything I saw and everyone I spoke with.

First, there’s huge interest in Open vSwitch performance, both in terms of measurement and improvement. The talks from Rackspace and Noiro Networks/Cisco led me to believe that we’ve reached the point where Open vSwitch performance is good enough on hypervisors for most applications, and often faster than competing software solutions such as the Linux bridge.

Talks from Intel and one from Luigi Rizzo at the University of Pisa demonstrated that by bypassing the kernel entirely through DPDK or netmap, respectively, we haven’t reached the limits of software forwarding performance. Based on a conversation I had with Chris Wright from Red Hat, this work is helping the Linux kernel community look into reducing the overhead of the kernel, so that we can see improved performance without losing the functionality provided by the kernel.

Johann Tönsing from Netronome also presented a talk describing all the ways that Netronome’s NPU hardware can accelerate OpenFlow and Open vSwitch; I’ve talked to Johann many times before, but I had never realized how many different configurations their hardware supports, so this was an eye-opening talk for me.

Next, enhancing Open vSwitch capabilities at L4 through L7 is another exciting area. Our own Justin Pettit was joined by Thomas Graf from Noiro to talk about the ongoing project to add support for NAT and tracking L4 connections, which is key to making Open vSwitch capable of implementing high-quality firewalls. A later talk by Franck Baudin from Qosmos presented L7 enhancements to this capability.

The final area that I saw highlighted at the conference is existing applications for Open vSwitch today. Peter Phaal from InMon, for example, demonstrated applications for sFlow in Open vSwitch. I found his talk interesting because although I knew about sFlow and had talked to Peter before, I hadn’t realized all of the varied uses for sFlow monitoring data. Vikram Dham also showed his uses for MPLS in Open vSwitch and Radhika Hirannaiah her use case for OpenFlow and Open vSwitch in traffic engineering.

I want to thank all of our participants and the organizing committee for helping to put together such an amazing event.

Ben

State of the State for Open vSwitch

This week, VMware will be hosting the Open vSwitch 2014 Fall Conference, with more than 200 attendees and nearly two dozen talks on a variety of subjects from a key participants.  The full schedule is available here, and we’ll be doing a wrap up of some of the takeaways from the conference a bit later.

For the uninitiated, Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license.  It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag).  In addition, it is designed to support distribution across multiple physical servers similar to VMware’s vDS or Cisco’s Nexus 1000V. See full feature list here

For more information on OVS, I encourage you to check out the OVS website.

In the mean time, take a read about latest Open vSwitch developments in this post on Network Heresy by OVS core contributors Justin Pettit, Ben Pfaff, and Ethan Jackson.

Accelerating Open vSwitch to “Ludicrous Speed”

Justin

Talking Tech Series: VMware NSX Edge Scale Out with Equal-Cost Multi-Path Routing

This post was written by Roie Ben Haim and Max Ardica, with a special thanks to Jerome Catrouillet, Michael Haines, Tiran Efrat and Ofir Nissim for their valuable input.

****

The modern data center design is changing, following a shift in the habits of consumers using mobile devices, the number of new applications that appear every day and the rate of end-user browsing which has grown exponentially. Planning a new data center requires meeting certain fundamental design guidelines. The principal goals in data center design are: Scalability, Redundancy and High-bandwidth.

In this blog we will describe the Equal Cost Multi-Path functionality (ECMP) introduced in VMware NSX release 6.1 and discuss how it addresses the requirements of scalability, redundancy and high bandwidth. ECMP has the potential to offer substantial increases in bandwidth by load-balancing traffic over multiple paths as well as providing fault tolerance for failed paths. This is a feature which is available on physical networks but we are now introducing this capability for virtual networking as well. ECMP uses a dynamic routing protocol to learn the next-hop towards a final destination and to converge in case of failures. For a great demo of how this works, you can start by watching this video, which walks you through these capabilities in VMware NSX.

Scalability and Redundancy and ECMP

To keep pace with the growing demand for bandwidth, the data center must meet scale out requirements, which provide the capability for a business or technology to accept increased volume without redesign of the overall infrastructure. The ultimate goal is avoiding the “rip and replace” of the existing physical infrastructure in order to keep up with the growing demands of the applications. Data centers running business critical applications need to achieve near 100 percent uptime. In order to achieve this goal, we need the ability to quickly recover from failures affecting the main core components. Recovery from catastrophic events needs to be transparent to end user experiences.

ECMP with VMware NSX 6.1 allows you to use upto a maximum of 8 ECMP Paths simultaneously. In a specific VMware NSX deployment, those scalability and resilience improvements are applied to the “on-ramp/off-ramp” routing function offered by the Edge Services Gateway (ESG) functional component, which allows communication between the logical networks and the external physical infrastructure.

ECMP Image 1

External user’s traffic arriving from the physical core routers can use up to 8 different paths (E1-E8) to reach the virtual servers (Web, App, DB).

In the same way, traffic returning from the virtual server’s hit the Distributed Logical Router (DLR), which can choose up to 8 different paths to get to the core network.

How is the path determined:

NSX for vSphere Edge Services Gateway device:

When a traffic flow needs to be routed, the round robin algorithm is used to pick up one of the links as the path for all traffic of this flow. The algorithm ensures to keep in order all the packets related to this flow by sending them through the same path. Once the next-hop is selected for a particular Source IP and Destination IP pair, the route cache stores this. Once a path has been chosen, all packets related to this flow will follow the same path.

There is a default IPv4 route cache timeout, which is 300 seconds. If an entry is inactive for this period of time, it is then eligible to be removed from route cache. Note that these settings can be tuned for your environment.

Distributed Logical Router (DLR):

The DLR will choose a path based on a Hashing algorithm of Source IP and Destination IP.

What happens in case of a failure on one of Edge Devices?

In order to work with ECMP the requirement is to use a dynamic routing protocol: OSPF or BGP. If we take OSPF for example, the main factor influencing the traffic outage experience is the tuning of the OSPF timers.

OSPF will send hello messages between neighbors, the OSPF “Hello” protocol is used and determines the Interval as to how often an OSPF Hello is sent.

Another OSPF timer called “Dead” Interval is used, which is how long to wait before we consider an OSPF neighbor as “down”. The OSPF Dead Interval is the main factor that influences the convergence time. Dead Interval is usually 4 times the Hello Interval but the OSPF (and BGP) timers can be set as low as 1 second (for Hello interval) and 3 seconds (for Dead interval) to speed up the traffic recovery.

ECMP Image 2

In the example above, the E1 NSX Edge has a failure; the physical routers and DLR detect E1 as Dead at the expiration of the Dead timer and remove their OSPF neighborship with him. As a consequence, the DLR and the physical router remove the routing table entries that originally pointed to the specific next-hop IP address of the failed ESG.

As a result, all corresponding flows on the affected path are re-hashed through the remaining active units. It’s important to emphasize that network traffic that was forwarded across the non-affected paths remains unaffected.

Troubleshooting and visibility

With ECMP it’s important to have introspection and visibility tools in order to troubleshoot optional point of failure. Let’s look at the following topology.

ECMP Image 3

A user outside our Data Center would like to access the Web Server service inside the Data Center. The user IP address is 192.168.100.86 and the web server IP address is 172.16.10.10.

This User traffic will hit the Physical Router (R1), which has established OSPF adjacencies with E1 and E2 (the Edge devices). As a result R1 will learn how to get to the Web server from both E1 and E2 and will get two different active paths towards 172.16.10.10. R1 will pick one of the paths to forward the traffic to reach the Web server and will advertise the user network subnet 192.168.100.0/24 to both E1 and E2 with OSPF.

E1 and E2 are NSX for vSphere Edge devices that also establish OSPF adjacencies with the DLR. E1 and E2 will learn how to get to the Web server via OSPF control plane communication with the DLR.

From the DLR perspective, it acts as a default gateway for the Web server. This DLR will form an OSPF adjacency with E1 and E2 and have 2 different OSPF routes to reach the user network.

From the DLR we can verify OSPF adjacency with E1, E2.

We can use the command: “show ip ospf neighbor”

 

ECMP Image 4

 

From this output we can see that the DLR has two Edge neighbors: 198.168.100.3 and 192.168.100.10.The next step will be to verify that ECMP is actually working.

We can use the command: “show ip route”

 

ECMP Image 5

 

The output from this command shows that the DLR learned the user network 192.168.100.0/24 via two different paths, one via E1 = 192.168.10.1 and the other via E2 = 192.168.10.10.

Now we want to display all the packets which were captured by an NSX for vSphere Edge interface.

In the example below and in order to display the traffic passing through interface vNic_1, and which is not OSPF protocol control packets, we need to type this command: “debug packet display interface vNic_1 not_ip_proto_ospf”

We can see an example with a ping running from host 192.168.100.86 to host 172.16.10.10

 

ECMP Image 6

 

If we would like to display the captured traffic to a specific ip address 172.16.10.10, the command capture would look like: “debug packet display interface vNic_1 dst_172.16.10.10”

 

ECMP Image 7

 

Useful CLI for Debugging ECMP

  • To check which ECMP path is chosen for a flow
    • debug packet display interface IFNAME
  • To check the ECMP configuration
    • show configuration routing-global
  • To check the routing table
    • show ip route
  • To check the forwarding table
    • show ip forwarding

Useful CLI for Dynamic Routing

  • show ip ospf neighbor
  • show ip ospf database
  • show ip ospf interface
  • show ip bgp neighbors
  • show ip bgp

ECMP Deployment Consideration

ECMP currently implies stateless behavior. This means that there is no support for stateful services such as the Firewall, Load Balancing or NAT on the NSX Edge Services Gateway. The Edge Firewall gets automatically disabled on ESG when ECMP is enabled. In the current NSX 6.1 release, the Edge Firewall and ECMP cannot be turned on at the same time on NSX edge device. Note however, that the Distributed Firewall (DFW) is unaffected by this.

For more in-depth information, you can also read our VMware® NSX for vSphere (NSX-V) Network Virtualization Design Guide

About The Authors

VMware-Roie Ben HaimRoie Ben Haim works as a professional services consultant at VMware, focusing on design and implementation of VMware’s software-defined data center products.  Roie has more than 12 years in data center architecture, with a focus on network and security solutions for global enterprises. An enthusiastic M.Sc. graduate, Roie holds a wide range of industry leading certifications including Cisco CCIE x2 # 22755 (Data Center, CCIE Security), Juniper Networks JNCIE – Service Provider #849, and VMware vExpert 2014, VCP-NV, VCP-DCV. Follow his personal blog at http://roie9876.wordpress.com/

VMware-Max ArdicaMax Ardica is a senior technical product manager in VMware’s networking and security business unit (NSBU). Certified as VCDX #171, his primary task is helping to drive the evolution of the VMware NSX platform, building the VMware NSX architecture and providing validated design guidance for the software-defined data center, specifically focusing on network virtualization. Prior to joining VMware, Max worked for almost 15 years at Cisco, covering different roles, from software development to product management. Max owns also a CCIE certification (#13808).

Using Differentiated Services to Tame Elephants

This post was co-authored by Justin Pettit, Staff Engineer, Networking & Security Business Unit at VMware, and Ravi Shekhar, Distinguished Engineer, S3BU at Juniper Networks.

********************

As discussed in other blog posts and presentations, long-lived, high-bandwidth flows (elephants) can negatively affect short-lived flows (mice). Elephant flows send more data, which can lead to queuing delays for latency-sensitive mice.

VMware demonstrated the ability to use a central controller to manage all the forwarding elements in the underlay when elephant flows are detected.  In environments that do not have an SDN-controlled fabric, an alternate approach is needed.  Ideally, the edge can identify elephants in such a way that the fabric can use existing mechanisms to treat mice and elephants differently.

Differentiated services (diffserv) were introduced to bring scalable service discrimination to IP traffic. This is done using Differentiated Services Code Point (DSCP) bits in the IP header to signal different classes of service (CoS). There is wide support in network fabrics to treat traffic differently based on the DSCP value.

A modified version of Open vSwitch allows us to identify elephant flows and mark the DSCP value of the outer IP header.  The fabric is then configured to handle packets with the “elephant” DSCP value differently from the mice.

Figure 1: Elephants are detected at the edge of the network and signaled to the fabric through DSCP.  Based on these code points, the fabric can treat elephant traffic differently from mice

Figure 1: Elephants are detected at the edge of the network and signaled to the fabric through DSCP. Based on these code points, the fabric can treat elephant traffic differently from mice

Detecting and Marking Elephants with Open vSwitch

Open vSwitch’s location at the edge of the network gives it visibility into every packet in and out of each guest.  As such, the vSwitch is in the ideal location to make per-flow decisions such as elephant flow detection. Because environments are different, our approach provides multiple detection mechanisms and actions so that they can be used and evolve independently.

An obvious approach to detection is to just keep track of how many bytes each flow has generated.  By this definition, if a flow has sent a large amount of data, it is an elephant. In Open vSwitch, the number of bytes and an optional duration can be configured. By using a duration, we can ensure that we don’t classify very short-lived flows as elephants. We can also avoid identifying low-bandwidth but long-lived flows as elephants.

An alternate approach looks at the size of the packet that is being given to the NIC.  Most NICs today support TCP Segmentation Offload (TSO), which allows the transmitter (e.g., the guest) to give the NIC TCP segments up to 64KB, which the NIC chops into MSS-sized packets to be placed on the wire.

Because of TCP’s slow start, the transmitter does not immediately begin sending maximum-sized packets to the NIC.  Due to our unique location, we can see the TCP window as it opens, and tag elephants earlier and more definitively. This is not possible at the top-of-rack (TOR) or anywhere else in the fabric, since they only see the segmented version of the traffic.

Open vSwitch may be configured to track all flows with packets of a specified size. For example, by looking for only packets larger than 32KB (which is much larger than jumbo frames), we know the transmitter is out of slow-start and making use of TSO. There is also an optional count, which will trigger when the configured number of packets with the specified size is seen.

Some new networking hardware provides some elephant flow mitigation by giving higher priority to small flows. This is achieved by tracking all flows and placing new flows in a special high-priority queue. When the number of packets in the flow has crossed a threshold, the flow’s packets from then on are placed into the standard priority queue.

This same effect can be achieved using the modified Open vSwitch and a standard fabric.  For example, by choosing a packet size of zero and threshold of ten packets, each flow will be tracked in a hash table in the kernel and tagged with the configured DSCP value when that flow has generated ten packets.  Whether mice are given a high priority or elephants are given a low priority, the same effect is achieved without the need to replace the entire fabric.

Handling Elephants with Juniper Devices

Juniper TOR devices (such as QFX5100) and aggregation devices (such as MX, EX9200) provide a rich diffserv model CoS to to achieve these goals in the underlay.  These include:

  • Elaborate controls for packet admittance with dedicated and shared limits. Dedicated limits provide a minimum service guarantee, and shared limits allow statistical sharing of buffers across different ports and priorities.
  • A large number of flexibly assigned queues; up to 2960 unicast queues at the TOR and 512K at the aggregation device.
  • Enhanced and varied scheduling methods to drain these queues: strict and round-robin scheduling with up to 4-levels of hierarchical schedulers.
  • Shaping and metering to control the rate of injection of traffic from different queues of a TOR in the underlay network. By doing this, bursty traffic at the edge of the physical network can be leveled out before it reaches the more centrally shared aggregation devices.
  • Sophisticated controls to detect and notify congestion, and set drop thresholds. These mechanisms detect possible congestion in the network sooner and notify the source to slow down (e.g. using ECN).

With this level of flexibility, it is possible to configure these devices to:

  • Enforce minimum bandwidth allocation for mice flows and/or maximum bandwidth allocation for elephant flows on a shared link.
  • When experiencing congestion, drop (or ECN mark) packets of elephant flows more aggressively than mice flows.  This will result in TCP connections of elephant flows to back off sooner, which alleviates congestion in the network.
  • Take a different forwarding path for elephant flows from that of mice flows.  For example, a TOR can forward elephant flows towards aggregation switches with big buffers and spread mice flows towards multiple aggregation switches that support low-latency forwarding.

Conclusion

By inserting some intelligence at the edge and using diffserv, network operators can use their existing fabric to differentiate between elephant flows and mice. Most networking gear provides some capabilities, and Juniper, in particular, provides a rich set of operations that can be used based on the DSCP.  Thus, it is possible to reduce the impact of heavy hitters without the need to replace hardware. Decoupling detection from mitigation allows each to evolve independently without requiring wholesale hardware upgrades.

 

VMware NSX Customer Story: Colt Decreases Data Center Networking Complexity

Adoption of network virtualization and SDN technologies from VMware and Arista Networks simplifies cloud infrastructure and enables automation to reduce timescales of cloud and network service provisioning

colt_logo_l_cmyk

Offering the largest enterprise-class cloud footprint in Europe, Colt, an established leader in delivering integrated network, data center, voice and IT services, has implemented software-  defined networking [SDN] and network virtualization to simplify how its managed IT and cloud-based networking environment is deployed, managed and scaled throughout its data centers.

Following an extensive review, Colt selected Arista to provide high speed 10 and 40 gigabit Ethernet cloud-centric switches as an underlay network fabric and VMware NSX™ network virtualization to deliver a fully decoupled software network overlay.

SDN paves the way for automated cloud service delivery

The shift to SDN will provide a flexible, scalable, efficient and cost effective way to support the delivery of Colt’s managed IT services, including cloud based services. This makes Colt one of the first service providers in Europe to adopt SDN in a production environment to remove  automate cloud service delivery.

As a result of deploying a new network architecture based on Arista and VMware networking technologies, the time for Colt to add, change or modify services will now take minutes  rather than days, and will enable Colt to onboard customers faster and expand its service portfolio quicker.

The big transformation in IT in recent years has been the development of cloud services with IT capacity purchased on-demand. In contrast, networking has remained relatively static.  The adoption of server virtualization over the past decade as the foundation for cloud computing and IT-as-a-service have resulted in a completely new operational model for provisioning and managing application workloads. However, the operating model of the network to which dynamic virtualized services are connected has not evolved to help businesses achieve the full benefits of mobile-cloud.

Mirko Voltolini, VP Technology and Architecture at Colt says, “The excitement around SDN and network virtualization is that – for the first time – networking is becoming more software orientated so we’re able to dynamically orchestrate service modification and activation in real-time.  In other words, network connectivity can now keep up when virtual machines and associated compute and storage change or are moved within distributed data centers.  Ultimately this means that servers, storage and now the network are in synch so that we can meet the specific needs of our customers in a timescale they demand.”

With more than 25,000 customers worldwide, Colt offers an information delivery platform comprising network, voice, data center and IT services sold directly to its enterprise customers or indirectly via channel partners and operators. In Europe, it has invested significantly to create a pan European network spanning 22 countries, 195 connected cities, around 19,800 buildings, as well as operating 42 metropolitan area networks.

Turning to the specialist technology firms has really delivered

Colt first considered adopting network overlay technology three years ago. It went out to tender approaching only large, mainstream technology suppliers and was disappointed by the response received.  The cost was too great and solutions not really mature enough to warrant changing. Eighteen months ago, it revisited the process given the technology had evolved, expanding the shortlist of suppliers asked to provide proposals to include specialist firms like Arista and VMware.

VMware NSX enables Colt to decouple the data center network from the underlying physical hardware to gain massive scale while simplifying network design and operations. With NSX, Colt is able to consolidate operations for four disparate physical networks running in the data center and manage these networks as a single logical network. Colt has developed a new data center architecture that leverages the scalability of a Layer 3 data center fabric and NSX’s overlay network virtualization platform.

Chris King, vice president, product marketing, networking and security business unit at VMware, said,  “Colt is an all too common story of an organization that simply hit the limits of what the physical network could provide in a virtualized world. VLAN limitations prevented Colt’s ability to scale. They needed to simplify the physical infrastructure in order to gain flexibility which in turn would allow them to adapt quickly to the business environment. VMware NSX helped Colt successfully execute a data center re-architecture which can now operate at cloud scale with better performance, easier management and lower overall costs.”

In addition to wanting to capitalize on the all the benefits offered by network overlays, the requirement for a new switch supplier was driven by Colt’s need to replace its existing legacy switches which had reached end of life and are not supported anymore.  Furthermore, the business wanted to reduce the total cost of ownership [TCO] of its networking equipment.

Voltolini explains, “Our target was to reduce the unit cost of our switches which includes the cost per port, along with maintenance, power, space and so on.  We wanted a step change in TCO which we will now achieve working with Arista.”

VXLAN addresses the limitations of Spanning Tree

From a technical perspective, Colt also wanted to move away from legacy protocols like Spanning Tree protocol which requires ports to be available – but not used – to deliver service availability. This underutilizes switch assets and adds unnecessary cost to its operation. Moreover, Colt required new switches which could scale to support increased connectivity capabilities both in terms of the number of ports [so that more customers can be connected] as well as logical scale.

Voltolini says, “The new VXLAN protocol removes traditional Ethernet limitations which is crucial for a service provider so that we can handle multiple tenants per port across numerous physical locations.”

Ultimately Arista switches will be installed in all Colt data center locations, the roll out of which will be driven by service and capacity demands.  The expectation is that this will happen over the next 18 to 24 months.  Deployment is made straightforward as all Arista switches – irrespective of port count or speed – feature the same network operating system, the Arista EOS.

Mark Foss, VP Global Operations and Marketing, concludes, “It is important to stress that this project is one of collaboration.  Being an innovative nimble company, we’re accommodating Colt’s requirements and helping shape their service design, while they’re guiding us in terms of our future product roadmap so we develop features pertinent to all cloud service providers.”