One of the core value propositions of VMware NSX is ability to take advantage of any underlying hardware infrastructure and deliver a fully decoupled virtualized network in software. VMware NSX loves a good hardware fabric,.
But that’s not the only hardware VMware NSX loves.
The votes have been cast and counted, and we are pleased to announce that VMware NSX was selected as the winner in the “Best Software Defined Infrastructure” category in the 2015 Modern Infrastructure Impact Awards. The awards were judged by the Modern Infrastructure e-zine editorial staff, in conjunction with users, readers, and industry experts.
The Modern Infrastructure Impact Awards recognize the top products, technologies and services in the essential areas of technology that Modern Infrastructure covers. The award-winning tools are those helping to run enterprise businesses with efficiency and insight — whether they’re used inside the data center or out.
VMware NSX delivers secure network services to applications running in the data center, resulting in instant and programmatic provisioning, fast and highly available infrastructure, and increased security and micro segmentation capabilities.
Over the past 12 months, VMware NSX momentum has continued to grow, as we’ve added new platform capabilities, expanded our partner ecosystem, and of course, had more than 250+ customers purchase NSX for deployment. And as interest in VMware NSX has grown with both customers and IT professionals looking to evolve their careers by adding certification in network virtualization, one of the most common questions that we get is “How can I get started with NSX?.”
We understand that there is a strong demand for individuals and organizations to get their hands on the NSX technology. Many of you are working towards your initial VCP-NV certification. Others of you are exploring NSX as a way to improve your organization’s agility and security while reducing overall costs.
Here are three ways individuals and companies can get started with NSX. Continue reading →
• Application Roll Out Reduced from Weeks to Minutes • VMware NSX Enables Better Agility, Flexibility and Security
Recently I had the opportunity to speak with the team at Schuberg Philis about their successful, production deployment of VMware NSX. As background, Schuberg Philis is an innovative business technology company and an important player in the field of mission critical outsourcing services. The company serves customers across financial services, retail suppliers and utilities, and therefore must comply with the highest international risk management and corporate governance standards, while remaining flexible to evolving customer needs.
The adoption of VMware NSX based network virtualization has transformed the way Schuberg Philis runs its IT. In order to provide 100 percent functional up time of its customers’ critical applications, Schuberg Philis continuously optimizes its infrastructure and processes. However, the company increasingly saw its network as a barrier to increasing business agility.
To solve this challenge and to accelerate application roll out, the Schuberg Philis implemented a software-defined data center environment, and deployed VMware NSX. Schuberg Philis is taking advantage of the VMware NSX platform’s flexibility, security and agility to accelerate the deployment of applications to customers. Schuberg Philis customers now have easy access to the flexibility of the cloud, but within a certified, auditable environment, which includes built in controls and security.
Funs Kessen, cloud architect at Schuberg Philis, explained, “The process for spinning up new applications for customers used to take weeks to complete. Now we can do it in a little more than 18 minutes. This allows our customers to respond more quickly to business requirements and opportunities.
By fully automating the process, Kessen and team can offer Schuberg Philis customers complete access to the flexibility of the cloud within a certified environment, complete with all controls and security built in, and we’ve made it fully auditable.”
The adoption of VMware NSX based network virtualization has transformed the way Schuberg Philis runs its IT.
Kessen noted, “With VMware NSX in our software-defined data center, we can focus on applications, and not on the infrastructure,”
Last week we hosted the Open vSwitch 2014 Fall Conference, which was another great opportunity to demonstrate our continued investment in leading open source technologies. To get a sense of the energy and enthusiasm at the event, take a quick view of this video we captured with attendees.
I’ve been thinking about the key takeaways from everything I saw and everyone I spoke with.
First, there’s huge interest in Open vSwitch performance, both in terms of measurement and improvement. The talks from Rackspace and Noiro Networks/Cisco led me to believe that we’ve reached the point where Open vSwitch performance is good enough on hypervisors for most applications, and often faster than competing software solutions such as the Linux bridge.
Talks from Intel and one from Luigi Rizzo at the University of Pisa demonstrated that by bypassing the kernel entirely through DPDK or netmap, respectively, we haven’t reached the limits of software forwarding performance. Based on a conversation I had with Chris Wright from Red Hat, this work is helping the Linux kernel community look into reducing the overhead of the kernel, so that we can see improved performance without losing the functionality provided by the kernel.
Johann Tönsing from Netronome also presented a talk describing all the ways that Netronome’s NPU hardware can accelerate OpenFlow and Open vSwitch; I’ve talked to Johann many times before, but I had never realized how many different configurations their hardware supports, so this was an eye-opening talk for me.
Next, enhancing Open vSwitch capabilities at L4 through L7 is another exciting area. Our own Justin Pettit was joined by Thomas Graf from Noiro to talk about the ongoing project to add support for NAT and tracking L4 connections, which is key to making Open vSwitch capable of implementing high-quality firewalls. A later talk by Franck Baudin from Qosmos presented L7 enhancements to this capability.
The final area that I saw highlighted at the conference is existing applications for Open vSwitch today. Peter Phaal from InMon, for example, demonstrated applications for sFlow in Open vSwitch. I found his talk interesting because although I knew about sFlow and had talked to Peter before, I hadn’t realized all of the varied uses for sFlow monitoring data. Vikram Dham also showed his uses for MPLS in Open vSwitch and Radhika Hirannaiah her use case for OpenFlow and Open vSwitch in traffic engineering.
I want to thank all of our participants and the organizing committee for helping to put together such an amazing event.
This week, VMware will be hosting the Open vSwitch 2014 Fall Conference, with more than 200 attendees and nearly two dozen talks on a variety of subjects from a key participants. The full schedule is available here, and we’ll be doing a wrap up of some of the takeaways from the conference a bit later.
For the uninitiated, Open vSwitch is a production quality, multilayer virtual switch licensed under the open source Apache 2.0 license. It is designed to enable massive network automation through programmatic extension, while still supporting standard management interfaces and protocols (e.g. NetFlow, sFlow, IPFIX, RSPAN, CLI, LACP, 802.1ag). In addition, it is designed to support distribution across multiple physical servers similar to VMware’s vDS or Cisco’s Nexus 1000V. See full feature list here
For more information on OVS, I encourage you to check out the OVS website.
In the mean time, take a read about latest Open vSwitch developments in this post on Network Heresy by OVS core contributors Justin Pettit, Ben Pfaff, and Ethan Jackson.
We’re excited to take to the road for another edition of our VMware Software-Defined Data Center Seminar Series. Only this time, we’ll be joined by some great company.
VMware & Palo Alto Networks invite you along for a complementary, half-day educational event for IT professionals interested in learning about how Palo Alto Networks and VMware are transforming data center security.
Thousands of IT professionals attended our first SDDC seminar series earlier this year in more than 20 cities around the globe. Visit #VirtualizeYourNetwork.com to browse the presentations, videos, and other content we gathered.
This free seminar will highlight:
The Software-Defined Data Center approach
Lessons learned from real production customers
Using VMware NSX to deliver never before possible data center security and micro-segmentation
Who should attend?
People who will benefit from attending this session include:
IT, Infrastructure and Data Center Managers
Network professionals, including CCIEs
Security & Compliance professionals
Networking Managers and Administrators
Security Managers and Administrators
8:30 a.m. Registration & Breakfast
9:00 a.m. VMware: Better Security with Micro-segmentation
10:00 a.m. Palo Alto Networks: Next Generation Security Services for the SDDC
This post was written by Roie Ben Haim and Max Ardica, with a special thanks to Jerome Catrouillet, Michael Haines, Tiran Efrat and Ofir Nissim for their valuable input.
The modern data center design is changing, following a shift in the habits of consumers using mobile devices, the number of new applications that appear every day and the rate of end-user browsing which has grown exponentially. Planning a new data center requires meeting certain fundamental design guidelines. The principal goals in data center design are: Scalability, Redundancy and High-bandwidth.
In this blog we will describe the Equal Cost Multi-Path functionality (ECMP) introduced in VMware NSX release 6.1 and discuss how it addresses the requirements of scalability, redundancy and high bandwidth. ECMP has the potential to offer substantial increases in bandwidth by load-balancing traffic over multiple paths as well as providing fault tolerance for failed paths. This is a feature which is available on physical networks but we are now introducing this capability for virtual networking as well. ECMP uses a dynamic routing protocol to learn the next-hop towards a final destination and to converge in case of failures. For a great demo of how this works, you can start by watching this video, which walks you through these capabilities in VMware NSX.
Scalability and Redundancy and ECMP
To keep pace with the growing demand for bandwidth, the data center must meet scale out requirements, which provide the capability for a business or technology to accept increased volume without redesign of the overall infrastructure. The ultimate goal is avoiding the “rip and replace” of the existing physical infrastructure in order to keep up with the growing demands of the applications. Data centers running business critical applications need to achieve near 100 percent uptime. In order to achieve this goal, we need the ability to quickly recover from failures affecting the main core components. Recovery from catastrophic events needs to be transparent to end user experiences.
ECMP with VMware NSX 6.1 allows you to use upto a maximum of 8 ECMP Paths simultaneously. In a specific VMware NSX deployment, those scalability and resilience improvements are applied to the “on-ramp/off-ramp” routing function offered by the Edge Services Gateway (ESG) functional component, which allows communication between the logical networks and the external physical infrastructure.
External user’s traffic arriving from the physical core routers can use up to 8 different paths (E1-E8) to reach the virtual servers (Web, App, DB).
In the same way, traffic returning from the virtual server’s hit the Distributed Logical Router (DLR), which can choose up to 8 different paths to get to the core network.
How is the path determined:
NSX for vSphere Edge Services Gateway device:
When a traffic flow needs to be routed, the round robin algorithm is used to pick up one of the links as the path for all traffic of this flow. The algorithm ensures to keep in order all the packets related to this flow by sending them through the same path. Once the next-hop is selected for a particular Source IP and Destination IP pair, the route cache stores this. Once a path has been chosen, all packets related to this flow will follow the same path.
There is a default IPv4 route cache timeout, which is 300 seconds. If an entry is inactive for this period of time, it is then eligible to be removed from route cache. Note that these settings can be tuned for your environment.
Distributed Logical Router (DLR):
The DLR will choose a path based on a Hashing algorithm of Source IP and Destination IP.
What happens in case of a failure on one of Edge Devices?
In order to work with ECMP the requirement is to use a dynamic routing protocol: OSPF or BGP. If we take OSPF for example, the main factor influencing the traffic outage experience is the tuning of the OSPF timers.
OSPF will send hello messages between neighbors, the OSPF “Hello” protocol is used and determines the Interval as to how often an OSPF Hello is sent.
Another OSPF timer called “Dead” Interval is used, which is how long to wait before we consider an OSPF neighbor as “down”. The OSPF Dead Interval is the main factor that influences the convergence time. Dead Interval is usually 4 times the Hello Interval but the OSPF (and BGP) timers can be set as low as 1 second (for Hello interval) and 3 seconds (for Dead interval) to speed up the traffic recovery.
In the example above, the E1 NSX Edge has a failure; the physical routers and DLR detect E1 as Dead at the expiration of the Dead timer and remove their OSPF neighborship with him. As a consequence, the DLR and the physical router remove the routing table entries that originally pointed to the specific next-hop IP address of the failed ESG.
As a result, all corresponding flows on the affected path are re-hashed through the remaining active units. It’s important to emphasize that network traffic that was forwarded across the non-affected paths remains unaffected.
Troubleshooting and visibility
With ECMP it’s important to have introspection and visibility tools in order to troubleshoot optional point of failure. Let’s look at the following topology.
A user outside our Data Center would like to access the Web Server service inside the Data Center. The user IP address is 192.168.100.86 and the web server IP address is 172.16.10.10.
This User traffic will hit the Physical Router (R1), which has established OSPF adjacencies with E1 and E2 (the Edge devices). As a result R1 will learn how to get to the Web server from both E1 and E2 and will get two different active paths towards 172.16.10.10. R1 will pick one of the paths to forward the traffic to reach the Web server and will advertise the user network subnet 192.168.100.0/24 to both E1 and E2 with OSPF.
E1 and E2 are NSX for vSphere Edge devices that also establish OSPF adjacencies with the DLR. E1 and E2 will learn how to get to the Web server via OSPF control plane communication with the DLR.
From the DLR perspective, it acts as a default gateway for the Web server. This DLR will form an OSPF adjacency with E1 and E2 and have 2 different OSPF routes to reach the user network.
From the DLR we can verify OSPF adjacency with E1, E2.
We can use the command: “show ip ospf neighbor”
From this output we can see that the DLR has two Edge neighbors: 22.214.171.124 and 192.168.100.10.The next step will be to verify that ECMP is actually working.
We can use the command: “show ip route”
The output from this command shows that the DLR learned the user network 192.168.100.0/24 via two different paths, one via E1 = 192.168.10.1 and the other via E2 = 192.168.10.10.
Now we want to display all the packets which were captured by an NSX for vSphere Edge interface.
In the example below and in order to display the traffic passing through interface vNic_1, and which is not OSPF protocol control packets, we need to type this command: “debug packet display interface vNic_1 not_ip_proto_ospf”
We can see an example with a ping running from host 192.168.100.86 to host 172.16.10.10
If we would like to display the captured traffic to a specific ip address 172.16.10.10, the command capture would look like: “debug packet display interface vNic_1 dst_172.16.10.10”
Useful CLI for Debugging ECMP
To check which ECMP path is chosen for a flow
debug packet display interface IFNAME
To check the ECMP configuration
show configuration routing-global
To check the routing table
show ip route
To check the forwarding table
show ip forwarding
Useful CLI for Dynamic Routing
show ip ospf neighbor
show ip ospf database
show ip ospf interface
show ip bgp neighbors
show ip bgp
ECMP Deployment Consideration
ECMP currently implies stateless behavior. This means that there is no support for stateful services such as the Firewall, Load Balancing or NAT on the NSX Edge Services Gateway. The Edge Firewall gets automatically disabled on ESG when ECMP is enabled. In the current NSX 6.1 release, the Edge Firewall and ECMP cannot be turned on at the same time on NSX edge device. Note however, that the Distributed Firewall (DFW) is unaffected by this.
Roie Ben Haim works as a professional services consultant at VMware, focusing on design and implementation of VMware’s software-defined data center products. Roie has more than 12 years in data center architecture, with a focus on network and security solutions for global enterprises. An enthusiastic M.Sc. graduate, Roie holds a wide range of industry leading certifications including Cisco CCIE x2 # 22755 (Data Center, CCIE Security), Juniper Networks JNCIE – Service Provider #849, and VMware vExpert 2014, VCP-NV, VCP-DCV. Follow his personal blog at http://roie9876.wordpress.com/
Max Ardica is a senior technical product manager in VMware’s networking and security business unit (NSBU). Certified as VCDX #171, his primary task is helping to drive the evolution of the VMware NSX platform, building the VMware NSX architecture and providing validated design guidance for the software-defined data center, specifically focusing on network virtualization. Prior to joining VMware, Max worked for almost 15 years at Cisco, covering different roles, from software development to product management. Max owns also a CCIE certification (#13808).
In 2013 we introduced VMware NSX Hands-on-Labs for the first time. The NSX 1303 Hands-on-lab has been by far one of the most popular labs, giving you an in-depth view of VMware NSX. Hands-on-labs are one of the best ways to get a good tour of the product. You can take all of these labs online at http://labs.hol.vmware.com/HOL/catalogs/ . It requires a registration, but is open to everyone. .
This year at VMworld we introduced several new NSX labs to give you a deeper look at NSX, and to showcase the depth of integration NSX provides with 3rd party partners and other VMware products. All of the new 2014 Hands-on-labs have been published and are available to you. Here is a quick tour of the labs and what you can expect to see.
If you are just getting started with NSX and want to know what Network Virtualization is all about, we recommend you start here.
This lab will walk you through five modules of exercises:
NSX Logical Switching – building VXLAN logical switches
NSX Logical Routing - Distributed Routing, Dynamic Routing with OSPF
NSX Distributed Firewall – Micro-segmentation with NSX
NSX Edge Services – Load-balancing, SSL VPN
Once you have completed the introductory lab, we recommend taking the advanced lab which is designed to showcase some of the new features in NSX 6.1. You can read and excellent summary of these new capabilities in Chris Wahl’s blog, “NSX 6.1 Announced, Contains Plethora of Enhancements.”
This lab covers the following areas:
Configuring DHCP Relay so that you can use NSX with external IPAM Services
Scaling out Layer 3 routing with Equal Cost Multi-Pathing (ECMP) and Dynamic Routing Protocols. Yes we actually build out the topology below in the lab! That’s the power of network virtualization.
Building out L2VPN services for multi-site and hybrid cloud connectivity services
Integration with 3rd parties using Service Composer and Trend Micro AV & IPS with NSX. You will see how to register services and how NSX is a platform to integrate with 3rd party services in this exercise.
Networking Monitoring with NSX & Riverbed Cascade – we will even show you how you can monitor with NetFlow in this exercise
The two labs above will surely give you a good view of NSX as a network virtualization platform. Next, let’s see how NSX integrates with other VMware products to build out a complete Software-Defined Data Center. This lab shows the integration capabilities offered by NSX with VMware management solutions.
First up, we will learn about Self-service IT, and how you can deliver applications quickly to your end-users with the integration of vCloud Automation Center and NSX. You will build out a multi-machine blueprint with networking and security, and then deploy it.
Next, if you want to learn about automation and the NSX API, we will walk you through exercises in using vCenter Orchestrator and using the NSX REST API to create a security group. This will give you the fundamentals of NSX automation which you can easily extend upon as you deploy NSX in your own environment.
The third exercise is about operations. We will show you the new NSX Management Pack in vCenter Operations. We will walk you through the dashboards and you will learn how you can actually not just monitor but also troubleshoot you network.
At this point you are surely on your way to become a NSX Ninja
If you want to use OpenStack with NSX and vSphere – we’ve got you covered too! We will walk you through OpenStack on vSphere itself and then show you how to connect it to deploy networks with NSX from OpenStack.
And Of Course There Are More
Those are the main labs I would recommend, but there are others too. There’s a lab where you can learn more about the IT Outcome of Fast Infrastructure Delivery and Application Automation (HOL-SDC-1413) which has some NSX goodness with vCloud Automation Center, or learn about the IT Outcome of Policy-based Compliance and Network Security (HOL-SDC-1414).
If you want to learn about NSX and the partner integration framework you can take HOL-PRT-1462 which will walk you through the NSX and Palo Alto Networks next-gen firewall integration labs and HOL-PRT-1464 which is focused on how you can use NSX Service Composer and Symantec Data Center Security: Server.
In all we have well over 24 hours of labs, and you can sign-up even if you did not go to VMworld. It is always available 24/7, so if you have a few spare hours and want to learn about NSX you can take the lab.
And I will let you in on a little secret. We actually run the labs on NSX. So as you learn, you are also a user of NSX!!!
This post was written by VMware’s John Dias, (VCP-DCV), Sr. Systems Engineer, Cloud Management Solutions Engineering Team, and Hadar Freehling, Security & Compliance Systems Engineer Specialist
Through a joint effort with Hadar Freehling, one of my esteemed peers here at VMware, we co-developed a proof-of-concept workflow for a network security use case. Hadar created a short video showing and explaining the use case, but in summary this is a workflow that reacts to and remediates a security issue flagged by third-party integration with VMware NSX. In the video, TrendMicro is used but it could be any other partner integration with vShield Endpoint.
Here’s what happens:
A virus is detected on a VM and is quarantined by the AV solution
The AV solution tags the VM with an NSX security tag
VMware NSX places the VM in a new Security Group, whose network policies steer all VM traffic through an intrusion prevention system (IPS)
vCenter Orchestrator (vCO) monitors the security group for changes and when a VM is added
a snapshot of the VM is taken for forensic purposes
a vSpan session (RSPAN) is set up on the Distributed Virtual Switch to begin capturing inbound/outbound traffic on the VM
once the VM has been removed from the security group, the vSpan session is removed
Watch the video below for a walk-through by Hadar:
You will note that there is a portion of the workflow that is handled natively by VMware NSX (Security Tag reaction, Security Group policy) but the snapshot and RSPAN are done via vCO workflow.
If you are interested in exploring this capability, I have provided the vCO workflow package for download. This is provided as-is and you should fully test it (and modify as needed) before using in your environment.
Assuming you have VMware NSX, vShield Endpoint and some third party integration already set up, you will need the following:
The NSX plugin for vCO (installed and configured)
The REST plugin with your NSX manager added as a REST host
vCenter plugin configured
The workflow package includes a good number of “helper” workflows which you will not need to run directly. The master workflow is in the root folder Security Reaction and is named “Set up VM Forensics RUN THIS” (just in case you had any doubt as to which one to run).
The Security Reaction Master Workflow
Running the master workflow will prompt you for three items:
The NSX Security Group to monitor – This is why the NSX plugin is required, so that you can browse the vCO managed objects and locate the desired Security Group.
A time to sleep in seconds – The master workflow will run continuously until manually stopped and will use a REST call to NSX to get the current membership for the Security Group. We have no recommendation on this poll time, although in testing we used 5-10 seconds. It would have been better to use some external event to kick off the vCO workflow but we could not find a way to do this from NSX. It may be possible to do via the partner solution, but we wanted this workflow package to be “partner neutral.”
Destination IPv4 address – This is the destination for the RSPAN (or vSpan session in vSphere API terms). The vSpan session is created with some defaults (for example sampling rate, normal traffic allowed, etc). If you want to change any of those properties, you will need to modify the Helper workflow named “Configure encapRemoteMirrorSource vSpan Session on DVS” (modify the “Create Port Mirror” script task).
Also note that this workflow doesn’t support VMs with multiple vNICs. Specifically, it will only create an RSPAN that includes the first vNIC found on a VM. You can modify the Helper workflow “Implement Forensics” and adjust the script task “Prep for Mirror Creation” so that the additional NICs (if any) are added to the sourcePorts array. It’s something we intended to fix but forgot about until after our final testing and video production – so as they say in the textbooks “this is left as an exercise for the reader.”
Of course, there are many other actions that can be taken besides setting up an RSPAN and getting a snapshot. This solution can be extended to practically any task required during such an event such as creating a ticket in your service desk software, spinning up additional workloads to replace the compromised VM, sending emails, guest OS file system operations…all of these and more can be accomplished using vCO in conjunction with NSX.
This post was co-authored by Justin Pettit, Staff Engineer, Networking & Security Business Unit at VMware, and Ravi Shekhar, Distinguished Engineer, S3BU at Juniper Networks.
As discussed in other blog posts and presentations, long-lived, high-bandwidth flows (elephants) can negatively affect short-lived flows (mice). Elephant flows send more data, which can lead to queuing delays for latency-sensitive mice.
VMware demonstrated the ability to use a central controller to manage all the forwarding elements in the underlay when elephant flows are detected. In environments that do not have an SDN-controlled fabric, an alternate approach is needed. Ideally, the edge can identify elephants in such a way that the fabric can use existing mechanisms to treat mice and elephants differently.
Differentiated services (diffserv) were introduced to bring scalable service discrimination to IP traffic. This is done using Differentiated Services Code Point (DSCP) bits in the IP header to signal different classes of service (CoS). There is wide support in network fabrics to treat traffic differently based on the DSCP value.
A modified version of Open vSwitch allows us to identify elephant flows and mark the DSCP value of the outer IP header. The fabric is then configured to handle packets with the “elephant” DSCP value differently from the mice.
Figure 1: Elephants are detected at the edge of the network and signaled to the fabric through DSCP. Based on these code points, the fabric can treat elephant traffic differently from mice
Detecting and Marking Elephants with Open vSwitch
Open vSwitch’s location at the edge of the network gives it visibility into every packet in and out of each guest. As such, the vSwitch is in the ideal location to make per-flow decisions such as elephant flow detection. Because environments are different, our approach provides multiple detection mechanisms and actions so that they can be used and evolve independently.
An obvious approach to detection is to just keep track of how many bytes each flow has generated. By this definition, if a flow has sent a large amount of data, it is an elephant. In Open vSwitch, the number of bytes and an optional duration can be configured. By using a duration, we can ensure that we don’t classify very short-lived flows as elephants. We can also avoid identifying low-bandwidth but long-lived flows as elephants.
An alternate approach looks at the size of the packet that is being given to the NIC. Most NICs today support TCP Segmentation Offload (TSO), which allows the transmitter (e.g., the guest) to give the NIC TCP segments up to 64KB, which the NIC chops into MSS-sized packets to be placed on the wire.
Because of TCP’s slow start, the transmitter does not immediately begin sending maximum-sized packets to the NIC. Due to our unique location, we can see the TCP window as it opens, and tag elephants earlier and more definitively. This is not possible at the top-of-rack (TOR) or anywhere else in the fabric, since they only see the segmented version of the traffic.
Open vSwitch may be configured to track all flows with packets of a specified size. For example, by looking for only packets larger than 32KB (which is much larger than jumbo frames), we know the transmitter is out of slow-start and making use of TSO. There is also an optional count, which will trigger when the configured number of packets with the specified size is seen.
Some new networking hardware provides some elephant flow mitigation by giving higher priority to small flows. This is achieved by tracking all flows and placing new flows in a special high-priority queue. When the number of packets in the flow has crossed a threshold, the flow’s packets from then on are placed into the standard priority queue.
This same effect can be achieved using the modified Open vSwitch and a standard fabric. For example, by choosing a packet size of zero and threshold of ten packets, each flow will be tracked in a hash table in the kernel and tagged with the configured DSCP value when that flow has generated ten packets. Whether mice are given a high priority or elephants are given a low priority, the same effect is achieved without the need to replace the entire fabric.
Handling Elephants with Juniper Devices
Juniper TOR devices (such as QFX5100) and aggregation devices (such as MX, EX9200) provide a rich diffserv model CoS to to achieve these goals in the underlay. These include:
Elaborate controls for packet admittance with dedicated and shared limits. Dedicated limits provide a minimum service guarantee, and shared limits allow statistical sharing of buffers across different ports and priorities.
A large number of flexibly assigned queues; up to 2960 unicast queues at the TOR and 512K at the aggregation device.
Enhanced and varied scheduling methods to drain these queues: strict and round-robin scheduling with up to 4-levels of hierarchical schedulers.
Shaping and metering to control the rate of injection of traffic from different queues of a TOR in the underlay network. By doing this, bursty traffic at the edge of the physical network can be leveled out before it reaches the more centrally shared aggregation devices.
Sophisticated controls to detect and notify congestion, and set drop thresholds. These mechanisms detect possible congestion in the network sooner and notify the source to slow down (e.g. using ECN).
With this level of flexibility, it is possible to configure these devices to:
Enforce minimum bandwidth allocation for mice flows and/or maximum bandwidth allocation for elephant flows on a shared link.
When experiencing congestion, drop (or ECN mark) packets of elephant flows more aggressively than mice flows. This will result in TCP connections of elephant flows to back off sooner, which alleviates congestion in the network.
Take a different forwarding path for elephant flows from that of mice flows. For example, a TOR can forward elephant flows towards aggregation switches with big buffers and spread mice flows towards multiple aggregation switches that support low-latency forwarding.
By inserting some intelligence at the edge and using diffserv, network operators can use their existing fabric to differentiate between elephant flows and mice. Most networking gear provides some capabilities, and Juniper, in particular, provides a rich set of operations that can be used based on the DSCP. Thus, it is possible to reduce the impact of heavy hitters without the need to replace hardware. Decoupling detection from mitigation allows each to evolve independently without requiring wholesale hardware upgrades.