Home > Blogs > The Network Virtualization Blog > Category Archives: NSX

Category Archives: NSX

Using Differentiated Services to Tame Elephants

This post was co-authored by Justin Pettit, Staff Engineer, Networking & Security Business Unit at VMware, and Ravi Shekhar, Distinguished Engineer, S3BU at Juniper Networks.

********************

As discussed in other blog posts and presentations, long-lived, high-bandwidth flows (elephants) can negatively affect short-lived flows (mice). Elephant flows send more data, which can lead to queuing delays for latency-sensitive mice.

VMware demonstrated the ability to use a central controller to manage all the forwarding elements in the underlay when elephant flows are detected.  In environments that do not have an SDN-controlled fabric, an alternate approach is needed.  Ideally, the edge can identify elephants in such a way that the fabric can use existing mechanisms to treat mice and elephants differently.

Differentiated services (diffserv) were introduced to bring scalable service discrimination to IP traffic. This is done using Differentiated Services Code Point (DSCP) bits in the IP header to signal different classes of service (CoS). There is wide support in network fabrics to treat traffic differently based on the DSCP value.

A modified version of Open vSwitch allows us to identify elephant flows and mark the DSCP value of the outer IP header.  The fabric is then configured to handle packets with the “elephant” DSCP value differently from the mice.

Figure 1: Elephants are detected at the edge of the network and signaled to the fabric through DSCP.  Based on these code points, the fabric can treat elephant traffic differently from mice

Figure 1: Elephants are detected at the edge of the network and signaled to the fabric through DSCP. Based on these code points, the fabric can treat elephant traffic differently from mice

Detecting and Marking Elephants with Open vSwitch

Open vSwitch’s location at the edge of the network gives it visibility into every packet in and out of each guest.  As such, the vSwitch is in the ideal location to make per-flow decisions such as elephant flow detection. Because environments are different, our approach provides multiple detection mechanisms and actions so that they can be used and evolve independently.

An obvious approach to detection is to just keep track of how many bytes each flow has generated.  By this definition, if a flow has sent a large amount of data, it is an elephant. In Open vSwitch, the number of bytes and an optional duration can be configured. By using a duration, we can ensure that we don’t classify very short-lived flows as elephants. We can also avoid identifying low-bandwidth but long-lived flows as elephants.

An alternate approach looks at the size of the packet that is being given to the NIC.  Most NICs today support TCP Segmentation Offload (TSO), which allows the transmitter (e.g., the guest) to give the NIC TCP segments up to 64KB, which the NIC chops into MSS-sized packets to be placed on the wire.

Because of TCP’s slow start, the transmitter does not immediately begin sending maximum-sized packets to the NIC.  Due to our unique location, we can see the TCP window as it opens, and tag elephants earlier and more definitively. This is not possible at the top-of-rack (TOR) or anywhere else in the fabric, since they only see the segmented version of the traffic.

Open vSwitch may be configured to track all flows with packets of a specified size. For example, by looking for only packets larger than 32KB (which is much larger than jumbo frames), we know the transmitter is out of slow-start and making use of TSO. There is also an optional count, which will trigger when the configured number of packets with the specified size is seen.

Some new networking hardware provides some elephant flow mitigation by giving higher priority to small flows. This is achieved by tracking all flows and placing new flows in a special high-priority queue. When the number of packets in the flow has crossed a threshold, the flow’s packets from then on are placed into the standard priority queue.

This same effect can be achieved using the modified Open vSwitch and a standard fabric.  For example, by choosing a packet size of zero and threshold of ten packets, each flow will be tracked in a hash table in the kernel and tagged with the configured DSCP value when that flow has generated ten packets.  Whether mice are given a high priority or elephants are given a low priority, the same effect is achieved without the need to replace the entire fabric.

Handling Elephants with Juniper Devices

Juniper TOR devices (such as QFX5100) and aggregation devices (such as MX, EX9200) provide a rich diffserv model CoS to to achieve these goals in the underlay.  These include:

  • Elaborate controls for packet admittance with dedicated and shared limits. Dedicated limits provide a minimum service guarantee, and shared limits allow statistical sharing of buffers across different ports and priorities.
  • A large number of flexibly assigned queues; up to 2960 unicast queues at the TOR and 512K at the aggregation device.
  • Enhanced and varied scheduling methods to drain these queues: strict and round-robin scheduling with up to 4-levels of hierarchical schedulers.
  • Shaping and metering to control the rate of injection of traffic from different queues of a TOR in the underlay network. By doing this, bursty traffic at the edge of the physical network can be leveled out before it reaches the more centrally shared aggregation devices.
  • Sophisticated controls to detect and notify congestion, and set drop thresholds. These mechanisms detect possible congestion in the network sooner and notify the source to slow down (e.g. using ECN).

With this level of flexibility, it is possible to configure these devices to:

  • Enforce minimum bandwidth allocation for mice flows and/or maximum bandwidth allocation for elephant flows on a shared link.
  • When experiencing congestion, drop (or ECN mark) packets of elephant flows more aggressively than mice flows.  This will result in TCP connections of elephant flows to back off sooner, which alleviates congestion in the network.
  • Take a different forwarding path for elephant flows from that of mice flows.  For example, a TOR can forward elephant flows towards aggregation switches with big buffers and spread mice flows towards multiple aggregation switches that support low-latency forwarding.

Conclusion

By inserting some intelligence at the edge and using diffserv, network operators can use their existing fabric to differentiate between elephant flows and mice. Most networking gear provides some capabilities, and Juniper, in particular, provides a rich set of operations that can be used based on the DSCP.  Thus, it is possible to reduce the impact of heavy hitters without the need to replace hardware. Decoupling detection from mitigation allows each to evolve independently without requiring wholesale hardware upgrades.

 

VMware NSX Customer Story: Colt Decreases Data Center Networking Complexity

Adoption of network virtualization and SDN technologies from VMware and Arista Networks simplifies cloud infrastructure and enables automation to reduce timescales of cloud and network service provisioning

colt_logo_l_cmyk

Offering the largest enterprise-class cloud footprint in Europe, Colt, an established leader in delivering integrated network, data center, voice and IT services, has implemented software-  defined networking [SDN] and network virtualization to simplify how its managed IT and cloud-based networking environment is deployed, managed and scaled throughout its data centers.

Following an extensive review, Colt selected Arista to provide high speed 10 and 40 gigabit Ethernet cloud-centric switches as an underlay network fabric and VMware NSX™ network virtualization to deliver a fully decoupled software network overlay.

SDN paves the way for automated cloud service delivery

The shift to SDN will provide a flexible, scalable, efficient and cost effective way to support the delivery of Colt’s managed IT services, including cloud based services. This makes Colt one of the first service providers in Europe to adopt SDN in a production environment to remove  automate cloud service delivery.

As a result of deploying a new network architecture based on Arista and VMware networking technologies, the time for Colt to add, change or modify services will now take minutes  rather than days, and will enable Colt to onboard customers faster and expand its service portfolio quicker.

The big transformation in IT in recent years has been the development of cloud services with IT capacity purchased on-demand. In contrast, networking has remained relatively static.  The adoption of server virtualization over the past decade as the foundation for cloud computing and IT-as-a-service have resulted in a completely new operational model for provisioning and managing application workloads. However, the operating model of the network to which dynamic virtualized services are connected has not evolved to help businesses achieve the full benefits of mobile-cloud.

Mirko Voltolini, VP Technology and Architecture at Colt says, “The excitement around SDN and network virtualization is that – for the first time – networking is becoming more software orientated so we’re able to dynamically orchestrate service modification and activation in real-time.  In other words, network connectivity can now keep up when virtual machines and associated compute and storage change or are moved within distributed data centers.  Ultimately this means that servers, storage and now the network are in synch so that we can meet the specific needs of our customers in a timescale they demand.”

With more than 25,000 customers worldwide, Colt offers an information delivery platform comprising network, voice, data center and IT services sold directly to its enterprise customers or indirectly via channel partners and operators. In Europe, it has invested significantly to create a pan European network spanning 22 countries, 195 connected cities, around 19,800 buildings, as well as operating 42 metropolitan area networks.

Turning to the specialist technology firms has really delivered

Colt first considered adopting network overlay technology three years ago. It went out to tender approaching only large, mainstream technology suppliers and was disappointed by the response received.  The cost was too great and solutions not really mature enough to warrant changing. Eighteen months ago, it revisited the process given the technology had evolved, expanding the shortlist of suppliers asked to provide proposals to include specialist firms like Arista and VMware.

VMware NSX enables Colt to decouple the data center network from the underlying physical hardware to gain massive scale while simplifying network design and operations. With NSX, Colt is able to consolidate operations for four disparate physical networks running in the data center and manage these networks as a single logical network. Colt has developed a new data center architecture that leverages the scalability of a Layer 3 data center fabric and NSX’s overlay network virtualization platform.

Chris King, vice president, product marketing, networking and security business unit at VMware, said,  “Colt is an all too common story of an organization that simply hit the limits of what the physical network could provide in a virtualized world. VLAN limitations prevented Colt’s ability to scale. They needed to simplify the physical infrastructure in order to gain flexibility which in turn would allow them to adapt quickly to the business environment. VMware NSX helped Colt successfully execute a data center re-architecture which can now operate at cloud scale with better performance, easier management and lower overall costs.”

In addition to wanting to capitalize on the all the benefits offered by network overlays, the requirement for a new switch supplier was driven by Colt’s need to replace its existing legacy switches which had reached end of life and are not supported anymore.  Furthermore, the business wanted to reduce the total cost of ownership [TCO] of its networking equipment.

Voltolini explains, “Our target was to reduce the unit cost of our switches which includes the cost per port, along with maintenance, power, space and so on.  We wanted a step change in TCO which we will now achieve working with Arista.”

VXLAN addresses the limitations of Spanning Tree

From a technical perspective, Colt also wanted to move away from legacy protocols like Spanning Tree protocol which requires ports to be available – but not used – to deliver service availability. This underutilizes switch assets and adds unnecessary cost to its operation. Moreover, Colt required new switches which could scale to support increased connectivity capabilities both in terms of the number of ports [so that more customers can be connected] as well as logical scale.

Voltolini says, “The new VXLAN protocol removes traditional Ethernet limitations which is crucial for a service provider so that we can handle multiple tenants per port across numerous physical locations.”

Ultimately Arista switches will be installed in all Colt data center locations, the roll out of which will be driven by service and capacity demands.  The expectation is that this will happen over the next 18 to 24 months.  Deployment is made straightforward as all Arista switches – irrespective of port count or speed – feature the same network operating system, the Arista EOS.

Mark Foss, VP Global Operations and Marketing, concludes, “It is important to stress that this project is one of collaboration.  Being an innovative nimble company, we’re accommodating Colt’s requirements and helping shape their service design, while they’re guiding us in terms of our future product roadmap so we develop features pertinent to all cloud service providers.”

VMware NSX Use Case – Simplifying Disaster Recovery (Part 2)

Nicolas Vermandé (VCDX#055) is practice lead for Private Cloud & Infrastructure  at Kelway, a VMware partner. Nicolas covers the Software-Defined Data Center on his blog www.my-sddc.om,

This is Part 2 in a series of posts the describes a specific use case for VMware NSX in the context of Disaster Recovery. Here’s part 1,

++++++++++++++++++++++++++++++++++

Deploying the environment

Now let’s see have a closer look at how to create this environment. The following picture represents the vSphere logical architecture and the associated IP scheme…

ipNSX

 

… and the networks mapping:

logicalnetNSX

First of all you have to create three vSphere clusters: one Management Cluster and two Compute Clusters, as well as  two distinct VDS, within the same vCenter. Each Compute cluster will be connected to the same VDS. One cluster will represent DC1, and the other one will represent DC2. The second VDS will connect to the Management and vMotion networks. Also, you have to create a couple of VLANs: one VLAN for VTEPs, used as the outer dot1q tag to transport VXLAN frames, two external transit VLANs to allow the ESGs to peer with your IP core and VLANs for traditional vSphere functions, such as Management, vMotion and IP storage if required.

Note: As this lab has been created for educational purpose, it is clearly not aligned with NSX design considerations for a production environment. I’ll probably dedicate another blog post for that.

Now let’s get our hands dirty. I have to assume that you already have the NSX Manager deployed as well as 3 controllers. All those virtual appliances should be placed in the Management Cluster and connected to the Management VDS. For the sake of simplicity you can use the same Management VLAN for both ESXi and NSX components management.

The first step after having deployed the controllers is to install the ESXi vibs:  go to the NSX vCenter Plugin, then under Installation, select the Host Preparation tab. Select your Compute Clusters and click Install.

01

Once done, click Configure under the VXLAN section to configure the VXLAN networking:

02

The VLAN field is the outer VLAN ID for your VXLAN overlay. Create a new IP pool named VTEP and use it at the reference pool for your VTEP configuration. Note that if you select “Load Balance – SRCID” or “Load Balance - SRCMAC” as the teaming policy, two VTEPs will be created within the same IP Pool. It means that if you want your VTEPs to reside in two different subnets, you have to use a DHCP server. Another thing I noticed: be sure to create the appropriate number of VDS uplinks BEFORE preparing the hosts, or the NSX manager may not see the right number of uplinks when you want to deploy multiple VTEPs.

03

Next step is to configure the Segment ID range, which will represent your pool of available VNIs. As we will be using unicast transport mode, we don’t need to configure any Multicast Group.

04

Then you can go under Logical Network Preparation > Transport Zones. Add two Transport Zones, as we’ll be simulating two distinct datacenters. Select Unicast as the Control Plane Mode.

05

Each simulated datacenter should end up with its own transport zone, as shown below:

27

Now it’s time to create the Logical Switches. In the Network & Security pane, go to Logical Switches. In the right pane click the “+” icon. Give it a name, and select the first Transport Zone.

07

Create a second Logical Switch, linked to the second Transport Zone. As both Logical Switches are in two different Transport Zones, they will be completely isolated, without any possibility to connect them to the same Logical Router.

08

For the sake of completeness and to match the initial design, you can create a second Logical Switch in each datacenter. Additional Logical Switches to be created create are those connecting Logical Routers to the upstream Edge Gateway. Name those Logical Switches dc1_transit and dc2_transit.

09

The next components to be deployed are the Logical Routers. In the Networking & Security Pane, go to NSX Edges. In the right pane, click the “+” icon. Select Logical Router, you can enable HA if you wish so (I know it’s kind of weird to configure the DLR under the NSX Edge menu…).

10

Configure the credentials and at the “Configure deployment” step, click the “+” icon under NSX Edge Appliances. Select you first datacenter cluster and the appropriate Datastore, Host and Folder.

11

Then configure the management interface by clicking Select, next to “Connected To”. You should select aDistributed Portgroup, not a Logical Switch.

Next, go under Configure interfaces of this NSX Edge and click the “+” icon. Give the interface a name and selectinternal as the interface type. Connect the interface to the first Logical Switch in DC1 and configure its IP address and subnet mask. Repeat the steps to connect a second internal interface to dc1_ls02.

14

As you can imagine, the Uplink interface type will be used to connect the Logical Router interface to thedc1_transit Logical Switch. Add this interface and configure its IP address and subnet mask. It is worth noting that in the case of an internal LIF, the IP address given must be the default gateway for the VMs belonging to that particular Logical Switch.

Here is a screenshot of what you should have as the final configuration:

15

You can then click Next, Next, Finish. Repeat the same operations to create a second Logical Router, but this time in the second datacenter. The cluster/resource pool parameter must be set do dc2 so you can have access to the NSX components available in that specific Transport Zone. Here is a screenshot of what you should have in the end:

16

The last components to be deployed are the NSX Edge Gateways, which connect to the Logical Routers Uplink LIF through the transit Logical Switch. The Edge Gateways must have both a VXLAN interface (the internal interface connecting to the Logical Router) and a VLAN interface, connecting to the external, physical network.

To deploy an Edge Gateway, go to NSX Edges and click on the “+” icon under NSX Edge Appliances. Select Edge Services Gateway as the install type, enable HA and give a name to the gateway.

17

Click Next and configure the credentials. Then click Next and select the appliance size. Compact is ok for a lab, bigger appliances support a higher number of adjacencies, firewall rules, etc.

18

Then click the “+” icon under NSX Edge Appliances and select dc1 as the Cluster/Resource Pool and the appropriate Datastore, Host and Folder.

Click Next, then the “+” icon to add an interface. Give the interface a name and connect it to thedc1_transit network as an internal interface. Configure the IP address and the subnet mask, click OK and repeat the procedure to create an Uplink interface connected to a VDS Portgroup that represents the external network (It can be a VDS or VSS Portgroup).

21

The end result should look like this:

22

Click Next and configure a default gateway if you wish so. However, it’s not strictly necessary in our scenario. You can then click Next, Next and Finish to deploy the Edge Gateway in the first datacenter. Repeat the deployment procedure for the second datacenter by selecting dc2 as the cluster/resource pool so you can connect the appliance to the NSX components available in the second Transport Zone.

Before activating dynamic routing protocols within the NSX environment, we must configure an external device to enable routing adjacency with Edge Gateways in both simulated datacenters. You can use a physical device but if you want to deploy this in your home lab or if you don’t have access to a physical device, I recommend using a Vyatta virtual appliance. It has decent routing capabilities and OSPF configuration is pretty straight forward. I’m using VC6.6R1 in my lab.

Your external routing device should have two interfaces: one connecting to the DC1 Edge Gateway external interface network and one connecting to the DC2 Edge Gateway external interface network. Refer to the global topology diagram for IP addresses and subnets. Here is a screenshot of my Vyatta hardware configuration (I’ve added a third VNIC to connect to the management network so I can SSH into the appliance):

28

Now let’s see how to activate OSPF on the Logical Router and the Edge Gateway:

Under Network & Security, go to NSX Edges and select the Logical Router for DC1. Double-click on it. Go toManage > Global Configuration and click Edit next to Dynamic Routing Configuration. Set a custom Router IDand click save (Don’t tick the OSPF box)

24

Then go to OSPF and click Edit next to OSPF Configuration. You have to set two IP addresses: the Protocol Address is used to establish adjacency with neighbors. The Forwarding Address is the actual address used by the ESXi kernel module in the data plane. They must be part of the same subnet and the Forwarding Address must be the IP address of the Logical Router Uplink interface you have previously configured.

23

Click on the “+” icon under Area Definitions and add Area 0.

25

Then go to Area to Interface Mapping and add the transit vNIC to Area 0. Don’t add the Logical Switch internal LIFs to Area 0 as they’re not participating in the OSPF process. Instead the Logical Switch routing information is redistributed into OSPF (redistribute connected routes). Don’t forget to Publish Changes.

26

Repeat the same procedure for the second Logical Router in DC2.

To activate OSPF within the Edge Gateways, configure the Router ID and tick the OSPF box. There is no need to split the control plane from the data plane as the Edge Gateway is a virtual appliance and as such, it doesn’t have any kernel module installed on the ESXi host. Another difference is that you have to add both the transit and the external network interfaces into Area 0.

Note that if you want to ping the DLR and the ESG from you external network, you’ll have to modify appropriate firewalls rules as both components may have a default deny rule on their local firewall.

If you have configured everything correctly, you should see OSPF information about all routes on the external routing device:

vyatta@vyatta:~$ sh ip route ospf
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       I - ISIS, B - BGP, > - selected route, * - FIB route

O>* 192.168.0.0/24 [110/1] via 192.168.6.2, eth0, 00:06:08
O>* 192.168.1.0/24 [110/1] via 192.168.6.2, eth0, 00:06:08
O   192.168.6.0/30 [110/10] is directly connected, eth0, 00:22:31
O>* 192.168.7.0/29 [110/11] via 192.168.6.2, eth0, 00:11:38
O>* 192.168.10.0/24 [110/1] via 192.168.14.2, eth1, 00:00:21
O>* 192.168.11.0/24 [110/1] via 192.168.14.2, eth1, 00:00:21
O   192.168.14.0/30 [110/10] is directly connected, eth1, 00:22:31
O>* 192.168.15.0/29 [110/11] via 192.168.14.2, eth1, 00:00:38

By default, the Vyatta appliance adds a cost of 10 to its ospf interfaces and the ESG adds a cost of 1. This is customizable, and so are OSPF Hello and Dead intervals.

Hopefully you’ve got everything working now! :-P

The next post will focus on the very cool part, which is how to use python and pyVmomi to perform both NSX and vSphere tasks to move things around.

A Customer Perspective: VMware NSX, Micro-Segmentation & Next-Generation Security

VMware NSX and Palo Alto Networks are transforming the data center by combining the Columbia-S12_WTR_MGHI_564fast provisioning of network and security services with next-generation security protection for East-West traffic. At VMworld, John Spiegel, Global IS Communications Manager for Columbia Sportswear will take the stage to discuss their architecture, their micro-segmentation use case and their experience. This is session SEC1977 taking place on Tuesday, Aug 26, 2:30-3:30 p.m.

Micro-segmentation is quickly emerging as one of the primary drivers for the adoption of NSX. Below, John shares Columbia’s security journey ahead of VMworld

+++++++++++++++++++++++++++++++++++++++

When I started at Columbia, we were about a $500 million company. Now we’re closing in on $2 billion and hoping to get to $3 billion rather quickly. So as you can imagine, our IT infrastructure has to scale with the business. In 2009, we embarked on a huge project to add a redundant data center for disaster recovery. As part of the project, we partnered with VMware and quickly created a nearly 100% virtualized datacenter.  It was a huge success. But something was missing; a security solution that matched our virtualized data center. There just wasn’t a great way to insert security in order to address east-west traffic between VMs, nor have the security tied to the applications as they moved around dynamically.

 We set out looking for a solution to bridge that gap.

To address our security needs in the data center, we looked at several different strategies and at that time, there really weren’t any good solutions. Many of the solutions were physical in nature. They required us to do some crazy configurations to apply security. We looked at the Cisco 6500 firewall blades, Juniper’s virtual solution and a few other lightweight security offerings, but they just didn’t have what we needed. The solutions at the time didn’t have what we needed. We kept looking.

At VMworld last year, we were introduced to VMware NSX. I saw the power of the platform, and it all started to click. And when Palo Alto Networks (our perimeter firewall vendor) announced they were a major partner, and that their technology integrated with NSX to give us an additional level of security, things really came together for us. The ability to drive security down into the infrastructure, down to the kernel level, and then take advantage of Palo Alto Networks next generation security was very attractive. Doing micro-segmentation with NSX, and then having the option of inserting next generation firewalling services from Palo Alto Networks in those areas of the business that require them, will really help us improve our overall security posture. A solution like this is where we need to be. These tools give us the ability to manage both physical and virtual security policies centrally with Palo Alto Networks management tool Panorama. I know that when workloads move the security and policies follow the workloads.

To me, that’s what it is about – advanced security inside the data center, plus automation via software that’s completely independent of the underlying physical infrastructure. With solutions such as NSX and the integration with Palo Alto Networks to provide advanced security services, we are going put security back in the data center, the right way.=

Jspiegel

John Spiegel
Columbia Sportswear

 

VMware NSX Use Case – Simplifying Disaster Recovery (Part 1)

Nicolas Vermandé (VCDX#055) is practice lead for Private Cloud & Infrastructure  at Kelway, a VMware partner. Nicolas covers the Software-Defined Data Center on his blog www.my-sddc.om,

This series of posts describes a specific use case for VMware NSX in the context of Disaster Recovery. The goal is to demonstrate the routing and programmability capabilities through a lab scenario. This first part presents the NSX components and details the use case. The second part will show how to deploy the lab and the third part will deal with APIs and show how to use python to execute REST API calls to recreate the required NSX components at the recovery site.

Introduction

When considering dual datacenter strategy with VMs recovery in mind, one important decision is whether to adopt an active/active or active/standby model. The former is generally much more complex to manage because it requires double the work in terms of procedures, testing and change controls. In addition, capacity management becomes challenging as you need to accommodate physical resources to be able to to run all workloads within whatever site. On top of that, stretched VLANs are sometimes deployed across datacenters so that recovered VMs can keep their IP addresses. This is generally very costly if you want to leverage proper L2 extension technology, such as Cisco OTV.

Alternatively, in a SDDC environment, you can leverage VMware NSX to efficiently manage connectivity and network changes required in the event of a full site failover. NSX gives you the ability to maintain the same IP address scheme for all you workloads by leveraging APIs, with little effort. Or with more granularity, you could even move a single subnet as part of a specific recovery plan. NSX will make this possible by providing:

  • An overlay network that allows you to decouple the backend VM network from the physical network. NSX-V is using VXLAN, each ESXi host acting as a VTEP.
  • Programmability through RESTful APIs that allows you to provision Logical Switches and modify Logical Routers configuration in seconds.
  • Dynamic routing protocol (OSPF, IS-IS, BGP) that will advertise VM subnets to your enterprise network, making them accessible for end users or other applications (North-South or East-West traffic)

NSX Components

As many NSX introduction blog posts can be found on the web (like here or here), I’m not gonna spend much time on this topic. NSX components are:

  • NSX Manager: it’s the single point of configuration and the REST API (and UI) interface. It is provided as a VM appliance and is actually the only appliance you have to manually install. There is a 1:1 mapping between the vCenter Server an the NSX Manager. The manager is responsible for deploying NSX Controllers, NSX Edge Gateways and Logical Router Controllers. It also installs the Distributed Routing and the firewall kernel modules on ESXi hosts, as well as the User World Agent (UWA). NSX configuration is accessible through vCenter once you’ve installed the NSX plugin.
  •  NSX Controller: it provides a control plane to distribute VXLAN Logical Routing and Switching network information to ESXi hosts. It also enables ARP suppression to reduce flooding. It is typically implemented as a 3-node cluster and maintains MAC, ARP and VTEP tables. It is finally responsible for installing routes into each ESXi host.
  • Logical Switch (LS): it acts as the L2 domain boundary for VMs, identified by a VXLAN ID (VNI) and associated with a specific subnet. Its vCenter representation is a distributed Portgroup with specific capabilities.
  • Distributed Logical Router (DLR): it’s the distributed L3 first-hop for VM traffic. As its name suggests, it’s completely distributed. You can think about it as an anycast gateway, where each ESXi corresponds to a node, sharing a single virtual IP and virtual MAC address. The data-path routing process runs within each ESXi in vmkernel space and enables East-West traffic optimisation, avoiding well-known hair-pinning effects when VMs want to talk to their default gateway.
  • Logical Router Control VM: it provides the DLR with a control plane and can be deployed as a redundant pair of VM appliances, in an active/standby fashion. It supports both OSPF and BGP as dynamic routing protocols. The Control VM receives its initial configuration from the NSX Manager.
  • Edge Services Gateway (ESG): it provides network perimeter services to the virtual environment. It is intended for North-South communication, i.e. between the physical and the virtual network or at the edge of your tenant. It is NOT distributed, meaning that its placement is critical. It can run in HA-mode, where the appliances are deployed in an active/standby fashion. The HA mechanism doesn’t rely on VMware HA (as some people at Cisco seem to think), but with minimum common sense, you’re gonna create a DRS anti-affinity rule to separate active and stanby VMs. Depending on specific requirements, the edge gateway can be deployed with several sizes:
    • Compact (1vCPU – 512MB RAM)
    • Large (2 vCPUs – 1GB RAM)
    • Quad-Large (4 vCPUs – 1GB RAM)
    • X-Large (6 vCPUs – 8GB RAM).

Available services include: Firewall, NAT, DHCP, Routing, Load-Balancing, Site-to-Site VPN, SSL VPN and L2VPN.

  • Distributed Firewall (DFW): It enables distributed security capabilities at VM NIC level as an East-West L2-L4 stateful firewall. The module is present on each ESXi host as a kernel module and therefore removes any form of bottleneck. If you need more bandwidth, just add a new host! It also includes the Service Composer feature, which allows you to create specific services by integrating additional 3rd party capabilities to the firewall, such as endpoint services (e.g. Anitivirus, Data Security) and deep packet inspection (Palo Alto). I have to say that this feature is one of the most compelling to me!

The following picture shows how those components fit together:

togetherNSX

Basic Understanding

To understand NSX concepts, it’s useful to map vSphere network components to NSX components:

In a traditional vSphere environment, a VM wishing to communicate with the outside world first hits a virtual port on the virtual switch. This virtual port is part of a Portgroup, which is basically a group of virtual network ports tagged with a specific VLAN ID. In the NSX world, when a VM is part of a Logical Switch, it hits a virtual port member of a Portgroup specifically created by the NSX Manager. It is created on every host member of the VDS, like a traditional distributed Portgroup. However, the difference is that all egress frames hitting this Portgroup will be forwarded inside a VXLAN tunnel, tagged with a specific external VLAN ID to transport the VXLAN frames on the physical network .

The role of the Logical Router is to connect two or more Logical Switches together, enabling routing between the corresponding subnets (you can assume 1 LS = 1 subnet). It also advertises (and learns) prefixes and routes to its neighbor(s) if a dynamic routing protocol has been activated. Alternatively, you can also configure static routes.

As an example, the following diagram shows the DLR establishing adjacency with the ESG, which is also running a dynamic routing protocol, and advertises VM subnets to the physical world. The ESG has its internal interface connected to a VXLAN and its uplink connected to a VLAN. As a result, the physical network can learn about the virtual network, and vice-versa.

example

Lab Architecture

Now that you’ve had a basic introduction to NSX principles, I can detail my scenario. In my lab environment, I’ve simulated the following architecture:

globalNSX

I didn’t actually deploy two sets of controllers and two managers linked to two different Virtual Centers in separate physical datacenters. Instead I’ve created logical containers called “Transport Zones” to make both virtual datacenters completely independent from a data-plane standpoint. The goal here is to demonstrate how to integrate virtual network operations into an orchestrated Disaster Recovery Plan with NSX. The only requirement is the ability to run a script as part of your DR procedures. This may be ultimately be achieved by VMware Site Recovery Manager, or another orchestration tool.

This architecture represents a traditional dual datacenter environment, connected over a L3 IP cloud. In a standard network environment, it basically means that you have to change VMs IP addresses upon recovery. (There are other alternatives, like host routes, RHI and NAT, but these solutions come at a certain complexity cost).

The main goal of the scenario is to show how to provide a flexible orchestrated Disaster Recovery solution without having to change VMs IP addresses. Let’s see how we can achieve this with NSX. The order of operations would be:

  1. Disconnect LS1 and LS2 in DC1.
  2. Create new LS in DC2: DR_LS1 and DR_LS2 (Or pre-create them without connecting the upstream DLR).
  3. Add two new interfaces to DLR2 in DC2, with the same IP addresses as previously used by DLR1 to connect LS1 and LS2. In this way, we don’t have to change the default gateway of the recovered VMs.
  4. Connect those interfaces to the corresponding LS.
  5. Recover VMs in DC2.
  6. Connect VMs to the appropriate LS.
  7. Boot VMs and test connectivity.
  8. Check route updates on the physical network.

Note: I’m assuming here that security devices configuration are synchronized between datacenters.

NSX

Because OSPF is running within the virtual network on both DLR1 and DLR2, routing updates will be sent up to the IP cloud to reflect that DR_LS1 and DR_LS2 subnets are now reachable through DC2. In the same way, because LS1 and LS2 have been disconnected from DLR1, corresponding routes will be removed to reflect that LS1 and LS2 subnets are not reachable in DC1 anymore. Magic??!! No, just awesome technology :-)

The next post will focus on how to deploy this lab environment.

Nicolas

 

Physical Networks in the Virtualized Networking World

[This post was co-authored by VMware's Bruce Davie and Ken Duda from Arista Networks, and originally appeared on Network Heresy]

Almost a year ago, we wrote a first post about our efforts to build virtual networks that span both virtual and physical resources. As we’ve moved beyond the first proofs of concept to customer trials for our combined solution, this post serves to provide an update on where we see the interaction between virtual and physical worlds heading.

Our overall approach to connecting physical and virtual resources can be viewed in two main categories:

  • terminating the overlay on physical devices, such as top-of-rack switches, routers, appliances, etc.
  • managing interactions between the overlay and the physical devices that provide the underlay.

The latter topic is something we’ve addressed in some other recent posts (herehere and here) — in this blog we’ll focus more on how we deal with physical devices at the edge of the overlay. Continue reading

Getting Started with VMware NSX Part II – Building Virtual Networks

In the previous post we took a look at the simplicity of deploying VMware NSX into a new or existing VMware environment. This post looks to develop upon our existing infrastructure and build out a three-tier application with a Web, App and Database tier.

Application Topology

Screen Shot 2014-06-26 at 3.10.48 pm

This application displayed above highlights what this blog seeks to build out. Note that there are three logical network segments – Web, App, DB and Uplink (Transport) – routing functionality provided by the Logical Distributed Router and an NSX Edge Services Gateway that is used to connect the logical network topology to the physical infrastructure. Continue reading

Micro-Segmentation: VMware NSX’s Killer Use Case

The advantages a software-defined data center, using network virtualization as a core underpinning, include service delivery speed, operational efficiency, reduced hardware dependency and lower cost. However, by far the most popular use case by customers thus far has been the use of NSX for network microsegmentation. Why? Because perimeter-centric network security has proven insufficient, and micro-segmentation has to date been operationally and economically infeasible. With NSX, security teams, in partnership with their network and virtualization teams, are benefiting from network micro-segmentation to begin to transform their data center security architecture. Then read the VMware SDDC Micro-Segmentation White Paper.

Rod

Getting Started with VMware NSX Part I – Preparation

This short series will focus on how virtualization administrators and network engineers alike can easily and efficiently deploy VMware NSX and network virtualization into their existing environments. From the simple and seamless installation, building your first virtual network to management and administration of an NSX environment, this series will highlight how easy it is to gain the benefits of network function virtualization.

Integrating NSX Manager into vCenter

Integration of the NSX manager into vCenter is the first task to be undertaken. NSX manager helps create a management plane for the NSX environment. When this is connected it will provide the Networking and Security plugin. It exposes a RESTful API for consumption by a customer or a cloud management platform. Such examples of those that can integrate with this API are vCloud Automation Center or OpenStack. Log into the NSX manager web interface with the credentials you specified during installation. Continue reading

VMware NSX Runs Great on VCE Vblock Systems

Vblock SystemsLast week at EMC World in Las Vegas, one of the industry’s best offerings in converged infrastructure was on display. The adoption of converged infrastructure is becoming increasingly common in many organizations. In fact, research estimates that the total addressable market for converged infrastructure will reach $402B by 2017. Companies are taking advantage of converged infrastructure to accelerate cloud and software-defined data center deployments. Converged infrastructure is used by IT organizations to reduce provisioning times, centralize the management of IT resources, and increase resource utilization rates – resulting in lower costs. These objectives are enabled by the creation of pools of compute, storage and networking resources that can be shared by multiple applications and managed in a collective manner using policy driven processes. Continue reading