network optical fiber cables and hub
VMware Cloud on AWS Networking

VMware Cloud on AWS Advanced Networking and Routing Features

VMware Cloud on AWS has been powering customers on their cloud journey for 4+ years.

Along the way, we have made significant enhancements to the SDDC Networking and Security capabilities. VMware Transit Connect and Multi-Edge accelerate connectivity from on-prem/ native cloud to VMware Cloud on AWS.  New offers such as NSX Advanced Firewall strengthen the SDDC security posture in keeping with the zero-trust mindset prevalent nowadays. Operational tools such as NSX Traceflow enable self-service troubleshooting for connectivity issues in the SDDC.

As customers expand their footprint on VMware Cloud on AWS, they uncover new use cases and needs. They need to scale beyond the limits imposed by AWS route advertisement quotas. Network operations teams and Managed Service Providers need a simple way to logically separate different departments or tenants hosted in the SDDC. Many customers have overlapping IP addresses among their applications across multiple data centers and desire to avoid renumbering their network segments and applications when they migrate them to the cloud.

Today, with the release of SDDC version 1.18, we are doubling down on the foundational virtual networking and routing capabilities of VMware Cloud on AWS.

  1. Route Aggregation for route advertisement over Transit Connect/ Direct Connect/ Connected VPC
Figure 1 SDDC Route Aggregation for Direct Connect  

Customers have often sought a way to scale networks beyond the limits imposed by AWS route advertisement quotas.  This feature enables customers to aggregate routes for advertisement over external interfaces (Transit Connect/ Direct Connect/ Connected VPC). By selecting aggregated routes for their workload segments, customers can significantly reduce their Transit Connect route table size even while they expand their workload segments.

Configuring Route Aggregation

Customers can utilize the NSX VMC Policy API or the NSX Manager UI to configure route aggregation. In the UI, the Global Configuration tab introduces Route Aggregation Prefix Lists that can be applied to Transit Connect/ Direct Connect (Intranet endpoint) or the Connected VPC (Services endpoint). As a manually configured route, these prefixes are always advertised to the respective end points. Note that the management network (MGW) is always advertised separately.

Shared Prefix List This feature enables customers to share a Managed Prefix List that is auto-populated with aggregated SDDC routes with the Connected VPC. VMware Cloud on AWS manages this shared prefix list, keeping it up to date with SDDC routing changes. As earlier, VMware Cloud on AWS configures the Connected VPC default route table with SDDC routing changes. Customers who opt for the Managed Prefix List gain the option to configure additional custom route tables in the Connected VPC by adding this prefix list, ensuring these route tables are in sync with SDDC routing updates.

Figure 2 SDDC Route Aggregation for Transit Connect and Connected VPC

2. Multi-CGW enables customers to create additional CGWs (Compute Gateways or Tier-1 Gateways) in the SDDC. 

Figure 3 Customer Managed CGW (Tier-1Gateways) in the SDDC

Every VMware Cloud SDDC is created with a standardized topology consisting of a Management Gateway (MGW) and a Compute Gateway (CGW) for routing network traffic inside the SDDC. Customers create logical segments on the CGW to connect workloads to the NSX overlay network in the SDDC.

The Multi-CGW feature powers more use cases for applications with overlapping network addresses and for Disaster Recovery testing. It introduces the ability for customers to create additional CGWs as Routed, NATted or Isolated CGWs and manage the lifecycle of these CGWs (also referred to as Tier-1 Gateways).

Customer Managed CGW (Tier-1 Gateway) Types

Routed CGWs are connected to the NSX overlay network. Workload VMs behind Routed CGWs can communicate with other CGW (including the default CGW) workloads. They can also reach the Internet using Elastic IP (EIP). Customers can configure route aggregation to enable Routed CGW workloads to communicate over Transit Connect/ Direct Connect (Intranet endpoint) or Connected VPC (Services endpoint) as described in the previous section. Only the explicitly configured addresses in route aggregation prefix lists are advertised externally, giving customers fine-grained control over reachability to workloads on additional CGW.

NATted CGWs require NAT to be configured to ensure connectivity to the SDDC’s NSX overlay network. As with Routed CGWs, workloads on NATted CGWs can communicate externally when using route aggregation. Addresses behind the NATed CGW are not advertised, which enables creation of overlapping CIDRs in the SDDC. Advanced customers find this capability useful when supporting tenants or applications with overlapping IP addresses. This lets customers avoid renumbering (re-IP) their applications when they migrate them to the cloud, saving a significant amount of time, effort and risk – a key benefit of VMware Cloud on AWS.

Isolated CGWs are designed to be disconnected from the rest of the SDDC. The isolated CGW serves as a local router without connectivity to the rest of the SDDC networks or to the external environment. Workload VMs on isolated CGW subnets can communicate among themselves but not to VMs on other CGWs. The Isolated CGW configuration is often used to simplify certain advanced use cases such as DR (Disaster Recovery) testing.

Figure 4 Routed, NATted and Isolated CGWs

Networking and Security Features for Customer Managed CGW

Customer managed CGWs offer a powerful set of networking and security capabilities for advanced use cases.

As discussed above, customers can establish external connectivity for additional CGW workloads by configuring route aggregation over TGW/ DX/ Connected VPC. By combining Routed/ NATted/ Isolated CGWs, customers can enable applications with overlapping addresses and unlock multi-tenancy use cases in the SDDC.

Figure 5 Connectivity and Routing for Multi-CGW

Customers can also establish Route-based or Policy-based VPN to Customer Managed CGW. All customer managed CGWs include Gateway Firewall/ NAT services to secure the VMs connected to them. The Tier-1 Gateway Firewall/ NAT rules are applied locally to the CGW.

Figure 6 VPN connectivity to Customer Managed CGW

Configuring Customer Managed CGW

Customers can create their additional CGW using the NSX VMC Policy API or NSX Manager UI. In the UI, you can use the newly introduced Tier-1 Gateways tab to create additional CGW as Routed, NATted or Isolated CGW. After creating a new CGW, you can create logical segments (in the Segments tab) to attach workload VMs. To configure a local DHCP server, you first setup a DHCP profile in the IP Management section. You can also advertise these segments externally using route aggregation prefixes configured in the Global Configuration tab.

Each additional CGW features a Gateway Firewall and NAT. Tier-1 Gateway Firewall/ NAT policies can be configured in the Gateway Firewall and NAT tabs in the UI. Each additional CGW can be selected and managed separately under the Tier-1 Gateways section. As in prior releases, the NAT service under the Internet section enables access to the Internet using Public IP or Elastic IP (EIP).

Similarly, Route-based or Policy-based VPN can be configured in the VPN tab under the required Tier-1 Gateway. To configure VPN directly to a customer managed CGW, you first create a VPN service for the required Tier-1 Gateway. After the Tier-1 VPN service is created, you can establish an IPSec session to a local VPN endpoint scoped to the required CGW.

NSX Traceflow is a tool to visualize the path of a packet from any source to destination in the SDDC. It helps customers identify the hop by hop path including DFW/ Gateway Firewall rules that affect the flow of packets in the SDDC. With this release, Traceflow also provides visibility into packet path on customer managed CGW.  

To summarize, this feature enables:

  • Multi-tenancy within an SDDC
  • Overlapping IPv4 address space across CGWs
  • Gateway Firewall/ NAT policies scoped to individual CGWs
  • Route/ Policy Based VPN connectivity to individual CGWs
  • Support for static routes on customer managed CGW
  • Access to the Connected VPC from customer managed CGW
  • Route Aggregation for route advertisement over Transit Connect/ Direct Connect/ Connected VPC
  • Share a Managed Prefix List auto-populated with aggregated SDDC routes with the Connected VPC.
  • Deployment of Isolated test ‘segments’ for Disaster Recovery (DR) testing or “sandbox” environments.

3. VMware Cloud on AWS – NSX Support for DNS FQDN Zones for Management Gateway 

Multi-tenant customers on VMC will benefit from the ability to configure DNS FQDN Zones for Management Gateway traffic.

All of these features are delivered in version 1.18 and are available exclusively on the NSX Manager UI.

VMware Cloud on AWS 1.16 introduced access to the NSX Manager UI (Networking UI in standalone mode). The streamlined layout of Networking, Security and Troubleshooting features in this UI powers a wider range of capabilities for advanced users and enables faster delivery of future enhancements to NSX in VMC. We encourage customers to familiarize themselves with the NSX Manager UI where the latest networking and security capabilities are delivered.

To learn more, please read the VMware Cloud TechZone deep dive guide: Designlet: VMware Cloud on AWS Static Routing on Multiple CGWs (T1s)

Also, you can learn more about VMware NSX Advanced Firewall add-on here: https://www.vmware.com/products/nsx-advanced-firewall-for-vmc.html

Additional Resources:

Sandeep Sharma

Sandeep is a Sr. Product Manager for Networking and Security on VMware Cloud on AWS. In this role, he helps enterprise customers run mission critical applications using hybrid cloud connectivity…

Related Articles