VMware Transit Connect has proven itself as a valuable tool to enable high bandwidth and speed connectivity for VMware Cloud on AWS customers and their Software Defined Data Centers (SDDCs). There are hundreds of customers using this feature across the fleet in a myriad of combinations. Since the initial offering in 2020 we have worked with our partner, AWS, to expand the service’s capabilities to include SDDC Grouping across multiple regions in addition to support for Transit/Security VPC models.  These capabilities combine to provide a comprehensive networking solution to address some of the most challenging networking requirements. However, there has been one gap in the connectivity – the ability to peer the VMware Managed Transit Gateway (VTGW) with a native AWS Transit Gateway (TGW).

At AWS re:Invent 2021, the ability to peer VTGWs to AWS TGWs in the same region, also referred to as intra-region peering was announced. VMware and AWS have been working on this solution diligently and we are excited to announce VMware Cloud on AWS support for this new capability in this announcement blog. Equally exciting is that this feature will be available to VMware Cloud on AWS customers with SDDCs that are on any version. To get access to this feature, please reach out to your VMware account team.

This article will delve into one of the use cases and the details of the configuration for this feature. Before we get into the configuration, let’s review the primary use case for this solution.

It’s not uncommon for our customers to have a footprint of native AWS services in a region that use AWS TGW connectivity in addition to a VMware Cloud on AWS SDDC footprint. Prior to the intra-region peering feature customers would have to either use IPSec VPNs or Transit VPC architectures to provide the connectivity they required. While this works, there is additional protocol and management overhead with either approach and potential bandwidth bottlenecks. With the intra-region peering option, topologies can transition from the one depicted in Figure 1 to the topology illustrated in Figure 2.

Figure 1 – Transit VPC Architecture for Intra-Region Connectivity

 

Figure 2 – Intra-Region VTGW to TGW Peering

 

The elimination of additional connectivity points simplifies the design by reducing the number of hops, route tables that would need management and reduces attachments.

Now that we’ve established the primary use case, we can focus on a specific topology and review the configuration steps. The topology in Figure 3 will be built throughout the remainder of this article.

Figure 3 – Intra-Region VTGW to TGW Peering Topology for Lab

To begin building this topology, we’ll start in the VMC Console and go to the SDDC Groups tab where we’ll navigate to the SDDC Group we want to configure the intra-region peering session in. In our topology, the group is called TPM-VTGW.  Once there, we’ll go to the External TGW tab and click on add TGW as shown below in Figure 4.

Figure 4 – Add TGW in External TGW Tab

When the Add TGW button is clicked the dialog in Figure 5 appears. Fill in the appropriate fields as described below.

  • AWS Account ID: ID of the AWS Account where the TGW being peered with resides
  • TGW ID: ID of the AWS TGW being peered with
  • TGW Location: Region where the TGW being peered with resides
  • VMC on AWS Region: Region where the VTGW resides
  • Routes: AWS resource destination prefixes reachable via this peering connection

Figure 5 – Add External TGW Dialog

Once the Add button is clicked, the process to establish a peering relationship between the VTGW and the AWS TGW begins. This is an asynchronous series of API calls that may take up to 10 minutes to complete. In the VMC console, the External TGW status will show as IN PROGRESS as shown in Figure 6.

Figure 6 – External TGW Peering In Progress

If you click on the account that’s being added you can see that the peering is in a PENDING ACCEPTANCE status. This indicates the associated must be accepted in the AWS Console of the target AWS account ID.  This can be seen in Figure 7.

Figure 7 – Pending Acceptance of the peering request

From this point, we’ll transition to the AWS console for the account ID that we are peering with to complete the peering request.  We’ll navigate to the VPC console and then into the Transit gateway attachments page where we can see the request coming in from the VTGW as shown in Figure 8.

Figure 8 – Pending Acceptance in AWS Console

The next step in the process is to click on the Actions drop down menu and Accept transit gateway attachment as shown in Figure 9.

Figure 9 – Accept TGW attachment in AWS Console

You will be prompted for a confirmation and once accepted it will transition to a pending status as shown in Figure 10.

Figure 10 – Intra-Region Peering attachment in pending status in the AWS Console

 

As mentioned earlier, this is a series of asynchronous API calls between VMware Cloud on AWS and the AWS service and to complete the peering connection. It may take up to 10 minutes to complete.

It is also important to note that for the AWS TGW peering session to pass traffic, a route table needs to be associated with the attachment. In some environments where a routing table is not associated by default, the user will need to go to the AWS console and associate a routing table with the attachment.

Once completed, we can observe some of the changes to the VTGW route table with the new destination prefix being populated as reflected in Figure 11.

Figure 11 – VTGW Members Routing Table

We can also observe this new route in the SDDC’s Transit Connect routing table as shown in Figure 12.

Figure 12 – Transit Connect Learned Routes in the SDDC

The last task to complete the network connectivity would be to configure the SDDC prefixes in the AWS TGW’s route table. It’s important to note that the VTGW does not propagate its routes to the AWS TGW so a static route must be added as shown in Figure 13.

Figure 13 – AWS TGW Route Table with static route configured

Depending on the AWS network topology routes may need to be added to AWS TGW connected VPCs to direct traffic destined for the VTGW to the AWS TGW. In our topology this wasn’t necessary. From a Day Two operations perspective, it’s good to remember that as prefixes are added to either the VMware Cloud on AWS or the AWS TGW topologies the various routing tables on both sides will need to be updated.

Now that network connectivity has been established there are still additional steps that need to be completed before workloads can communicate across the expanded network. First, the Compute Gateway (CGW) Firewall in the SDDC must be configured to allow traffic between the new destinations. Customers can elect to manually define IP prefix ranges in the CGW to be very granular in their security policy. Another option is to use the system defined CGW Groups as sources and destinations. This option simplifies the firewall administrator’s job as these groups are automatically updated with prefixes as they are added and removed from the VTGW. Either choice works equally well depending on your requirements.

In Figure 14 we can see a CGW policy that uses the predefined group “Transit Connect Customer TGW Prefixes” for both source and destination operators in two different rules.

Figure 14 – CGW Policy using predefined Customer TGW Group

Also note the selection of the Direct Connect Interface in the Applied To field. This is best practice as it applies the policy only to the interface where the traffic will ingress and egress the network.

Depending on your topology, additional AWS Security Group rules may need to be updated to reflect the new networks that are reachable.

Finally, from an operational perspective, the VMware NSX-T Traceflow utility, which is available from the NSX Manager UI (requires SDDC 1.16 or later), can be used to validate the path and firewall policy through the SDDC. In this example we have a trace from a guest in the SDDC, Desktop-01 at IP address 10.1.100.10, to an EC2 instance in the VPC connected to the AWS TGW with an IP address of 172.31.1.82. The Traceflow output shows the path through the SDDC, egressing via the Intranet interface, crossing the VTGW then the peering connection to the AWS TGW. It’s good to note that Traceflow’s detailed analytics and path analysis are limited to the confines of the SDDC.  Figure 15 depicts the output.

Figure 15 – Traceflow Output

With that, we’ve established an intra-region VTGW to TGW peering session, updated route tables and security policies appropriately and verified connectivity end to end. VMware and AWS are very excited about the new networking opportunities available to customers with the addition of Intra-Region peering. Watch for additional blogs and demos showcasing increasingly comprehensive VMware Transit Connect topologies.

Additional Resources

VMware Transit Connect Enhancements – https://blogs.vmware.com/cloud/2021/09/23/vmware-transit-connect-enhancements/

VMware Cloud on AWS Release Notes – https://docs.vmware.com/en/VMware-Cloud-on-AWS/0/rn/vmc-on-aws-relnotes.html

VMware Cloud Tech Zone – https://vmc.techzone.vmware.com/

VMware Configuration Maximums – https://configmax.esp.vmware.com/home

VMware Cloud on AWS FAQ – https://cloud.vmware.com/vmc-aws/faq#networking-general

VMware Cloud on AWS Networking and Security Documentation – https://docs.vmware.com/en/VMware-Cloud-on-AWS/services/com.vmware.vmc-aws.networking-security/GUID-0CD747E8-143D-476C-BE17-7DB991B32D37.html