Home > Blogs > vCloud Architecture Toolkit (vCAT) Blog


Migrating VMware vCloud Director vApps across Distributed Virtual Switches

An interesting topic that came to our attention is how to migrate VMware vCloud Director® vApps from one distributed virtual switch to another. Recently, from the experience of one of our field consultants, Aleksander Bukowinski, we received a detailed procedure to overcome the possible service disruptions due to such a move. Aleksander has also authored a whitepaper on this topic that will soon be available for our audience in VMware Partner Central. The paper also covers in detail an additional use case with Cisco Nexus 1000V and provides PowerShell and API call samples.

Depending on connectivity mode, we can have five different types of vApps in vCD: directly connected, routed, connected to routed vApp networks, isolated, and fenced. The migration process would not require shutting down the vApps while the migration happens, but rather could generate brief network outages in case the VMs are connected to a vCloud Director Edge Gateway, or no outage at all if the VMs use isolated networks with no dependency to the Edge.

Requirements

The following prerequisites are required to perform the migration:

  • VMware vSphere® 6.x (enables cross-DVS VMware vSphere vMotion®, which is crucial for the online migration process)
  • Supported VMware vCloud Networking and Security™ versions listed below

migrating-vcloud-director-vapps-across-dvs-1

  • Supported vCloud Director versions listed below (crossed with vSphere and vCNS versions)

migrating-vcloud-director-vapps-across-dvs-2

migrating-vcloud-director-vapps-across-dvs-3

  • Supported vSphere PowerCLI™ (optional) versions

migrating-vcloud-director-vapps-across-dvs-4

Preparation

To get ready for the migration, a new distributed virtual switch (DVS) and a new vSphere cluster must be provisioned. The target cluster must have access to the same networks and datastores as the source cluster. The same port groups from the source DVS must be created on the target DVS to keep the configuration identical, and all the networks should be carefully mapped between the two clusters. For networks based on VXLAN the virtual wire ID is identical between the DVSs. The usage of VMware vCenter Server® tags to label the resource pools of the clusters would streamline the selection process of the right resources.

To prepare for the migration, the following steps must be taken:

  1. Back up the vCloud Director and vCenter Server databases and record all configuration information.
  2. Create a new vSphere cluster. Make sure that the target cluster has access to the same networks (VLANs) and datastores as the source cluster.
  3. Create a new DVS and connect it to the VMware ESXi™ hosts in the target vSphere cluster.
  4. Create all required external port groups on the target DVS. Make sure that the VLAN IDs and uplinks are assigned correctly. The port group configuration should match the port groups on the source DVS.
  5. Prepare the target cluster for VXLAN in the VMware CNS Manager.
  6. Create a new Provider VDC on the target vSphere cluster in vCloud Director.
  7. Create new external networks in vCloud Director based on port groups created on the target DVS.
  8. Merge the target Provider VDC with the source one, making sure that the target Provider VDC is primary.
  9. Disable the source vSphere cluster resource in vCloud Director. This step will stop vCloud Director from provisioning objects in the source vSphere cluster.
  10. Tag the resource pools on the target vSphere cluster with vCenter Server tags to distinguish them from the source vSphere cluster. It will help differentiate between the resource pools because the names will be identical in both vSphere clusters.
  11. Start the vApp migration process based on vApp types.

Migration Procedure

Directly connected vApps have VMs with networks directly connected to vCloud Director external networks without any Edge Gateway. They can then be migrated online without any concern about network disruption. To migrate them follow these steps:

  1. Enable the source vSphere cluster resource in vCloud Director. This step is required to allow changing networks in VMs. It only applies to VMs with directly connected Org networks.
  2. Edit the vApp in vCloud Director and add new Org networks mapped to external networks created on the target DVS. The target networks will replace the source networks later on.
  3. Use vSphere vMotion to migrate VMs that are part of the vApp to the target cluster, and select the correct port group on the target DVS. This operation can only be done using VMware vSphere Web Client.
  4. Edit the VMs in the vApp and change network assignment to the target networks. If DHCP mode is not used and IP address persistence is required, select Static – Manual mode and enter the IP addresses.
  5. Add and later remove an additional empty disk drive to every VM in the vApp. This will trigger vCloud Director database records to be updated.
  6. Remove the source external networks from the vApp. Because the vApp is running, you cannot remove them through the vCloud Director user interface. However, you can use vCloud API or vSphere PowerCLI instead.
  7. Disable the source vSphere cluster resource pool in vCloud Director.

Routed vApps (through an Edge Gateway deployed at the organization VDC level) have dependencies from the Edge Gateway, and as such might incur network disconnection during the migration, with a duration based on the service configured on the Edge. To migrate them follow these steps:

  1. Make sure that the source vSphere cluster resource is disabled in vCloud Director.
  2. Use vSphere vMotion to migrate the VMs that are part of the vApp to the target cluster. Select the correct port group on the target DVS and resource pool on the target vSphere cluster. This operation can only be done using the vSphere Web Client.
  3. Edit the Edge Gateway. Select configure services and save the NAT, static routing, and VPN service configuration. Go to Edge Gateway Properties and take a note of the Sub-Allocate IP Pools and IP settings. This step can be automated using API calls.
  4. Remove the NAT, static routing, and VPN services configuration from the Edge Gateway. Other services do not reference external networks and do not have to be removed. The removal of the configurations will start the network connectivity outage phase.
  5. Edit the Edge Gateway properties and replace the source external networks with the target external networks. This will trigger the Edge Gateway to redeploy, because the target external networks are available only on the target DVS connected to the target vSphere cluster.
  6. After the Edge Gateway redeploy operation is complete, edit the services and apply the NAT, static routing, and VPN services again. Also, apply the Sub-Allocate IP Pools and check IP settings in the Edge Gateway Properties screen. The application of these configurations will end the external connectivity outage phase.
  7. Add and later remove an additional empty disk drive for every VM in the vApp. This will trigger vCloud Director database records to be updated.
  8. Remove the source networks from the vApp. Because the vApp is running, you cannot remove them through the vCloud Director user interface. However, you can use vCloud API or vSphere PowerCLI instead.

vApps with VMs connected to routed vApp networks through an Edge Gateway, when the Edge does not exist at the Organization VDC level, but is created and removed dynamically by vCloud Director (management of these Edge Gateways is not exposed to the vCD interface). Since the VMs must be moved between VXLAN a network connectivity outage is necessary. To migrate them follow these steps:

  1. Make sure that the source vSphere cluster resource is disabled in vCloud Director.
  2. Save the information regarding the services (DHCP, firewall, NAT and static routing) from the source vApp networks.
  3. Edit the vApp in vCloud Director and add the target vApp networks. Keep the same network settings, but do not enable NAT services. The target vApp networks will replace the source vApp networks.
  4. Apply DHCP and firewall services on the target vApp networks.
  5. Use vSphere vMotion to migrate VMs that are part of the vApp to the target cluster. Select the correct port group on the target DVS and resource pool on the target vSphere cluster. This operation can only be done using vSphere Web Client.
  6. Edit VMs in the vApp and change the network assignment to the target networks. If IP address persistence is required, select Static – Manual mode and enter the IP address. After changing the network assignment to the target network, the VM will not be able to communicate with the VMs that are still on the source networks.
  7. Enable NAT and modify IP translation mappings of the target vApp network. Like all other operations, this step can be automated using vCloud API.
  8. Add and later remove an additional empty disk drive for every VM in the vApp. This will trigger vCloud Director database records to be updated.
  9. Remove the source vApp networks from the vApp. Because the vApp is running, you cannot remove them through the vCloud Director user interface. However, vCloud API or vSphere PowerCLI can be used instead.

Fenced vApps use case is similar to the previous case. The VMs are not connected to vApps networks, but rather to Org networks and the connection uses NAT through a dynamically created Edge Gateway. Even in this case the VMs must be moved between VXLANs and might experience connectivity disruption. To migrate them follow these steps:

  1. Make sure that the source vSphere cluster resource is disabled in vCloud Director.
  2. Save the information regarding the services (DHCP, firewall, NAT) from the source fenced Org networks.
  3. Edit the vApp in vCloud Director and add the target Org network/networks. Keep the same network settings, but do not enable NAT service. The target fenced Org networks will replace the source fenced Org networks.
  4. Configure DHCP and firewall services on the target fenced Org networks.
  5. Use vSphere vMotion to migrate VMs that are part of the vApp to the target cluster. Select the correct port group on the target DVS and resource pool on the target vSphere cluster. This operation can only be done using vSphere Web Client.
  6. Edit VMs in the vApp and change the network assignment to the target fenced Org networks. If IP address persistence is required, select Static – Manual mode and enter the IP address. After changing the network assignment to the target network, the VM will not be able to communicate with the VMs that are still on the source network.
  7. Enable NAT and create IP translation mappings in the NAT service of the target fenced Org networks.
  8. Add and later remove an additional empty disk drive for every VM in the vApp. This will trigger vCloud Director database records to be updated.
  9. Remove the source fenced Org networks from the vApp. Because the vApp is running, you cannot remove them through the vCloud Director user interface. However, you can use vCloud API or vSphere PowerCLI instead.

vApps with VMs connected to isolated vApps networks do not have external connectivity. They are connected to VXLAN-based isolated networks and therefore no network changes are required after the migration. To migrate them follow these steps:

  1. Make sure that the source vSphere cluster resource is disabled in vCloud Director.
  2. Use vSphere vMotion to migrate VMs that are part of the vApp to the target cluster. Select the correct port group on the target DVS and resource pool on the target vSphere cluster. This operation can only be done using vSphere Web Client.
  3. Add and remove an additional empty disk drive for every VM in the vApp. This will trigger vCloud Director database records to be updated.

Post-migration tasks

After all existing vApps and templates have been migrated to the target DVS and target vSphere cluster, the source vSphere cluster can now be decommissioned from the vCloud Director. The following steps should be taken:

  1. Verify that the source vSphere cluster resource is disabled in vCloud Director.
  2. Make sure that all the migrated VMs have been updated by adding and removing a disk drive in vCloud Director.
  3. Detach the vSphere cluster resource pool in vCloud Director.
  4. Make sure all source networks have been removed from vApps.
  5. Remove Org VDC networks that are connected to networks based on the source DVS.
  6. Remove external networks that are based on the source DVS.
  7. Disable and remove any storage policies that use datastores available only on the source cluster (if any).
  8. After all objects that have references to the source vSphere cluster and source DVS are removed from vCloud Director, you can safely delete the source vSphere cluster and DVS from the vSphere environment.

 

This blog post highlights the use cases for the migration of vCloud Director vApps across distributed virtual switches. For more details on other vCloud Air Network solutions, reference architecture examples and use cases visit the  VMware vCloud Architecture Toolkitfor Service Providers site.

Save