VMotion between Data Centers—a VMware and Cisco Proof of Concept
VMotion, the VMware feature that enables live VM migration between ESX hosts is one of the major attractions of vSphere and before that, VMware Infrastructure (or VI for short). It’s simply quite amazing to watch a VM continue operation and maintain sessions while moving from one host to another.
As cool as this is, we’re often asked, “How do we take that one step further, and perform VMotion between datacenters?” This, of course, is a non-trivial thing to do. There is the challenge of moving a VM over distance (which involves some degree of additional latency) without dropping sessions. To maintain sessions with existing technologies means stretching the L2 domain between the sites--not pretty from a network architecture standpoint. And then there is the storage piece. If you move the VM, it has to remotely access its disk in the other site until a Storage VMotion occurs.
Last year, Cisco and VMware began the task of trying to solve these long distance VMotion issues with the target of seamlessly migrating a VM between two datacenters separated by a reasonable distance. The joint Cisco/VMware lab in San Jose has run number of tests over varying distances (simulated with reels of optic fiber) as a proof of concept. We will demonstrate this proof of concept at Cisco Live this week in San Francisco. The demo as it stands incorporates a distance of 80km (50 miles). That’s around 400us latency each way over fiber or a round trip just under 1ms.
This proof of concept is aimed at the following requirements:
- Load balance compute power over multiple sites: Migrate VMs between datacenters to “follow the sun” or to simply load balance over multiple sites. Enterprises with multiple sites can also conserve power and cooling by dynamically consolidating VMs to fewer datacenters (automated by VMware Dynamic Power Management (DPM))—another enabler for the Green datacenter of the future.
- Avoid downtime during DC maintenance: applications on a server or datacenter infrastructure requiring maintenance can be migrated offsite without downtime.
- Disaster Avoidance: Data centers in the path of natural calamities (e.g. hurricanes) can proactively migrate the mission critical application environment over to another data center.
Use cases #2 and #3 above also require a Storage VMotion to move the disk image to the alternative datacenter.
Remember, this is a proof of concept, so we still have work to do in multiple areas. e.g. the storage VMotion for disaster avoidance and so on.
See and hear about it at Cisco Live this week…
Cisco Live is on this week in San Francisco. We will feature briefings in the VMware theatre (booth #531 …adjacent to the big Cisco booth). Refer to the theatre schedule posted at the VMware booth for session times. (of course, you can just come and ask us about it anytime)
See an update at VMworld in San Francisco—August 2009
We will demonstrate this again at VMworld (http://www.vmworld2009.com/) in San Francisco in August where we will also hold a technical breakout session on VMotion between datacenters. This will cover the proof on concept, test results, and reveal a little more of our plans to solve some of the remaining issues. Look for this session in the Technology and Architecture track.
Posted by Guy Brunsdon on June 29, 2009 | Permalink |
This is quite interesting! I remember one year back I did a VMotion across two of our datacenters using a shared iSCSI SAN, but of course my problem was in the IP addressing part after the VM is migrated to the new DC. So I’m very interested to know more about the networking design in this PoC. I can’t wait for the VMworld session!
Posted by: Hany Michael | June 29, 2009 at 11:53 AM
Hany - If you read again you will see they say "To maintain sessions with existing technologies means stretching the L2 domain between the sites--not pretty from a network architecture standpoint."
Whats this mean? It means the vlan (aka network) can be active at multiable sites and the IP will work work from both DCs. If setup correctly, it works fine and you can move your VM from one DC to the other without issues.
Posted by: bitsorbytes | June 29, 2009 at 03:24 PM
Idea wise is cool, but still think that practically to locate your storage or synchronize the storage require for both side will be difficult. We have SRM to work on the DR today, and VMotion across data center could be useful if we plan to relocate the data center
Posted by: Craig | June 30, 2009 at 02:00 AM
bitsorbytes, Yes, the article says that this is the traditional way! Now I’m interested to know what the PoC offers in this perspective. There is no more information given here about the network design, this is why I’m very interested to know more about the underlying network infrastructure used in the PoC.
Posted by: Hany Michael | June 30, 2009 at 02:14 AM
We need new DRS Rule options that allow us to create (automate) DRS Storage VMotion after a DRS VMotion.
Posted by: Erik Bussink | June 30, 2009 at 04:34 AM
The potential for global workload balancing is fantastic using this kind of technology.
Posted by: Alex Bewley | June 30, 2009 at 12:36 PM
you can make L2 cross datacenter with l2tp v3 vpn
Posted by: Corfdir Francois | July 01, 2009 at 07:22 PM
You can do that today already.
You just need DataCore software.
It creates an Active/Active SAN storage mirror. vSphere sees each side of the mirror (may be located on different sites)as one path, so VMotion works seamlessly.
Posted by: Miguel Veguillas | July 08, 2009 at 11:47 AM
Craig,
The underlying Cisco networking infrastructure is Catalyst 6500 VSS configuration with MEC between 2 sites 80 KMs apart for the LAN extension part. You get highly resilient architecture with up to 80 Gbps bandwidth. One can also use the Nexus 7000 vPC for this type of configuration (may need to use ONS for longer distances). For Storage extension, we used CWDM for native FC and FCIP (sharing bw on the IP side)
Posted by: Balaji Siva | July 12, 2009 at 07:28 PM
Having dark fiber connectivity as in the network illustration is one thing. In reality, data centers will be interconnected over different types of wide area networks including IP, over much longer distances. There will also be need to extend storage networks (including fibre channel SANs).
Our company provides technology called Accelerated Secure Edge (ASE) which extends compute and storage networks over IP/MPLS wide areas. ASE provides a virtual wire extension (encrypted, MTU transparent, ultra low latency, native FC protocol support, etc.) which enables VMotion and storage motion over long distances.
ASE is deployed in real networks, connecting data centers between countries. More information is available from our web site: www.aforesolutions.com/ASE/ or contact me rob_marson@aforesolutions.com.
Posted by: RMarson | July 20, 2009 at 04:46 AM
Visit AFORE at VMWorld in the in the New Innovator Pavilion, booth 1438J. We will be providing a live demonstration accessing a cloud computing service provider's real network, performing live long distance VMotion using our Virtual Fiber and Virtual Wire solutions implemented in our
AFORE ASE3300 service delivery platform.
Posted by: RMarson | August 29, 2009 at 08:32 AM
The joint Cisco/VMware validated reference architecture paper is posted here
http://www.cisco.com/en/US/solutions/collateral/ns340/ns517/ns224/ns836/white_paper_c11-557822.pdf
or check out recording of the TA3105 at VMworld2009
Posted by: Balaji Sivasubramanian | September 04, 2009 at 10:45 AM