Home > Blogs > VMware vSphere Blog

VMotion between Data Centers—a VMware and Cisco Proof of Concept

VMotion, the VMware feature that enables live VM migration between ESX hosts is one of the major attractions of vSphere and before that, VMware Infrastructure (or VI for short). It’s simply quite amazing to watch a VM continue operation and maintain sessions while moving from one host to another.

As cool as this is, we’re often asked, “How do we take that one step further, and perform VMotion between datacenters?” This, of course, is a non-trivial thing to do.  There is the challenge of moving a VM over distance (which involves some degree of additional latency) without dropping sessions. To maintain sessions with existing technologies means stretching the L2 domain between the sites–not pretty from a network architecture standpoint. And then there is the storage piece. If you move the VM, it has to remotely access its disk in the other site until a Storage VMotion occurs.

Last year, Cisco and VMware began the task of trying to solve these long distance VMotion issues with the target of seamlessly migrating a VM between two datacenters separated by a reasonable distance. The joint Cisco/VMware lab in San Jose has run number of tests over varying distances (simulated with reels of optic fiber) as a proof of concept. We will demonstrate this proof of concept at Cisco Live this week in San Francisco. The demo as it stands incorporates a distance of 80km (50 miles). That’s around 400us latency each way over fiber or a round trip just under 1ms.

This proof of concept is aimed at the following requirements:

  1. Load balance compute power over multiple sites: Migrate VMs between datacenters to “follow the sun”  or to simply load balance over multiple sites. Enterprises with multiple sites can also conserve power and cooling by dynamically consolidating VMs to fewer datacenters (automated by VMware Dynamic Power Management (DPM))—another enabler for the Green datacenter of the future.
  2. Avoid downtime during DC maintenance: applications on a server or datacenter infrastructure requiring maintenance can be migrated offsite without downtime.
  3. Disaster Avoidance: Data centers in the path of natural calamities (e.g. hurricanes) can proactively migrate the mission critical application environment over to another data center.

Use cases #2 and #3 above also require a Storage VMotion to move the disk image to the alternative datacenter.

Remember, this is a proof of concept, so we still have work to do in multiple areas. e.g. the storage VMotion for disaster avoidance and so on.

See and hear about it at Cisco Live this week…

Cisco Live is on this week in San Francisco. We will feature briefings in the VMware theatre (booth #531 …adjacent to the big Cisco booth). Refer to the theatre schedule posted at the VMware booth for session times. (of course, you can just come and ask us about it anytime)

See an update at VMworld in San Francisco—August 2009

We will demonstrate this again at VMworld (http://www.vmworld2009.com/) in San Francisco in August where we will also hold a technical breakout session on VMotion between datacenters. This will cover the proof on concept, test results, and reveal a little more of our plans to solve some of the remaining issues. Look for this session in the Technology and Architecture track.   


18 thoughts on “VMotion between Data Centers—a VMware and Cisco Proof of Concept

  1. Hany Michael

    This is quite interesting! I remember one year back I did a VMotion across two of our datacenters using a shared iSCSI SAN, but of course my problem was in the IP addressing part after the VM is migrated to the new DC. So I’m very interested to know more about the networking design in this PoC. I can’t wait for the VMworld session!

  2. bitsorbytes

    Hany – If you read again you will see they say “To maintain sessions with existing technologies means stretching the L2 domain between the sites–not pretty from a network architecture standpoint.”
    Whats this mean? It means the vlan (aka network) can be active at multiable sites and the IP will work work from both DCs. If setup correctly, it works fine and you can move your VM from one DC to the other without issues.

  3. Craig

    Idea wise is cool, but still think that practically to locate your storage or synchronize the storage require for both side will be difficult. We have SRM to work on the DR today, and VMotion across data center could be useful if we plan to relocate the data center

  4. Hany Michael

    bitsorbytes, Yes, the article says that this is the traditional way! Now I’m interested to know what the PoC offers in this perspective. There is no more information given here about the network design, this is why I’m very interested to know more about the underlying network infrastructure used in the PoC.

  5. Miguel Veguillas

    You can do that today already.
    You just need DataCore software.
    It creates an Active/Active SAN storage mirror. vSphere sees each side of the mirror (may be located on different sites)as one path, so VMotion works seamlessly.

  6. Balaji Siva

    The underlying Cisco networking infrastructure is Catalyst 6500 VSS configuration with MEC between 2 sites 80 KMs apart for the LAN extension part. You get highly resilient architecture with up to 80 Gbps bandwidth. One can also use the Nexus 7000 vPC for this type of configuration (may need to use ONS for longer distances). For Storage extension, we used CWDM for native FC and FCIP (sharing bw on the IP side)

  7. RMarson

    Having dark fiber connectivity as in the network illustration is one thing. In reality, data centers will be interconnected over different types of wide area networks including IP, over much longer distances. There will also be need to extend storage networks (including fibre channel SANs).
    Our company provides technology called Accelerated Secure Edge (ASE) which extends compute and storage networks over IP/MPLS wide areas. ASE provides a virtual wire extension (encrypted, MTU transparent, ultra low latency, native FC protocol support, etc.) which enables VMotion and storage motion over long distances.
    ASE is deployed in real networks, connecting data centers between countries. More information is available from our web site: http://www.aforesolutions.com/ASE/ or contact me rob_marson@aforesolutions.com.

  8. RMarson

    Visit AFORE at VMWorld in the in the New Innovator Pavilion, booth 1438J. We will be providing a live demonstration accessing a cloud computing service provider’s real network, performing live long distance VMotion using our Virtual Fiber and Virtual Wire solutions implemented in our
    AFORE ASE3300 service delivery platform.

  9. motivation to lose weight

    I believe everything composed was actually very logical.
    But, what about this? suppose you wrote a catchier title?

    I ain’t saying your content is not good, but suppose you added something
    to maybe get people’s attention? I mean VMotion between Data Centers-a VMware and Cisco Proof of Concept | VMware vSphere Blog – VMware Blogs is kinda vanilla.
    You could peek at Yahoo’s front page and note how they write
    article headlines to get people to open the links.
    You might try adding a video or a related picture or two to grab people excited about everything’ve written.
    In my opinion, it might bring your website a little bit more interesting.

  10. maphack for dota 2

    Many women think that in order to get that
    guy, they have to meet his every request. Part of this micro designer handbag’s appeal is that
    it is available in many different colors and fabrics, including exotic skins like lizard.
    getting her to miss you is more about leaving her
    alone and allowing her the time to experiencing the breakup in
    a very solo way.

    Feel free to surf to my blog post :: maphack for dota 2

  11. Pingback: Vmware 5.1 Vmotion Between Datacenters – title 1

Leave a Reply

Your email address will not be published. Required fields are marked *