Home > Blogs > VMware Consulting Blog


Using Secondary Management Network for vSphere Replication

By Sunny Dua, Enterprise Solution Architect, VMware Professional Services

Sunny DuaI have written a lot about vSphere Replication best practices. This time, I want to share an experience of implementing SRM and vSphere Replication in a brownfield virtual infrastructure.

As we all know, vSphere Replication uses the management interface for the ESXi servers to send the replication traffic to the DR site vSphere Replication Appliance. It is important that we understand the network flow clearly before diving deeply into configuration of the networks. The diagram below illustrates how the data flows.

Understanding Network Flow
Let’s see how the traffic flows in generic sense before we add IP addressing to it:

Network Flow

  1. Changed blocks are captured by the VR Filter on the ESXi server in the primary site.
  2. This data is sent to DR Site VR Appliance using the primary management interface of the ESXi server.
  3. The VR Appliance in the DR Site passes the data to the ESXi servers in the DR Site using the NFC Service.
  4. This data is then written on to the designated DR Site data store.

Note: Just reverse this sequence to do a reverse replication while doing re-protect in SRM.

A Real-Life Scenario
Let’s look at a real life setup and see how this replication will flow. Here’s a quick view of my setup, along with the IP addresses:

vSphere setup

 

Let’s look at each component one by one:

  1. This is the IP address of the vCenter Server. Notice that the IP sub-nets are different in the Primary Site and DR Site.
  2. This is the IP address of the SRM Server. Notice that the IP sub-nets are different in the Primary Site and DR Site, similar to vCenter Server.
  3. The IP address of the VRA server is not in the same range, because you do not want to use same IP segment as the management network. In this case I have a point-to-point connectivity between the site and the IP configured is on the 10.12.12.x sub-net. This is configured on both sites, as the VRA server will receive the traffic from the ESXi servers on this interface. Remember, this appliance will connect on a virtual machine port group.The default gateway for this sub-net is 10.12.12.1 at the Primary Site and 10.11.12.1 at the DR Site.
  4. VMK0 is the primary management network interface. This is used to manage the ESXi servers in the primary site. Notice that ESXi and vCenter are on the same sub-net.
  5. VMK1 is configured for vMotion on a non-routable VLAN, which is why I have a completely different IP segment here.
  6. VMK2 is the third VMKernel interface I have configured. This is to use the point-to-point connectivity for vSphere Replication. I want the vSphere Replication traffic to go out of this VMK interface and reach the vSphere Replication appliance on the DR Site.
  7. One of the most important things to note that in case of ESXi, the Default Gateway will always be defined with VMK0. Hence, all the VMKernel port-groups will have the same default gateway.

The last point is the problem for me. I do not want the vSphere Replication traffic to hit the gateway (172.16.3.1) in the DR site when the traffic is sent to the vSphere Replication appliance in that site. I want it to hit the gateway configured for 10.11.12.x sub-net. To be precise, the default gateway is 10.11.12.1.

Define a Static Route
This is not possible until you define a static route to force the vSphere Replication Traffic to go through the vSphere Replication Interface (VMK2) and then hit the vSphere Replication appliance on the DR Site with the default gateway. Remember, you will have to just reverse this action and add a static route on the ESXi servers in the DR site for (10.12.12.1) default gateway in the primary site.

Here are the commands to do define a static route:

~ # esxcli network ip route ipv4 add –gateway <Gateway for vSphere Replication Subnet> –network <IP range for vSphere Replication Network in DR Site>

So in my case I would run the following command:

~ # esxcli network ip route ipv4 add –gateway 10.12.12.1 –network 10.11.12.0/50

You also need to add this line to the rc.local to make this setting consistent across reboots.

~ # vi /etc/rc.local.d/local.sh

Add the following line just before the exit command in the script:

~ # esxcli network ip route ipv4 add –gateway 10.12.12.1 –network 10.11.12.0/50

Save and exit from this file and you are done on the primary site. You need to do the same on the ESXi servers in the DR site for reverse replication to work. The command for DR site ESXi servers is:

~ # esxcli network ip route ipv4 add –gateway 10.11.12.1 –network 10.12.12.0/50

Note: Remember to add this to the local.sh script as you did in the primary site.

Now let’s see how the traffic flows in this case:

vSphere Traffic Flow 2

 

Here is a KB article from VMware that might help you with this setup:

Configuring static routes for vmkernel ports on an ESXi host (2001426)

Hope this makes things easy for you and allows you to setup vSphere Replication on your preferred network interface.


Sunny Dua is an Enterprise Solution Architect for VMware’s Professional Services Organization, focused on India and SAARC countries. He holds several certifications including VCP, VCAP-DCA, and VCAP-DCD and the VMware vExpert award. Sunny also speaks regularly at VMware User Groups conferences. Most of his ramblings on virtualization and cloud computing can be found on his personal blog: vxpresss.blogspot.com. He tweets as @sunny_dua.

14 thoughts on “Using Secondary Management Network for vSphere Replication

  1. Marian

    In the add route command, I think the network part should be an IP or a subnet represented with /## bit subnet mask.
    Are you sure that you can use /50 as a range? What if it was a range of 24? Would you write it as /24? This would be iterpreted as a subnet mask.

    Reply
  2. Angelo

    I believe the route add should be the following:

    esxcli network ip route ipv4 add –gateway 10.11.12.1 –network 10.12.12.50

    The static route is only for the single IP address of the VR Server. This ensure all replication traffic (to the VR server) is routed out the interface that is on the same network as the gateway that is defined.

    Reply
  3. Pingback: Replication Diagram |

  4. Almero

    GREAT arcticle .

    What version of vSphere was this based on ? 5.1 ?
    Do you have an updated Data Flow for vSphere 6 now that you can pin VR and VR-NFC traffic ? I assume no Static routes required here any more ?

    Reply
  5. Tapas Mallick

    Great Article.

    can you please blog another post considering ESXi 6 which having Two Replication VMkernel with Different TCP/IP Stack to isolate Replication traffic effectively?

    Should we go for a VMware Replication Server with 3 Interfaces (Management, Incoming Replication Traffic and Another to forward to NFC VMKernel Port)?

    Please suggest.

    Reply
    1. Drex

      In order to implement VMKernel replication adapters, your entire environment has to be on 6.x. You can’t use it with vCenter 6.x and ESXi 5.5 even though vSphere Replication is supported in that configuration.

      Reply
  6. Kooliiy

    This architecture should NOT work. It does NOT mention how VR Appliance communicate with VC. Are 172.x and 10.x routable network? Does VR Appliance or VC has dual vNic? VR Appliance 5.x does NOT support multi vNic.

    Reply
  7. chris

    Hi Sunny

    if the site-site replication goes over the mgmt vmk by default (vmk0) and we set our mgmt vmk to use 1Gb adapters … I assume that limits all site-site replication to 1Gb ?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*