Application Rule Manager (ARM) Practical Implementation – Healthcare

This post originally appears as part of a series of VMware NSX in Healthcare blogs on Geoff Wilmington’s blog, vWilmo. To read more about VMware NSX and its applications in healthcare, check out Geoff’s blog series.

VMware NSX 6.3 introduces a new toolset within NSX that can be leveraged for quick micro-segmentation planning. The Application Rule Manager (ARM) within NSX, provides a new way to help create security rulesets quickly for new or existing applications on a bigger scale than Log Insight, but smaller scale than vRNI.   With that in mind, we’re going use ARM in NSX to create our rulesets using the same basic methodologies for an application.

The Application Rule Manager in VMware NSX leverages real-time flow information to discover the communications both in and out, and between an application workload so a security model can be built around the application.  ARM can monitor up to 30 VMs in one session and have 5 sessions running at a time.  The beauty of ARM is that it can correlate the information that you would typically have to either have or look up in Log Insight to create your rulesets and significantly reduces time to value.  ARM can also show you blocked flows and the rules that are doing the blocking.

Use case – Provide a Zero Trust security model around a Healthcare Organization’s EMR system.  Facilitate only the necessary communications both to the application and between the components of the application.

• Allow EMR Client Application to communicate with EMR Web/App Server
• Allow EMR Web/App Server to communicate with EMR Database Server
• Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.
• Allow bi-directional communication between the Infrastructure Services and the entire EMR Application

Problem – The Healthcare organization does not have a clear understanding of how the application communicates within and outside the organization.  Organization wants to lock down the EMR application so that only known good workstations can access.

Technology used –   

Windows Clients:

• Windows 10 – Clinician Desktop – Client-01a (192.168.0.36)
• Windows 10 – HR Desktop – Client-02a (192.168.0.33)
• Mac OSX – Unauthorized Desktop – (192.168.0.18)

VMware Products

• vSphere
• vCenter
• NSX 6.3.0+ and Application Rule Manager

Application in question –

Open Source Healthcare Application:

• OpenMRS – Open Source EMR system
  • Apache/PHP Web Server
  • MySQL Database Server

Infrastructure Services:

• NTP

The first thing we need to do to utilize ARM is establish the VM’s that we’re going to look at the flows from.  These systems will represent the session that we will build and run for a period of time to examine what flows are coming in and going out, and between these systems.

Since Infrastructure Services are generally global services that affect more than one system within a data center, we’re going to build the rules that accommodate our NTP virtual machine first.  We’re going to see more flows to other VM’s in the lab, but we’ll be narrowing it down to just the EMR systems.  Then when we’ve got those working, we’ll do the EMR-based rules.

First, we need to establish our naming scheme and methodology to writing our rule sets.  Below is a chart I use when creating my rules for the applications.  It helps me lay things out logically and then apply it quickly in the DFW.

 

You will notice that I like to nest my Security, Services, and Groups.  I do this because at scale, this makes changing things much simpler and more efficient.  If I have to swap in a service or change a port, I want the changes to be reflected to all affected groupings down the table without having to seek out where else I might need to make a change.  This has consequences as well, you must pay attention to what you’re doing when you make a change as it can have effect on other applications.  This is why I name things the way I do, to ensure that any change you make that you understand the implications.

Now that we have all our custom services and naming convention in place, we can go back to the ARM and start resolving Services, Service Groups, and building out rules.  We’re going to take these flows and squeeze them down even further to create very succinct rules with as few entries as possible.

We’re going to create a session called INFRA MONITOR and add the NTP-01a server into the monitor so we can see all the flows coming in and out of it. (Figure 1)

The amount of time that you collect flows for is up to you.  My advice is the following:

• Understand the application you’re planning for. If this application is a billing cycle application that does 30 day monthly cycles, ARM my not be the tool to use.  ARM was built to monitor flows in real-time over the maximum of a 7-day period.  If the application you’re looking at has periodic flows that occur over a longer period of time, I suggest using vRealize Network Insight for those applications.  It has a longer retention period for looking up flows.
• Keep the applications small. ARM has a 30 VM upper limit on monitoring of flows per session, with up to 5 sessions.  While most applications may fall into this range, some applications may have more than 30 machines.  In that case, my suggestion is breaking up the application into smaller chunks and running the chunks in different sessions to keep the numbers down.
• ARM uses NSX Manager resources. Just like Flow Monitoring, ARM uses resources from the NSX Manager to collect flow data.  Be careful of NSX Manager utilization during this process.  Unlike Flow Monitoring, if you exit the interface, ARM will continue to run until you stop the session collection.

Once we hit OK, the flow collections will begin. When we’re comfortable that we’ve captured all the flows from the VM, we can stop the collection and ARM will take the number of flows, de-duplicate the flows down to unique flows for us. (Figure 2)

Now we’re going to analyze our flows.  This is going to resolve any VM names it can as well as resolve the ports to protocols that they could be. (Figure 3) I’ve hidden other flows to other systems within the lab for simplicity sake for now, and only focused on finding the flows to the EMR system.  You’ll also notice a column that I added into the View Flows tab.  The column is Rule ID.  The Rule ID column will show the applied rule that affects that flow in the View Flows table.

As you can see from the output, all the flows appear to be hitting the Rule ID of 1001.  If you click on the Rule ID 1001 link, it will pop open the details of the Rule from the DFW (Figure 4).  While this is showing an allowed flow, we will see later, that we can also see flows being blocked by a Rule ID.

The View Flows tab only resolves to VM names.  We’ll need to take this information and create appropriate Security Group, Service, and Service Group names for our rules.  We’ll click on the gear icon next to the EMR-DB-01a VM and select ‘Create Security Group and Replace’. We will name the Security Group after the naming scheme you’ll find below of EMR-SG-DB and we’ll statically add in the VM, EMR-DB-01a. Note: A user can use any static or dynamic criteria to create the Security Group. Application Rule Manager will auto-create the security group and check whether the VM (in this case EMR-DB-01a) will be part of the new security group.

What we’ll find is that the Security Group now replaces the virtual machine as the Source.  But since we’re both of the VM’s that make up the EMR talking to the same destination, we’ll create an EMR-SG-ALL Security Group and nest the individual VM Security Groups within it.  To do this, we can click on the gear for the Source we just changed, and select ‘Create Security Group and Replace’ same as before but this time we’re going to add the EMR-SG-DB Security Group to the EMR-SG-ALL Security Group we’re going to create. We should now see that the Source has changed to the EMR-SG-ALL GROUP.

For the EMR-WEB-01a VM, we’ll do the same thing.  Create a Security Group called EMR-SG-WEB and add that VM to it. We’ll then click on the gear and select ‘Add to existing Security Group and Replace’.  Then select the EMR-SG-ALL group, and this will nest the EMR-SG-WEB Security Group into the main group. We repeat this same process for the NTP-01a VM according to the layout above.  Create an INFRA-SG-NTP and add the NTP-01a system.  Create a INFRA-SG-ALL and add the INFRA-SG-NTP Security Group to it.  This will allow us to add more SG’s to the main group later as needed without having to write another rule to do it.

When done, it should look like this.

You only need to do this to one rule as you’ll see later, we’ll grab both of these flows to write one rules.  We focus in on the Services that we found now.  If you click on the ‘2 Services’ link, you’ll see what ARM resolved the ports to, protocol-wise.

I like creating my own Services and Service Groups.  This way we know if we make changes, we know what could possibly be impacted.  So with this, we’re going to perform a similar workflow as we did above.  Create a Service and Replace.  Create a Service Group and Replace.  This will add the custom Service we create to the custom Service Group we create. Leaving us with this final result.

We’re now ready to create our rule from the two flows.  Selecting both rules, we right click and select ‘Create Firewall Rule’ as shown in Figure 11.

Clicking on Create Firewall Rule will auto-populate the ruleset as show in Figure 12. We’ll see some unresolved items in the list, but we’re going to make modifications so that this one rule covers both flows.  We’re going to:

• Remove the NTP-01a as the Destination leaving the INFRA-SG-ALL
• Remove the UDP 123 Service leaving the INFRA-SVG-ALL
• Remove the vNIC for NTP-01a and add EMR-SG-ALL and INFRA-SG-ALL

Once we click on OK, we’ll have a new rule created in the ‘Firewall Rules’ tab. That is a temporary cache for you to see the rules, edit it and maybe get approval for the rules. Double check our rule and make sure it looks correct and then hit ‘Publish’.  We’re then prompted to create a section name and select where to insert it.

We should get confirmation that the publish was successful and we can go to the ‘Firewall’ interface and verify our work.

Everything looks good! Now we can focus on the EMR system.

Talking with our applications team, we have determined that the VM’s in question are:

• EMR-WEB-01a
• EMR-DB-01a

Now that we know the names of the VM’s, we can go into the Flow Monitoring section of the NSX Management console and select Application Rule Manager.  From here we can Start New Session.

We’ll select the VM’s we discussed above, EMR-WEB-01a and EMR-DB-01a, and we’ll add them to the session.  This will start collection of flow data from the vNIC’s of these VM’s and post them in the View Flows table.

The amount of time that you collect flows for is up to you.  My advice is the following:

• Understand the application you’re planning for. If this application is a billing cycle application that does 30 day monthly cycles, ARM my not be the tool to use.  ARM was built to monitor flows in real-time over the maximum of a 7-day period.  If the application you’re looking at has periodic flows that occur over a longer period of time, I suggest using vRealize Network Insight for those applications.  It has a longer retention period for looking up flows.
• Keep the applications small. ARM has a 30 VM upper limit on monitoring of flows per session, with up to 5 sessions.  While most applications may fall into this range, some applications may have more than 30 machines.  In that case, my suggestion is breaking up the application into smaller chunks and running the chunks in different sessions to keep the numbers down.
• ARM uses NSX Manager resources. Just like Flow Monitoring, ARM uses resources from the NSX Manager to collect flow data.  Be careful of NSX Manager utilization during this process.  Unlike Flow Monitoring, if you exit the interface, ARM will continue to run until you stop the session collection.

From here we will create a new session, calling it EMR Monitor, and add in our VM’s to the session.  Once we hit OK, the collection will start and we can stop the collection when we’re comfortable with the period of time we collected for.  In this instance, I’m going to collect data for 15 minutes.  I have automated tasks running to provide traffic to the EMR so we can see flows that run every 5 minutes, so this should be long enough for this demonstration.

Once we have stopped the collection, we should see the View Flows table has several flows showing.  ARM will attempt to de-duplicate repeated flows as much as possible.  As you can see from the picture below, we went from 17 flows seen down to 7 unique flows total.

From here, we need to Analyze the flows.  When you click on Analyze, ARM will attempt to correlate the flow information into VM names as necessary or IP addresses if the flows fall outside of the NSX environment.  It will also attempt to resolve services that are being leveraged in that flow as well.

Now that ARM has analyzed our flow data and matched where it can we can see a few things:

• Direction – This tells us in what direction the flow came to the source
  • IN – Inbound
  • OUT – Outbound
  • INTRA – Between
• Source – Resolved to a VM name if the VM falls under the scope of the vCenter/NSX Manager relationship. If an IP address, represents an external IP system that is not resolvable in the vCenter/NSX Manager relationship.
• Destination – Resolved to a VM name if the VM falls under the scope of the vCenter/NSX Manager relationship. If an IP address, represents an external IP system that is not resolvable in the vCenter/NSX Manager relationship.
• Service – Resolved to all services that exist within NSX. If more than one service is shown, this means that the user will need to manually pick the service it correlates to because NSX has more than one service definition with that corresponding port number. If you want to create a custom Service or Service Group, which we will

With the information shown we have the following:

• Inbound 192.168.0.99 > EMR-WEB-01a TCP 8080
  • 168.0.99 is identified as the Management Jumpbox for the infrastructure.
• Outbound EMR-DB-01a > NTP-01a UDP 123
• Outbound EMR-WEB-01a > NTP-01a UDP 123
• Inbound 192.168.0.36 > EMR-WEB-01a TCP 8080
  • 168.0.36 is identified as the Clinician Desktop
• Intra EMR-WEB-01a > EMR-DB-01a TCP 3306

From this view, we can start creating our Security Groups, IP Sets, and Services as necessary for our rule sets.  But first, we need to establish our naming scheme and methodology to writing our rule sets.  Below is a chart I use when creating my rules for the applications.  It helps me lay things out logically and then apply it quickly in the DFW.

We also notice that the infrastructure services for NTP are showing up in our flows as well.  We’ll need to account for these services, but we’ll put them in their own section as other applications may need access to those services as well.  Below is the layout for the infrastructure services based on the groupings we created for the EMR above.

 

You will notice that I like to nest my Security, Services, and Groups.  I do this because at scale, this makes changing things much simpler and more efficient.  If I have to swap in a service or change a port, I want the changes to be reflected to all affected groupings down the table without having to seek out where else I might need to make a change.  This has consequences as well, you must pay attention to what you’re doing when you make a change as it can have effect on other applications.  This is why I name things the way I do, to ensure that any change you make that you understand the implications.

Now that we have all our custom services and naming convention in place, we can go back to the ARM and start resolving Services, Service Groups, and building out rules.  We’re going to take these flows and squeeze them down even further to create very succinct rules with as few entries as possible.

The first thing we need to do is to create our Security Group structure that we defined above.  We’ll start with the EMR servers and then do the Infrastructure systems.  In the Source Field for the EMR-WEB-01a, we’re going to click the gear icon in the box and select ‘Create Security Group and Replace’

This will bring up the Security Group configuration dialogues and we’ll statically add the EMR-WEB-01a VM to the Security Group.

Once we complete this step, the View Flows section for the Source of EMR-WEB-01a will be replaced by the Security Group for EMR-SG-WEB.

We’ll repeat this process for all the rest of the VM’s that were resolved in the Flows and then we’ll deal with the IP addresses of the Jumpbox and Clinician Desktops. For repeated VM names where you’ve already created the Security Group, you can select ‘Add to existing Security Group and Replace’ instead.  This will bring up the Security Group memberships and allow you to choose the appropriate one.

We should look like this now.

For the IP addresses, since we’re not using VM’s themselves, we’ll need to create the IP Sets for each of these machines.  Similar to how we created the Security Groups, above, we will create the IP Sets, and then the Security Groups, and replace the IP addresses with the EMR-SG-ACCESS Security Group with the IP Set’s in each.  This will continue to follow our methodology similarly to how we did the VM’s.

Once we’re all done we should look like this now.

Now if you take a look at the results posted above, we could group a few of these Security Groups into ALL groups and narrow down our rules even further.  For example, we have both the DB and the WEB systems talking to NTP.  We could create an ALL group with both Security Groups inside, and remove one of the flows from the table.  Here’s what we need to do.  We’re going to pick one of the flows and select ‘Create Security Group and Replace’.

We’ll create the EMR-SG-ALL that we had in our table above.  And we’ll add the EMR-SG-WEB and EMR-SG-DB which contains both servers into this ALL group.

Since we now have a Security Group that has both VM’s in it, we can hide one of the flow entries as it’s already covered in the EMR-SG-ALL Group.

We then hide the other unneeded entry and we’re getting closer to the minimum number of rules necessary to segment this application.  As you can see we also have another set of Sources talking to the same Destination over the same protocol of TCP 8080.  We can collapse these two flows as well.

We’re now left with the minimum flows and we can now resolve Services to our custom Services we were planning to create. This process is no different than what we did with the Security Groups and then putting them into an ALL Security Group. We’re going to create the custom Service, and then replace it with a Service Group with that custom Service inside it.

Repeating the process until we’re all done, nets us this final result.

With all of this work now done, we can go ahead and create our DFW rule sets.  We’ll start with the Infrastructure Section and the associated rules.  To create a rule from the View Flows table, we simply right click on the row and select ‘Create Rule’.

You’ll be prompted to put in a name and confirm the entries in the other lists.  ARM will default to picking the vNIC’s of the VM’s in question.  If you recall with Flow Monitoring, you have to specify a vNIC to capture flows from. ARM is taking that model and applying it here as well.  The nice thing is is that you do not have to constrain the DFW rule to the vNIC of the VM.  What you should do is change the ‘Applied To’ field to the Source.  We also need to change the ‘Direction’ to Out instead of In/Out.

When you complete the rule build, you can select the ‘Firewall Rules’ tab and see the new rule you just created.

We’re going to repeat this process for the other two flows as well.

Create a section for the EMR and publish the rules to the DFW.

Verify our results in the DFW interface.

We can now add our block rules to the DFW for the EMR application.  Unless we specifically allowed the traffic to pass, we’re blocking all other communications.

Now we test our verification of our requirements:

• Allow EMR Client Application to communicate with EMR Web/App Server
• Allow EMR Web/App Server to communicate with EMR Database Server
• Block any unknown communications except the actual application traffic and restrict access to the EMR application to only a Clinician Desktop system running the EMR Client Application.

 

When we attempt to access the EMR from the Jumpbox system, we can see that we’re getting prompted to login.  This means that the Web and DB systems are talking to each other properly.  Looking at Live Flows of the vNIC of the EMR-WEB-01a VM, we can see that we’re hitting the proper Firewall rule sets.

We can also see that the block rule is working as I attempt to access the EMR system from another workstation 192.168.0.18, and I’m getting blocked by the DFW.  We just need to check access from the Clinician Desktop and verify working properly.

Looks like we’re fulfilling all the requirements so far.  Time for our last check.

• Allow bi-directional communication between the Infrastructure Services and the entire EMR Application

Here we have EMR-WEB-01a reaching out to the NTP server.

Here we have EMR-DB-01a reaching out to the NTP server.

As you can see, ARM is very adept at helping simply micro-segmentation with VMware NSX.  Taking the concepts we learned and leveraged with Log Insight, we can remove quite a bit of manual processes involved with resolving services and VM’s IP addresses to their names.  I was able to perform this process in about 15 minutes outside of flow capture time from start to finish and that’s what makes ARM another very useful toolset to use for reducing the complexity of micro-segmentation.

---