Don’t Leave Security Off the Table

By Bill Mansfield, VMWare Professional Services Consultant

I find myself at a large majority of my enterprise customers discussing non-technical issues. Brokering a truce between operational organizations that have evolved in their own silos, and who don’t play well with others.  In the early days of Virtualization, it was difficult to get three key parties in the same room in large shops to hash out architectural requirements and operational process. Networking, Storage, and Virtualization were typically at odds with each other for any number of reasons, and getting everyone to play nice was difficult. These days, it’s primarily Security that’s left out of the room.  A large government customer recently told me flat out “We don’t care about security”, implying that it was another department’s responsibility. Indeed, the SecOps (Security Operations) and SecEngineering (Security Engineering) teams had never been brought into a Virtualization meeting in the 7 years virtualization had been in house.

This segregation of the Security team, whether intentional or not, causes some serious problems during a security incident. Typically SecOps only has a view into the core network infrastructure and some agent based sensors that may or may not make it onto the VMs that are being investigated. Network sensors typically only exist at the edges of the network, and occasionally at the core in larger shops. Any VM to VM traffic may or may not even transit the physical network at any given time.  For a long time, the ability to watch Virtual Switches for data was not available and the Security teams got used to that. These days, all the traditional methods of monitoring and incident investigation are readily available within vSphere. The vSphere 5.1 Distributed Virtual Switch can produce NetFlow data for consumption by any number of tools. RSPAN and ERSPAN can provide full remote network monitoring or recording. Inter VM traffic is no longer invisible to Security tools. Security teams just need to be involved, and need to hook their existing toolset into the Software-defined data center. No need to reinvent the wheel. Sure we can enhance capabilities, but first we need to get the Security teams to the table and allow them to use the tools they already have.

So what are some typical questions from Security Operations about the Software-defined data center? Some of them I can answer, some of them are still works in progress.  All of which deserve their own write-ups.

How do we monitor the network?

• Port Mirroring has been around for a while, and Netflow, RSPAN and ERSPAN capabilities now allow us to function with a great deal of industry standard tools.

How do we securely log events?

• SEIM integration is fairly straightforward via Syslog or direct pulls from the relevant vSphere databases.

Where do we put IDS/IPS?

• Leave the traditional edge monitoring in place, enhance with solutions inside the vSphere stack.
• vSphere accommodates traditional agent based IPS as well as a good number of agentless solutions via EPSec and NetX API integration.  Most of the major vendors have some amount of integration.

Can you accommodate for segregation of duties?

• vSphere and vCNS vShield Manager both provide role based segregation and audit capability.

Can you audit against policy?

• This is a big topic. We can audit host profiles and admin activity in vCenter. We can audit almost anything in vCenter Configuration Manager at all levels of the stack.
• We can baseline the network traffic of the enterprise with vADP (Application Discovery Planner, not to be confused with our backup API.) We can periodically check for deltas with vADP to find anomalous traffic.

What tools work with VMware to assist with forensics and incident management?

• Again, this is another big topic. Guests are just data, and a VM doesn’t know when it’s had a snapshot taken. I’ve worked with EnCase, CAINE, BackTrack, and other tools to look at things raw. Procedurally it’s fairly simple. DD off the datastore to run through one of the usual tools and/or run the tool against copies of the VMDKs in question.
• On the Network side, tie ERSPAN to Wireshark, and use traditional methodology. If you’re feeling clever you can look at live memory by recording a vMotion.

How does legal chain of custody work for forensics on a VM?

• I’m not a lawyer. I’m not a certified forensic examiner. So, I’ve always had someone from a firm who specializes in forensics like Foundstone with me to handle the paperwork.

Is this a comprehensive list? Not at all. It’s just the beginning. The first step is getting Security to the table, and getting them actively participating in design and operational decisions. With higher and higher consolidation rations it becomes more important than ever to instrument the Virtual Infrastructure. For larger organizations, tools like EMC NetWitness can provide insight into all aspects of software-defined data center. SEIM engines like ArcSight can correlate events and provide an enterprise wide threat dashboard. For small organizations, there’s a large amount of Open Source tools available.

Security professionals, where are you seeing resistance while trying to do your jobs in the software-defined data center? What requirements are you finding most challenging to address? Let us know in the comments below!