Change Management – seen as both a blessing and a curse by IT. When I ran infrastructure operations for a global ASP back in the day, I saw it as a necessary evil that I learned to love over time. While at VMware, I’ve worked with customers who either don’t have change management or don’t fully, completely, and consistently apply it. The result being chaos, late nights with nothing to show because of back-outs, or worse a poorly implemented change resulting in outages down the road if not right away. I’ve even seen this happen with customers who have a well-defined and executed Request for Change and Change Advisory Board process. Why? Because they don’t complete the process by properly orchestrating the change window or performing the post change audit. While a blog isn’t the appropriate place to fully address these change execution challenges, there is room to highlight one addition to the change window that can make an enormous difference – monitoring during the change window. This may seem obvious, but it’s amazing how often it’s not included as an explicit part of the change management process.
Let’s use a simple example of a distributed firewall ruleset change in the context of NSX micro-segmentation. Let’s say we’re either implementing or changing an inter-application firewall ruleset effecting communication between App A and App B in different criticality or confidentiality zones like that shown below.
Depending on the VMware tools you have at your disposal, there are at least three ways to easily monitor the change as it happens to ensure nothing untoward happens with communications between the applications.
As of NSX for vSphere 6.3, a new capability called Application Rule Manager and accessible via the vSphere Web Client, can be used to easily see flows and affected rules between VMs as shown below:
While the power of Application Rule Manager is really for modeling application flows for creating Security Groups and whitelisting firewall rules, it works well for quickly capturing flows generated by tests during the change window to determine if the new or modified rules are working as expected.
VMware vRealize Log Insight is also a great tool for monitoring firewall rule changes during the change window (as well as after, of course). You can quickly see application traffic and rules affected as shown below.
And of course, you can easily drill into the details by rule-ID using Interactive Analysis to see the details.
Finally, if you have an NSX deployment of any scale, you’ll probably have VMware vRealize Network Insight which is purpose built for intelligent operations of your NSX-based environment. There are many ways to view and analyze traffic flow and firewall rules in-play with vRealize Network Insight but probably the simplest is to view the flow between VMs via the ‘Plan Security” selection. From here you can select VM in question select Analyze and see an easy to read graphic like that shown below.
From here you can drill in to see endpoint (including port) and more detailed flow information generated by the tests being run during the change window.
Monitoring what’s going on as you test the changes being implemented not only proves the change is working as expected but also provides operations with insights into what to expect after the change is complete and the change window closed. This also points out the best practice of including operations in your change planning as well as explicitly including the tests you’re going to run as well as the expected results as a standard section in your Request For Change. Doing so will not only increase the success of your changes, but improve production monitoring post-change, and create an even better working relationship with operations, which is always a good thing.
=======
I want to call out Geoff Wilmington of our Network and Security Business Unit for the great screenshots, thanks Geoff!
Kevin Lees is the field Chief Technologist for IT Operations Transformation at VMware, focused on how customers optimize the way they operate VMware-supported environments and solutions. Kevin serves as an advisor to global customer senior executives for their IT operations transformation initiatives and leads the IT Transformation activities in VMware’s Global Field Office of the CTO.
