by: Lead Application Administrator Gopala Krishna Sandur Kustigi
To read more about how VMware IT has implemented micro-segmentation using NSX Data Center in various applications, click here
VMware IT employs Kafka as a message processing broker for a variety of applications including our ticketing system, content management and license management systems. This is due to the fact that it provides a highly scalable, durable, reliable, high-volume message processing capacity. Kafka’s mission-critical nature also means security is paramount. That’s why we leveraged VMware NSX Data Center for micro-segmentation.
Enablement of micro-segmentation via NSX Data Center was planned and executed within a short duration, using post QA cycles. Micro-segmentation enables fine-grained network security policies—based on a virtual machine’s (VM’s) purpose—that are distributed throughout the network and follow a VM as it moves. This is important as many successful infrastructure-related attacks are due to mistakes made over time or poor maintenance. Since the policy follows the VM, the application is protected throughout its lifecycle, and administrative overhead is reduced.
Life without micro-segmentation
The Kafka connectivity architecture employs firewall rules that protect external access, but east-west communications are not secured.
Micro-segmentation process
Using NSX Data Center to do micro-segmentation, we were able to implement a security model which specified permitted traffic on specific ports. This was accomplished with Kafka using a well-defined process, one that starts with application discovery (the process of identifying unique data flows), and then followed by dynamic security groups and policy creation, traffic analysis, firewall review, testing, and deployment.
The process is as follows:
- Server identification. Captured server names and grouped them according to functionality of application–producer microservices, consumer microservices applications, etc. We checked sys log settings and firewall exceptions, so they could be configured at the VMware ESXi™ host level. The sys logs were forwarded to log management tools such as VMware vRealize® Log Insight™.
- Server groups. Divided servers into functional groups. For example, log aggregator, analytics, and ticketing system. By applying firewall rules to a group instead of individual VMs, management was simplified and centralized.
- Group security policies. Wrote separate firewall rules for each group, then mapped the security policies to the security groups.
- Traffic capture/analysis. Used Log Insight to capture, identify, and analyze flows in and out of Kafka.
- Firewall rules. Reviewed the firewall rules for east-west traffic with the application development teams for each module, including the usual traffic between the load balancer, micro services.
- Testing and deployment. Conducted extensive testing in a pre-production environment to ensure Kafka functioned normally for all the producer and subscriber application/micro services. We reviewed issues that were reported during pre-production testing, checked the relevant source-target IPs and port numbers, and made the required modifications. We followed the same process to implement the rules into production.
Kafka with micro-segmentation
The entire application is secured using NSX Data Center distributed firewalls that protect integrations and the application layers. Each VM now acts as its own perimeter. Unauthorized traffic is blocked because security policies are implemented depending on groups and ports.
Micro-segmentation rules
Below is the complete rule set for the Kafka micro-segmentation, excluding monitoring and security management tools.
Micro-segmentation key learnings
As our team made the transition, an issue occurred with one of the applications in Prod. One of the Consumer micro-services that connect to the Kafka broker could not establish a connection to Kafka. Due to this, a significant number of messages were stuck in Kafka, causing a lag in message processing. We soon noticed the 9092 port of the Kafka broker was not allowed from the VMs hosting a few of the consumer microservices. Once discovered, we remedied the situation.
VMware on VMware blogs are written by IT subject matter experts sharing stories about our digital transformation using VMware products and services in a global production environment. Contact your sales rep or [email protected] to schedule a briefing on this topic. Visit the VMware on VMware microsite and follow us on Twitter.
