Over the recent years, App modernization & Kubernetes adoption is gaining huge momentum. According to the latest VMware Kubernetes study, the use of Kubernetes in production is climbing year on year.
To help organizations in their Kubernetes journey, VMware has recently released Application Transformer for VMware Tanzu, it leverages vRealize Network Insight to discover applications and helps modernize/containerize the monolith and legacy applications.
While Application Transformer for VMware Tanzu is how you modernize your applications, what about after? With the increase in container workloads in the data center, it is critical to get network visibility and secure it. In this blog post, we will see how to use vRealize Network Insight & Antrea CNI to monitor and secure container workloads.
Antrea is an Opensource CNI Network Plugin providing network & security for Kubernetes, maintained by VMware. It is based on Open vSwitch, it is a widely accepted & high-performance networking solution that provides network overlays for Pod-Pod communication.
Solution Overview:
We will leverage Antrea Netflow (IPFIX) to get container network observability in vRealize Network Insight. Antrea agent sends IPFIX data to Antrea Flow aggregator, it processes and aggregates the flows, then it forwards to vRealize Network Insight Netflow collector. After a few minutes, the vRealize Network Insight platform process the flows and it will be available for consumption. We can leverage search queries, applications, and pinboards to monitor & secure the container workloads.
Prerequisites :
- Kubernetes Clusters (Native Kubernetes or Tanzu Kubernetes Grid 1.5.1 & above) with Antrea CNI
- Please Note: This solution doesn’t work on vSphere with Tanzu as configmap modification is not allowed
- Latest vRealize Network Insight Enterprise edition or Cloud version with an admin user
- vRealize Network Insight Netflow collector
Note: Default CNI in Tanzu Kubernetes Grid is Antrea. If you have Native Kubernetes, please install Antrea CNI
Steps overview
- Deploy dedicated vRealize Network Insight Netflow collector
- In a Kubernetes + Antrea setup, enable flow exporter in the Antrea agent
- Deploy Flow aggregator service and set Netflow Collector address as IPFIX collector IP
- Verify if Antrea agents send flows to the Flow aggregator
- In a few hours, flow data will be available in vRealize Network Insight
Detailed instructions:
Deploying dedicated Netflow Collector in vRealize Network Insight.
Please follow the below steps, if you don’t have a dedicated Netflow Collector.
- Log in to vRealize Network Insight with admin access
- Go to the settings
- Select Accounts and Data Sources
- Click the Add Source button
- Choose Netflow Collector as the data source type (on the bottom)
- Click the Add Collector VM button. This will generate a shared secret key, copy it to the clipboard. We will use it when deploying the collector.
- Deploy a new vRealize Network Insight Collector
- Note the collector IP address, we will use it later while configuring Antrea Flow Aggregator
Enabling Netflow(IPFIX) in Antrea
Instructions for Native Kubernetes:
- Find the antrea agent config file
$ kubectl get configmaps -n kube-system |grep antrea-config
antrea-config-2k82d6f6t4 3 95d
- Edit the Antrea Config, Uncomment FlowExporter, and change the value from “false” to “true”
$ kubectl edit configmaps antrea-config-2k82d6f6t4 -n kube-system
- To take effect, we need to restart Antrea Agent pods from All nodes. After restart, verify if all agent pods in running state
$ kubectl rollout restart ds/antrea-agent -n kube-system
$ kubectl get pods -n kube-system|grep antrea-agent
antrea-agent-hdgd2 2/2 Running 0 2m59s
antrea-agent-j6kjv 2/2 Running 0 3m10s
antrea-agent-m4jsf 2/2 Running 0 3m21s
antrea-agent-vzvr9 2/2 Running 0 2m48sInstructions for Tanzu Kubernetes Grid
- Tanzu Kubernetes Grid(TKG) controller doesn’t allow editing of configmap files and it will revert any manual changes
- To modify the configuration in the TKG system, we need to leverage FeatureGates, which allows us to edit settings
- FlowExporter option doesn’t exist in the TKG system by default but is available in Antrea, we need to add it in TKG config files to leverage it
- Use your favorite text editor to add the below config files
- “
ANTREA_FLOWEXPORTER: false" --> .config/tanzu/tkg/providers/config_default.yaml "FlowExporter: #@ data.values.ANTREA_FLOWEXPORTER" --> .config/tanzu/tkg/providers/ytt/02_addons/cni/antrea/antrea_addon_data.lib.yaml"ANTREA_FLOWEXPORTER": ["vsphere", "aws", "azure", "docker"]" -->.config/tanzu/tkg/providers/ytt/lib/config_variable_association.star
Verify if added successfully
Note: From TKG 1.6 onwards, the FlowExporter option will be available by default, you can directly add the Featuregate.
When you are creating a new TKG cluster, add the below FeatureGate to the cluster config file. The new cluster will have IPFIX enabled and it will start forwarding to flow-aggregator.flow-aggregator.svc, which we will deploy shortly.
ANTREA_FLOWEXPORTER: "true"
Deploying Antrea Flow Aggregator
The main functionality of the Antrea Flow Aggregator is to store, correlate and aggregate the flows received from the Antrea agent and forwards them to the External Flow collector
Download Flow Aggregator yaml file
$ wget https://raw.githubusercontent.com/antrea-io/antrea/main/build/yamls/flow-aggregator.yml- Add the Netflow Collector Ip
$vi flow-aggregator.yml → externalFlowCollectorAddr: "Netflow_Collector_Ip:2055:udp" kubectl apply -f flow-aggregator.yml
After successful deployment, you can start seeing flows in a few hours. We can use a series of search queries, pinboards, and applications to get traffic insights from Antrea flows.
For more information regarding Antrea IPFIX, please check the Antrea documentation
Visibility in vRealize Network Insight
Conclusion:
This is a first step toward getting container network insights via Antrea Flows. More monitoring and security capabilities are under development and will be available in the future.
