Credit to NICO GUERRERA for this blog post (Bio Below)!
As we all know, Kubernetes and container technologies are currently exploding in adoption in data centers and public clouds around the world. With such aggressive growth of this new technology, we need to make sure we properly operationalize it to ensure a successful deployment. A key operational consideration for any new data center technology is log aggregation. VMware already has great log aggregation products in vRealize Log Insight and its SaaS counterpart, Log Intelligence. With the few simple steps outlined below, we can easily configure our container logs to flow into vRLI/LInt and be comfortable in the fact that we can do root cause analysis, configuration management, and other event-based analytics as easily with Kubernetes as we can with other technologies such as ESXi and vCenter.
Setting up Fluentd and the Fluentd plugin for vRLI
So, why Fluentd over something else like the vRLI guest agent? We’re going to look at using Fluentd as our log data collector because it is the Kubernetes recommended data collector, its open source, and it is very flexible. You can send logs via many different output plugins and many different protocols. It is compatible with dozens of data sources and plugins are easy to deploy, as we will see below.
1.) Install Fluentd, and the vRLI and Kubernetes metadata filter plugins on your Kubernetes nodes.
This first step requires that you have Ruby installed on your nodes so you can use RubyGems to install the plugins. This is the easiest method that I have found. Installing Ruby on your nodes depends on your Linux OS and package manager and should be done before we continue.
Once you have Ruby installed, install Fluentd first.
Second, install the vRealize Log Insight and Kubernetes metadata plugins.
Ruby builds these plugins in ‘/var/lib/gems’, which means you would need to specify the full path of the gem directory where the plugins were built when starting Fluentd. To make things easier, I copied the plugin files to ‘/etc/fluentd/plugins’, which is the default directory that Fluentd looks for plugins. No extra parameters needed on startup.
2.) Setting up the Fluentd configuration file to send logs to vRealize Log Insight
Now that we have Fluentd set up, we need to build a configuration file. Luckily, our friends at VMware who built the vRLI plugin for Fluentd provided us with a very good basic configuration file on Github.
https://github.com/vmware/fluent-plugin-vmware-loginsight
All we need to do is copy their configuration file into our /etc/fluentd/fluentd.config file and change the hostname parameter to our Log Insight VIP or Log Intelligence Cloud Proxy. I also disabled SSL verify in my lab since I don’t have SSL set up properly in my vRLI instance.
3.) Starting the Fluentd service
Once we have the configuration file in place, we can manually start Fluentd with
sudo fluentd -c /etc/fluentd.conf &
The & is to run the process in the background. The Fluentd gem doesn’t come with /etc/init.d/ scripts. You should use process management tools such as daemontools, runit, supervisord, upstart, or system to autostart the service.
If Fluentd starts properly you should see the output in the console saying that it successfully parsed the config file.
Now we are ready to query Log Insight or Log Intelligence for our Kubernetes logs!
Using vRealize Log Insight to Query Kubernetes Logs
We’re finally at the exciting part. Now once we log into vRLI, we should be able to query. Let’s start by querying for all logs from our K8 pods that contain ‘nginx’.
Since we have Nginx installed in our pod, we can see HTTP 200 calls from the logs from each container. Next, let’s scale up our Nginx K8 deployment and add another container.
Now we have three containers available. Let’s check to see if Log Insight captured that.
I searched for the phrase ‘Scaled up’ on my K8 node and sure enough, vRLI returns the event where my replica set was increased to three pods. From here, we can make a custom chart so we can see how many times our pod scales up over a period of time. The chart can tell us when our peak processing periods are by the number of pods that are added over the day.
Then, when the chart is created, we can check it every time we log in to vRLI to see container spin up activity within our pod.
We can also create an alert to email us if a massive number of pods are created in a short period of time, or if none are created over a period of time, signaling that something might be wrong.
Conclusion
Hopefully, this is a useful beginner’s guide for capturing Kubernetes logs with Log Insight. Things only get more complicated from here, but this is a solid starting point for anyone who needs to begin operationalizing Kubernetes and capturing log output. This is also a good primer for Fluentd, as any good Kubernetes admin should know how to configure Fluentd, and its output plugins, so they can aggregate and forward logs in a standard, well-supported way. Finally, if you’re running Kubernetes on VMware, Log Insight and Log Intelligence will provide a single pane of glass view for root cause analysis and event correlation all the way up and down the stack, from application to storage.
Nico Guerrera is a senior technical account manager who has been with VMware since 2016. He is a captain for the cloud management TAM tech lead team and focuses on vRealize Log Insight and vRealize Log Intelligence. Nico has 13 years of VMware career experience and is also an avid Linux/open source software enthusiast.
