For detection of life signs on other ships or planets the Star Trek fleet uses “A sophisticated array of charged cluster quark resonance scanners” which “provide detailed biological data across orbital distances”. (From “The Star Trek: The Next Generation Technical Manual“, written by Rick Sternbach and Michael Okuda).
We are still designing the intergalactic version (I have heard it is on the roadmap), but in the meantime I would like to show, how to detect Docker containers running on VMs in your vSphere world.
Yes, Docker containers! Some of those potentially illegal and dangerous species may have infiltrated your virtual environment and live in a symbiotic way inside apparently harmless Linux virtual machines. You may have been betrayed by developers who looked in your eyes and asked for “just a standard linux VM” and told you, they will take care of installing applications all by themselves.
Our quark resonance scanner we are going to use to detect those life forms is Log Insight. Our main bridge display is provided by vRealize Operations Manager showing VM compliance.
Every time Log Insight is involved things turn out to be simple… The Docker host is writing logs, logs are collected by Log Insight. We just need to define the proper keywords and send automatic alerts to the vRealize Operations Manager. In vROps we will receive notification event, which can be used in an alert definition. You could combine it with other metrics or properties (for example alert on Docker running on a production VM, or alert on Docker if the load is getting high) or just leave it as a single symptom.
vROps policies and groups will let you create dynamic or static groups of VMs where Docker is allowed and others for which you should receive alerts. I certainly don’t consider Docker to be evil but you may want to create dedicated “play grounds” for containers where they are safe and keeping them out of systems where other production software is running. By the same token, we don’t let children play in the factories or on the highways.
Here is the technical part: how to implement the Docker detector.
1.Retrieve Docker logs
We need to retrieve relevant logs. The keyword docker0 will appear in /var/log/messages as soon as docker is started. Docker is also writing its own log file, called /var/log/docker. In our example we create an agent group in Log Insight (assuming log insight agents are included in every linux VM) and add following config section:
[filelog|docker]
directory=/var/log/
include=docker
tags={“compliance”:”alien”}
The advantage of using agent groups and propagation of a central configuration via the server component is obvious: we don’t need to login to each and every VM and change syslog configuration manually. Log Insight will take care of it.
2. Create an Alert in Log Insight and forward it to vROps
Once we have the Docker log file included we can start some containers and check messages in Log Insight. We will notice messages containing keywoard “containers” and different actions, like create, start, wait:
To keep it simple, we will just create an alert based on the query for messages containing the keyword “containers/create” and send this alert to vROps. There are some more possibilities to explore: for example we could run a query of unique count of Docker container IDs, check how many containers are started or stopped or which repositories are used to download the images.
No one wants to get an email for every container started, so we will just send the alert to vRealize Operations manager. In vROps it will appear as an notification event which can be used as a symptom (or one of many symptoms) to raise an alert or change VM’s compliance badge. Thanks to Log Insight / vROps integration we will automatically receive the alert on the right VM object, although it is actually coming from operating system inside the VM.
3. Create a compliance alert on vROps
On the vROps side we create a symptom for Virtual Machine (base object) which is based on “Message event” and the message contains a keyword “Docker”:
We use the symptom and add it to an existing VM compliance alert or create a new compliance alert for “Virtual Machine” base object with critical risk impact and we can also add custom recommendations (like “Please call your compliance hotline”) and actions (“Move VM to docker playground cluster”)
And that’s it. Now we can enable or disable this new compliance alert for certain parts of our environment and enjoy looking at the compliance badge of the virtual machine:
We can also create a new view and use it in dashboards and alerts, the view will display the names of all VMs running Docker. Note that we use an Alert as a subject and get the names of VMs through alert property (object name).
There are certainly other ways to detect Docker, especially if you have the new endpoint agent installed on linux (vROps End Point agent). In that case, we can check for the Docker daemon and monitor it, or run a script and count all the active containers and display a metric with a number of containers. We also have some other possibilities in Log Insight, for example to extract the field containing the container ID and run unique count or create alerts only in case of untrusted remote repositories which are used to download images.
We hope to see further EPOPS agent and Log Insight enhancements in the Docker area in near future. Quark resonance scanner is possibly on the roadmap too.
P.S. Thanks to the anonymous customer who brought up this use case and Steve Flanders for his Log Insight support!
