This blog posting, like many others, was prompted by a field request from a customer. The customer wanted to understand two things:
When/How admins were logging into ESXi (DCUI or SSH)
What were they doing when they were logged in
This blog post will show you how capturing and parsing ESXi’s syslog output can help you get this information. In this blog post I’ll use VMware’s new Log Insight tool. That is not to say you need or require Log Insight! You can use other syslog or SIEM (Security Incident and Event Management) tools.
<product pitch>
If you haven’t tried Log Insight, do it. As my colleague Alan Renouf said, “…one of the easiest and intuitive installers and configuration wizards ever!” Seriously, this virtual appliance is awesome. I recall when the beta came out that someone posted that they had been using LI for a a couple of hours before they realized they hadn’t read the docs! (High praise indeed!!)
</product pitch>
My setup
A Windows 2008 domain controller
VMware vCenter Server Appliance
2 ESXi servers (SSH enabled)
1 Log Insight virtual appliance
ESXi server esx-01a.corp.local is configured with local authentication.
ESXi server esx-02a.corp.local is configured with Active Directory authentication. See Kyle’s blog for how to do that.
Normally, you wouldn’t run your ESXi systems this way, but for the purposes of the demonstration, I have them set up like that so I can show the different login messages from different authentication sources.
If you are using standalone ESXi servers, Alan Renouf’s bulk ESXi host configuration script is a handy way to modify ESXi servers to send their logs to Log Insight. You could use that script to modify your ESXi systems to send logs to the syslog receiver of your choice. If you are using vCenter+ESXi, using the configure-esx command in Log Insight is the way to go.
Logging settings
In all my tests, I defaulted to using “info” as the logging setting unless otherwise noted. Running logging set to trivia gives a lot of info, but really should only be used during a debugging process as it adds additional load on the host system and fills up your syslog receiver very quickly!
When and How are administrators logging into ESXi
The first question is “When/How are my admin’s logging into ESXi?” Let’s take these use cases.
Logging in via the DCUI
Logging in via SSH
Logging in via Client or API
Logging in via the built-in web server
Note that all syslog output has a timestamp so the “when” part of the question is rather easy to answer. What we’ll go into in more detail is the how.
Click on any image to make it bigger
After successfully logging in, let’s see what is in the syslog output.
Note: Logs are displayed in descending order so read them from the bottom up
DCUI Login
You’ll see the two events that you can search for. In the image, I’m searching for “DCUI” and “root” but what you really should search for is “authentication of user”. Here’s what that search looks like for all instances in my setup.
Authentication of User
You’ll see in this example that I’m collecting from both ESXi servers. Over the two days displayed I enabled Active Directory authentication on ESXi-02a. In the latest login, you can now see that instead of “root” logging in, it’s “[email protected]”.
Failed DCUI Login
It’s not just enough to log successful logins, but what about failed? As you can see from the image, there’s no shortage of strings to search for when it comes to failed DCUI logins! Looking for “DCUI” and “failed with authentication failure” should catch that straight away.
Let’s move on to SSH. For logging in via the DCUI, the string we searched for was “authentication of user”. For SSH, the string is a little different. In the case of SSH, we are looking for “session opened” or “session opened for user”
SSH Login
You can see in the picture all the SSH sessions that have successfully logged in. Some were the “root” account logging in and later, an Active Directory login from [email protected].
SSH Failure
SSH login failures are also an excellent source for search strings. You could search for “LW_ERROR_PASSWORD_MISMATCH”, “pam_sm_authenticate” or “PAM: Authentication failure”. Any of those will find you interesting login failure information, including where the login came from. (Note “from controlcenter.corp.local” in the top log entry. That’s the system I SSH’d from)
Client and API based logins
What if I log in using the PowerShell or vCLI? Or the VI Client (a.k.a. the “C#” client)?
Because we are connecting directly to ESXi servers in this blog article, we won’t be using the vSphere Web Client in any of these examples.
PowerCLI Login
A hint for figuring out if this is PowerCLI or not? Search for “MS Web Services Client Protocol”. You’ll see going forward, this isn’t used elsewhere.
Failed PowerCLI Login
Again, another target rich environment for strings regarding a failed login attempt using PowerCLI. Some good ones are “rejected password for user”, the same “LW_ERROR_PASSWORD_MISMATCH” as before and “Cannot login”.
vCLI Login
Searching for a client-specific string to show we are using vCLI/ESXCLI is more difficult. Unfortunately, setting logging to “verbose” or “trivia” didn’t add any additional useful information. The closest thing I could find for a unique string for vCLI logins was searching for “vimsvc” + “Auth” + “User”.
Failed vCLI Login
Error messages from a failed login attempt via vCLI are similar to others. I was unable to find a unique string that says “This is a failed vCLI login”. Searching for “Rejected password for user” and “Cannot login” will give you both vCLI and PowerCLI error messages.
VI Client/C# client example
The VI Client example is probably the easiest and certainly the clearest to figure out. Searching for “VI Client” will give you all the instances of those users logging in via the VI Client. However, that string is only available for successful logins.
Failed VI Client login
Failed VI Client logins are also using similar strings to PowerCLI and vCLI failed logins. The difference here is that you can put just about any combination of username credentials in the login box and they will be passed without change, unlike the examples from PowerCLI.
What were they doing?
The Shell
When logging into the ESXi shell, all commands are logged to syslog. This is very handy for debugging what others may have done. Here’s two examples. The first is commands done by the root account and the second is commands done by the Active Directory user.
Root Account Commands
Active Directory Account Commands
In both examples above, the user logged in and, reading from the bottom up, executed the “dir” and “ls” commands. Then they changed their directory to /etc and displayed the /etc/shadow file. Clearly not doing something they should be doing!
The common string to search for here is “shell” but if you search for just that, you’ll get much more info than you need. Fortunately, Log Insight knows how to parse the logs. I clicked on “appname” as a search constraint with the value of “shell” and I was able to find all the logs. If you’re using another syslog solution, many have a similar capability.
appname search
Enriching your search
In addition, as I mentioned above, using AD credentials is a great way to see exactly who is logging in. You could see that in the examples where I was using “CORPAdministrator”. If nobody can generally log in as “root” and can only use AD credentials, you can search for the domain name (e.g. “CORP” & “@corp.local”), thereby enriching your searches. And if the “root” account password is vaulted yet “root” shows up as “logged in” from SSH or DCUI, then you know something is definitely fishy!
Built-in Web Server
Each ESXi server has a built-in web server to host access to the MOB (Managed Object Browser) and the Datastore Browser. I’m going to go thru the logs and see what we can find for the initial homepage, the MOB and the Datastore Browser.
ESXi homepage
When are people opening the ESXi homepage
To see when people are opening this webpage, you can search for “docroot”. With that you’ll see all the HTML elements and images that are loaded when that page opens.
MOB Login
This one is pretty cool, however, you’ll need at least “verbose” logging turned on for it to be useful. “Info” won’t work in this situation. Normally, I’d just search for “root” but instead, I found when “root” logged in (the last entry in the image above). You’ll see “Accepted password for user root from 127.0.0.1” The next entry is Hostd creating the session ID for “root”. In this example, it’s 25703b90. If I continue following that session ID up I’ll see the event where I see “homeUrl: /mob”.
Datastore Browsing
With datastore browsing, in order to get much more than just user logged in messages, you need to turn logging up to Trivia. As you can see above, we can now see the files that I have been browsing. In the example, I’m browsing host files. The search string I used was “Inspecting path” and that returned all the files that I was looking at. “Inspecting path” also will give you the browsing going on within the datastores via the web page.
What files were downloaded
For datastore files, like .VMX and .VMDK files, these also only show up when Trivia logging is enabled. In the example above, I used the search string of “Set OK response for GET /folder” and found that I had downloaded two VMX files.
Conclusion
In my examples I hope I’ve shown you that it’s relatively easy to get security-related information out of ESXI’s syslog output. One thing that bears repeating is that access to the ESXi shell should be limited to as few people as possible. ESXi is not a general purpose operating system and should not be treated as such.
To take that to the next level, ESXi (and vCenter and any other control plane interface) management should be done on a separate, controlled-access network to limit exposure to only those who should be there. If you have limits on access to your physical datacenter, it’s only right to do the same thing with your virtual datacenter!
There are no silver bullets in infrastructure security! Good security is about applying layers of defense (a.k.a. Defense in Depth). Unique identities, management/control plane isolation, logging, alerting, etc… All these methods, used together, compliment each other and will help you protect your infrastructure. If one fails, you have the others to cover you. There will always be new challenges but none I’ve see so far haven’t been covered by using these techniques and methods.
Thanks for reading and if you want me to blog about vSphere security stuff that you’re interested in reading about, drop me an email! I’m always looking for new topics to blog about.
mike
P.S. For more info on how to get the most out of ESXi logs…
Follow fellow VMware employee Steve Flanders on Twitter. He’s got some great stuff on his blog! Go check it out!
Also, you HAVE to follow my colleague William Lam. He has written extensively about ESXi logs. Follow him on Twitter for a wealth of vSphere knowledge!
