How to run PowerCLI scripts from vCenter Alarms.

Background
vCenter provides an alarm feature which causes vCenter to automatically look for certain events or conditions within your virtual environment, and automatically take some action when conditions are met. There are quite a lot of criteria supported, you can generate alarms based on vCenter's event stream, so for example you could create an alarm any time a virtual machine is powered off. Alarms could also be created based on vCenter statistics, for example an alarm can be generated if an ESX host's memory utilization is above 90% for too long.

There are a number of actions you can take in response to an alarm. For example you can send email or generate an SNMP trap. You can also run scripts. A popular topic on the PowerCLI forum lately is how to run PowerCLI scripts in response to vCenter actions. Unfortunately the process is pretty tricky, so I've created some sample scripts and laid out a process that will hopefully give you everything you need to get going.

Note: These instructions were prepared based on vSphere 4.0. If you're using VirtualCenter, things may vary a bit (or maybe a lot), I haven't actually tried it. If you try this with VirtualCenter, please leave a note with your experiences.

Step 1: Within vCenter, decide on a well-defined location for your scripts.
Your life will be a lot easier if you put everything in one place, so you should decide where you want all your scripts and data to go. I recommend using a directory called %ALLUSERSPROFILE%\application data\VMware\VMware VirtualCenter\scripts. You'll have to create this directory but using this location has the advantage that your scripts will exist alongside a lot of other vCenter data.

One thing to note is that vCenter doesn't actually make use of environment variables, so henceforth I'm always going to use the location of C:\Documents and Settings\All Users\\application data\VMware\VMware VirtualCenter\scripts. Your drive letter and location may vary, so make the adjustment accordingly.

Step 2: Copy all the sample scripts into this location.
Getting the scripts just right is a tedious and painful process, so I've created some starter content that should make it a lot easier. After you've decided on your script location, copy all the files from the Alarms repository below into your directory.

After you've done that, your target directory will look a little something like this:

Step 3: Decide on an authentication strategy.

If you don't want to log in to vCenter in response to an alarm, skip this section. However there's a good chance that when an alarm is fired, you want to do something to vCenter in response to it. Of course, you'll need to authenticate to vCenter to do that. PowerCLI supports SSPI and single-sign-on, but chances are that won't help you here. The reason is that vCenter runs alarm scripts using the same account that is used for the vCenter service. By default this is the local system account, and a script running as local system won't be able to use SSPI to log in the vCenter.

There are 3 ways to deal with this:

1. Code usernames and passwords into your scripts. This is the easiest and least secure approach.
2. Change the vCenter service to use an actual domain login. To be honest I'm not sure if this is even possible, but if it is it will be difficult to use this approach if you're not already using it. On the other hand this approach would allow use of SSPI and would not require any ongoing maintenance.
3. Store encrypted credentials for the local system account to use when running the script. This approach is much more secure than the first approach yet is not nearly as difficult as the third approach. It's also the approach I took, so I'll spend a bit of time discussing how to do it.

Approach 3 relies on the fact that PowerShell makes it pretty easy to deal with the Windows Data Protection API. This allows us to store a credential in a file in such a way as it can only be decrypted by someone who has logged in using the same identity that was used to export the credential. If you look at the screenshot above, there's a file called systemCredentials.enc.xml. This is my exported credential.

The trick is that the alarm script is going to be running as local system, so to make it possible to read this credential, I have to export this credential as local system. If I export it as Administrator, the alarm script won't be able to use it. This means that somehow I have to run PowerShell as local system. There are tricks for starting command prompt as local system, if you're running pre-Vista it's pretty simple to launch one using the "at" command. If you're on Windows 2008 you can use the psexec utility (also described in the same link).

Once you manage to get a local system command prompt, launch PowerShell and navigat to your script directory. Then "dot source" the credentialManagement.ps1 file and run Export-PSCredential. This brings up a prompt for credentials, so type in the user name and password you want to use against vCenter. This will cache your credential and you're ready to start running alarm scripts securely. Check your directory against my screenshot above to ensure that things appear to be in order.

. .\credentialManagement.ps1

Export-PSCredential -path systemCredentials.enc.xml

The downside to this approach is that if your vCenter password ever changes, you need to go in and re-cache the credential.

The other thing to note is that my scripts assumes you use this approach. If you don't, you'll have to edit each of them accordingly.

Step 4: Associate an alarm with a sample script.
You may think that from here it's just as simple as associating an alarm with your favorite .ps1 file. Unfortunately no, vCenter will only run batch files. So we need to create an intermediate batch file that will call our PowerShell file. This is the purpose of redirect.bat, we can use this batch file to call any PowerShell file we want. Since redirect.bat takes a script as an argument, we avoid the situation of having one batch file per PowerShell file.

Let's walk through the whole process of associating an alarm with a PowerCLI script. We'll create an alarm that fires any time a host's CPU utilizatoin is less than 75% and launches a script that will create a new VM on this not-busy-enough host. Of course this is a pretty stupid thing to do, but it's a good way to prove that everything is working as it should be.

First, let's create an alarm. If you've never created an alarm you may have a hard time figuring out how to actually do it. To do it, go to the very top of your inventory tree and right click on the root node, then select Alarm > Add Alarm

There are 4 tabs to fill out with the new alarm. Until you master all of this stuff I recommend just copying what I've got here.

In the first tab, within that Alarm Type group, select Hosts under the Monitor pulldown.

In the Triggers tab, add a trigger based on Host CPU Usage (%). Set the condition to "Is below". Set the condition length for both warning and alert to 30 seconds.

In the Reporting tab change "Repeat triggered alarm every" to 1 minute.

In the Actions tab, within the Frequency group change "Repeat actions every" to 1 minute. Then create an action. For the action type select "Run a command". On the right-hand side there are 4 pulldowns based on the state transitions. Set the first 3 of these to Repeat. Now comes the important part: within Configuration you need to enter the path to the redirect batch script, giving it an argument of the PowerShell script to run. For our purposes enter:

"C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\scripts\redirect.bat" "C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\scripts\sampleHostAlarm.ps1"

This extremely long line needs to go in without spaces or linefeeds. If you're using a different directory from the one I suggested, you'll also need to adjust that here.

Now we're ready to go. If you've done everything right, all of your not-too-busy hosts will start getting new VMs in less than a minute. That will continue until you disable or delete the alarm, so don't let it run too long. You can modify or delete an alarm by clicking the Alarms tab.

Step 5: Decide if you want to play it safe.
Your scripts may never exit. There are thousands of reasons why, it may enter an infinite loop, it may sit waiting for input, who knows? When vCenter runs a script, it times the operation and kills it after 5 minutes. Unfortunately it only kills the command prompt that it launches and not any child processes the command prompt may have created, so in reality vCenter is unable to reign in any runaway scripts.

One way we can cope with that is: before running any alarm script, check for old leftover scripts and kill them. If you want to play it safe and do that for yourself, I've provided a script called alarmCleanup that will do it. To use it, all you have to do is uncomment the relevant line in redirect.bat and you'll have extra protection against runaway scripts.

Step 6: Write your own scripts, using the samples as a guide.
I've provided 3 sample scripts to get you going, but there are two things worth pointing out that are not really obvious as you start writing your own scripts.

First, when your script is run, several environment variables will be defined, as this table shows.

If you want to get the object that actually triggered the alarm, the best thing to use is the ID. Most PowerCLI Get cmdlets support a -ID argument that lets you load objects by ID. However, the ID that PowerCLI cmdlets use are a bit different from the IDs that will be set in the environment variable. For example, in the table above we have an id of host-1560. However, we can't call Get-VMHost -ID host-1560 and expect it to work, instead we have to prepend the term HostSystem- to it. In other words we would run:

    Get-VMHost -Id HostSystem-host-1560

to get the host that triggered the alarm. The same is true of other objects, and you can see a table of prefixes below.

Since alarms can only be associated with one type of object, you can hard-code this string inside the script that will respond to the alarm. Again there are samples to get you started in the right direction.

Troubleshooting.
Unfortunately this process is pretty tricky and you're probably not going to get it right the first time. If you need to diagnose what's going on, I recommend downloading Process Explorer.Some of the key things to note, when the alarm fires you should see a new process spawn as a child of vpxd.exe, then quickly disassociate from it (this is due to the use of "start" within redirect.bat).

Another important thing to note, if your PowerShell process isn't behaving properly, right-click on it and take a look at its environment and command line.

Expect the process of getting alarm scripts to take a bit of experimentation, but once you've got the first one working the rest is very easy. Good luck!