Author Archives: Carter Shanklin

San Diego VMUG October 2009 Recap

If you didn’t manage to make it to the San Diego VMUG you missed a pretty good show. Darin Pendergraft from Quest was there showing off EcoShell and I had cooked up a couple of demos of my own.

One of the more popular topics was a script I wrote to determine LUN latencies and the VMs that are writing to particular LUNs. One of the trickiest challenges when adopting virtualization is the effective use and partitioning of shared resources. Technologies like VMware DRS address exactly this challenge, but it’s safe to say that we’re still nearer the beginning than the end when it comes to automatic resource allocation and leveling.

On the other hand, PowerCLI and the vSphere API give you all the tools you need to monitor your environment, allowing you to proactively move or reallocate VMs away from storage hotspots. In particular I showed off a couple of scripts that tell you what your LUN read and write latencies are, as well as listing all VMs on a given LUN. The best part, the scripts are extremely simple, requiring less than 25 lines of code for both of these things! Don’t believe me? See for yourself:



You can easily adapt Get-VMHostLunLatency to run periodically (say, every night) to tell you if you’re experiencing any storage slowness. If you are you can easily follow that up with Get-LunVM to identify VMs that should be split off to separate LUNs. PowerCLI makes it all really simple.

Note that if you want to run Get-VMHostLunLatency against vCenter you will need to have your stats level set to 2 or higher.

As for the stuff I presented at the VMUG, I’ve posted my slides to slideshare:

And you can also download all the scripts I used here:

Come see PowerCLI at the San Diego VMUG, October 22 2009

If you’re in the San Diego area next week don’t miss the San Diego VMUG because I, your humble servant, will grace you with my presence and the vast stores of wisdom one only obtains through thousands of lonely nights in front of the computer.

I’ll be there to talk about – what else, PowerCLI and how it will help you automate your way to bliss.

As always I’ll be spending a lot of time on hands-on demos that will give you a first hand feel for what PowerCLI is and how it works. I’ll be covering reporting, provisioning and storage management, three of the most popular topics among PowerCLI users, with lots of new tips that are sure to save you lots of time. I’ll even be previewing a few new features from our upcoming release, so there’s something for everyone from beginner to expert.

Most people would call that a full day. BUT WAIT THERE”S MORE! You’ll also get the rare opportunity to see a presentation on the Virtualization Ecoshell. Now how much would you pay?

Hope to see you there.

When was the last time that VM was powered on?

People often want to know the last time that a VM was powered on. Unfortunately there is no completely fool-proof way to figure this out using just vCenter or ESX, but PowerCLI offers an approach that is pretty good, good enough that you can create a simple report and identify VMs that may be ready for that big hypervisor in the sky.

This is also a popular topic on the PowerCLI community, where LucD offers a solution to the problem, which relies on using events to determine when the poweron took place. Since Luc gave this solution a few things have happened:

  1. The bug that caused Get-VIEvent to only return 1000 events is fixed in PowerCLI 4.0.
  2. More importantly, you can pipe objects into Get-VIEvent, so you don’t have to deal with huge arrays of events. This is significant because a typical vCenter will have tens if not hundreds of thousands of events, covering everything from trivial to important things.

To take advantage of these facts I’ve written an updated PowerCLI script which generates a last-powered-on report.

One important note, ESX and vCenter actually create two events in connection with a VM poweron, one event when a user attempts to power a VM on, then another event if that power on actually succeeds. The code above actually generates a report of the last time a user attempted to power a VM on. When you attempt to power a VM on there are a lot of reasons it may fail, for instance your datastore may be full, in which case the poweron fails because ESX can’t create the memory swap file it needs (something I run into all the time).

Here’s a screen capture I made after running this against one of my ESX servers:

capture1

We can also use the always-useful, though often cryptic, select cmdlet to enhance this report by adding the total amount of space the VM uses, as follows.

capture3

This report suggests that the VM named 2k8 64 bit is a good candidate for removal, and in fact it is since I only used it to create this blog post, so it hasn’t been powered on since March 2009.

On the other hand you can see the report above is far from perfect, for example several powered on VMs have no entry for LastPoweron. Why? The reason is that, by default, ESX 4 only retains 1000 events (this is re-occurrence of the PowerCLI bug, I checked). Some of my VMs were powered on so long ago that their events no longer exist.

This brings us to another important point. ESX is not meant to store events for extended periods of time, instead vCenter will handle this for us. So the best way to generate this last powered on report is to run it against our vCenter instance, which will give us a much longer view.

Still this view does not stretch back forever, even in vCenter events will eventually expire. In vCenter 4, the event retention policy is even customizable. If you’re running vCenter 4, I’ve written this function to help you figure out what your retention policy is.

Here’s what it looks like in action:

capture2

If you know your retention policy and you know that all poweron/poweroff actions are performed through your vCenter server, you can be very confident that the information in the report is accurate, and use it to start moving unused VMs to archive storage, or even deleting them completely.

It is very important to note, though, that this report is pretty good, not perfect!! Don’t go around deleting stuff without careful human review beforehand.

How to run PowerCLI scripts from vCenter Alarms.

Background
vCenter provides an alarm feature which causes vCenter to automatically look for certain events or conditions within your virtual environment, and automatically take some action when conditions are met. There are quite a lot of criteria supported, you can generate alarms based on vCenter's event stream, so for example you could create an alarm any time a virtual machine is powered off. Alarms could also be created based on vCenter statistics, for example an alarm can be generated if an ESX host's memory utilization is above 90% for too long.

There are a number of actions you can take in response to an alarm. For example you can send email or generate an SNMP trap. You can also run scripts. A popular topic on the PowerCLI forum lately is how to run PowerCLI scripts in response to vCenter actions. Unfortunately the process is pretty tricky, so I've created some sample scripts and laid out a process that will hopefully give you everything you need to get going.

Note: These instructions were prepared based on vSphere 4.0. If you're using VirtualCenter, things may vary a bit (or maybe a lot), I haven't actually tried it. If you try this with VirtualCenter, please leave a note with your experiences.

Step 1: Within vCenter, decide on a well-defined location for your scripts.
Your life will be a lot easier if you put everything in one place, so you should decide where you want all your scripts and data to go. I recommend using a directory called %ALLUSERSPROFILE%\application data\VMware\VMware VirtualCenter\scripts. You'll have to create this directory but using this location has the advantage that your scripts will exist alongside a lot of other vCenter data.

One thing to note is that vCenter doesn't actually make use of environment variables, so henceforth I'm always going to use the location of C:\Documents and Settings\All Users\\application data\VMware\VMware VirtualCenter\scripts. Your drive letter and location may vary, so make the adjustment accordingly.

Step 2: Copy all the sample scripts into this location.
Getting the scripts just right is a tedious and painful process, so I've created some starter content that should make it a lot easier. After you've decided on your script location, copy all the files from the Alarms repository below into your directory.

After you've done that, your target directory will look a little something like this:

Vcenterfolder

Step 3: Decide on an authentication strategy.

If you don't want to log in to vCenter in response to an alarm, skip this section. However there's a good chance that when an alarm is fired, you want to do something to vCenter in response to it. Of course, you'll need to authenticate to vCenter to do that. PowerCLI supports SSPI and single-sign-on, but chances are that won't help you here. The reason is that vCenter runs alarm scripts using the same account that is used for the vCenter service. By default this is the local system account, and a script running as local system won't be able to use SSPI to log in the vCenter.

There are 3 ways to deal with this:

  1. Code usernames and passwords into your scripts. This is the easiest and least secure approach.
  2. Change the vCenter service to use an actual domain login. To be honest I'm not sure if this is even possible, but if it is it will be difficult to use this approach if you're not already using it. On the other hand this approach would allow use of SSPI and would not require any ongoing maintenance.
  3. Store encrypted credentials for the local system account to use when running the script. This approach is much more secure than the first approach yet is not nearly as difficult as the third approach. It's also the approach I took, so I'll spend a bit of time discussing how to do it.

Approach 3 relies on the fact that PowerShell makes it pretty easy to deal with the Windows Data Protection API. This allows us to store a credential in a file in such a way as it can only be decrypted by someone who has logged in using the same identity that was used to export the credential. If you look at the screenshot above, there's a file called systemCredentials.enc.xml. This is my exported credential.

The trick is that the alarm script is going to be running as local system, so to make it possible to read this credential, I have to export this credential as local system. If I export it as Administrator, the alarm script won't be able to use it. This means that somehow I have to run PowerShell as local system. There are tricks for starting command prompt as local system, if you're running pre-Vista it's pretty simple to launch one using the "at" command. If you're on Windows 2008 you can use the psexec utility (also described in the same link).

Once you manage to get a local system command prompt, launch PowerShell and navigat to your script directory. Then "dot source" the credentialManagement.ps1 file and run Export-PSCredential. This brings up a prompt for credentials, so type in the user name and password you want to use against vCenter. This will cache your credential and you're ready to start running alarm scripts securely. Check your directory against my screenshot above to ensure that things appear to be in order.

. .\credentialManagement.ps1

Export-PSCredential -path systemCredentials.enc.xml

The downside to this approach is that if your vCenter password ever changes, you need to go in and re-cache the credential.

The other thing to note is that my scripts assumes you use this approach. If you don't, you'll have to edit each of them accordingly.

Step 4: Associate an alarm with a sample script.
You may think that from here it's just as simple as associating an alarm with your favorite .ps1 file. Unfortunately no, vCenter will only run batch files. So we need to create an intermediate batch file that will call our PowerShell file. This is the purpose of redirect.bat, we can use this batch file to call any PowerShell file we want. Since redirect.bat takes a script as an argument, we avoid the situation of having one batch file per PowerShell file.

Let's walk through the whole process of associating an alarm with a PowerCLI script. We'll create an alarm that fires any time a host's CPU utilizatoin is less than 75% and launches a script that will create a new VM on this not-busy-enough host. Of course this is a pretty stupid thing to do, but it's a good way to prove that everything is working as it should be.

First, let's create an alarm. If you've never created an alarm you may have a hard time figuring out how to actually do it. To do it, go to the very top of your inventory tree and right click on the root node, then select Alarm > Add Alarm

Createanalarm

There are 4 tabs to fill out with the new alarm. Until you master all of this stuff I recommend just copying what I've got here.

In the first tab, within that Alarm Type group, select Hosts under the Monitor pulldown.

Screen1

In the Triggers tab, add a trigger based on Host CPU Usage (%). Set the condition to "Is below". Set the condition length for both warning and alert to 30 seconds.

Screen2

In the Reporting tab change "Repeat triggered alarm every" to 1 minute.

Screen3

In the Actions tab, within the Frequency group change "Repeat actions every" to 1 minute. Then create an action. For the action type select "Run a command". On the right-hand side there are 4 pulldowns based on the state transitions. Set the first 3 of these to Repeat. Now comes the important part: within Configuration you need to enter the path to the redirect batch script, giving it an argument of the PowerShell script to run. For our purposes enter:

"C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\scripts\redirect.bat" "C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\scripts\sampleHostAlarm.ps1"

This extremely long line needs to go in without spaces or linefeeds. If you're using a different directory from the one I suggested, you'll also need to adjust that here.

Screen4

Now we're ready to go. If you've done everything right, all of your not-too-busy hosts will start getting new VMs in less than a minute. That will continue until you disable or delete the alarm, so don't let it run too long. You can modify or delete an alarm by clicking the Alarms tab.

Step 5: Decide if you want to play it safe.
Your scripts may never exit. There are thousands of reasons why, it may enter an infinite loop, it may sit waiting for input, who knows? When vCenter runs a script, it times the operation and kills it after 5 minutes. Unfortunately it only kills the command prompt that it launches and not any child processes the command prompt may have created, so in reality vCenter is unable to reign in any runaway scripts.

One way we can cope with that is: before running any alarm script, check for old leftover scripts and kill them. If you want to play it safe and do that for yourself, I've provided a script called alarmCleanup that will do it. To use it, all you have to do is uncomment the relevant line in redirect.bat and you'll have extra protection against runaway scripts.

Step 6: Write your own scripts, using the samples as a guide.

I've provided 3 sample scripts to get you going, but there are two things worth pointing out that are not really obvious as you start writing your own scripts.

First, when your script is run, several environment variables will be defined, as this table shows.

Environment
Variable Name

Sample
Value

VMWARE_ALARM_TRIGGERINGSUMMARY

Metric Usage = 0%                              

VMWARE_ALARM_DECLARINGSUMMARY

([Yellow metric Is below 75%; Red metric Is b…

VMWARE_ALARM_OLDSTATUS

Green                                          

VMWARE_ALARM_TARGET_NAME

192.168.1.11                                   

VMWARE_ALARM_ALARMVALUE

Current values for metric/state                

VMWARE_ALARM_TARGET_ID

host-1560                                      

VMWARE_ALARM_NAME

New Alarm                                      

VMWARE_ALARM_NEWSTATUS

Red                                             

VMWARE_ALARM_EVENTDESCRIPTION

Alarm 'New Alarm' on 192.168.1.11 changed fro…

VMWARE_ALARM_ID

alarm-46                                       

If you want to get the object that actually triggered the alarm, the best thing to use is the ID. Most PowerCLI Get cmdlets support a -ID argument that lets you load objects by ID. However, the ID that PowerCLI cmdlets use are a bit different from the IDs that will be set in the environment variable. For example, in the table above we have an id of host-1560. However, we can't call Get-VMHost -ID host-1560 and expect it to work, instead we have to prepend the term HostSystem- to it. In other words we would run:

    Get-VMHost -Id HostSystem-host-1560

to get the host that triggered the alarm. The same is true of other objects, and you can see a table of prefixes below.

When
dealing with this:

Prepend
this to the ID

Virtual Machines

VirtualMachine

Hosts

HostSystem

Clusters

ClusterComputeResource

Datacenters

Datacenter

Datastores

Datastore

Since alarms can only be associated with one type of object, you can hard-code this string inside the script that will respond to the alarm. Again there are samples to get you started in the right direction.

Troubleshooting.
Unfortunately this process is pretty tricky and you're probably not going to get it right the first time. If you need to diagnose what's going on, I recommend downloading Process Explorer.Some of the key things to note, when the alarm fires you should see a new process spawn as a child of vpxd.exe, then quickly disassociate from it (this is due to the use of "start" within redirect.bat).

Procexp

Another important thing to note, if your PowerShell process isn't behaving properly, right-click on it and take a look at its environment and command line.

Procexp2

Expect the process of getting alarm scripts to take a bit of experimentation, but once you've got the first one working the rest is very easy. Good luck!

Onyx Update

I've gotten a lot of questions around when people will be able to get their hands on Onyx. We collected about 50 names at VMworld, so we're very happy to see all the different people who are interested. We haven't forgotten about you, we haven't changed our minds, we're getting things in order to preview Onyx and invites will be going out soon.

Thanks for your patience!

What is “Onyx”? Visit us at VMworld and get a chance to find out.

Sorry, no clippy. Do you enjoy reading huge piles of API documentation? Do you use wireshark to reverse-engineer wire protocols? Do you like using strace to figure out what the heck a program is doing?

Me neither, which is why I’m really excited about a little project we’ve been working on called “Project Onyx”. What exactly is Onyx? Quite simply, Onyx converts mouse clicks into PowerCLI scripts. Sort of like Excel’s macro record, except it’s for vSphere and doesn’t come with a stupid paper clip.

Does this sound like a tool you could use? We’re looking for a handful of dedicated people with a burning need for automation to help us get Onyx really solid. I would probably describe the tool as somewhere between alpha and pre-alpha and it’s a bit hard to set up at the moment, so we need people who are dedicated enough to work through the tough initial phases. Still it really is worth it. I’ve been working with vSphere API for about 3 years now and even I learned a lot by just toying around with Onyx.

So if you want to give Onyx a spin, here’s the deal. Come by session VM2241 at VMworld next week, and bring your business card. During the session we’ll be showing how to take scripts that Onyx generates and turn them into reusable bits of PowerShell code, that can be used in the pipeline with all the PowerCLI commands you know and love. There are two sessions, one on Tuesday at Noon and one on Tuesday at 5:30. Give us your card and we’ll pick 5 names from each session and give you first dibs on the raw power of Onyx. Enjoy the teaser video and hope to see you next week.

VMworld 2009: PowerCLI and PowerShell roundup.

There’s a lot of stuff related to PowerCLI and PowerShell going on at VMworld, so it really deserves a summary of everything. It’s not enough to rename the event to PowerCLIWorld 2009 (maybe next year) but it’s still a pretty big list. Each of the sessions below either talks about PowerCLI directly and in depth, or introduces a new PowerShell tool.

  1. VM2241 – Managing vSphere with PowerCLI
  2. TA2650 – Take PowerCLI to the next level
  3. DV2507 – VMware View – Upcoming Attractions
  4. EA2526 – Sharepoint vSphere Best Practices
  5. TA3521 — vNetwork Monitoring/Troubleshooting Demystification and Ecosystem Tool Demonstration

So if you’re looking to get more automated, scale out your existing environment, or just save a lot of time, be sure to check all these sessions out. Visit vmworld.com to add them to your schedule.

How to list datastores and VMs that are on shared storage.

A lot of the magic VMware provides relies on shared storage. If a VM is on local storage, you can’t VMotion it, HA won’t restart it, and a host of other limitations. Do you know which of your VMs, if any, are on local storage? This script can make answering that question pretty easy. First, here’s a function that will locate datastores that use shared storage.

To use, launch PowerCLI, connect to your vCenter server, load the function above and run:

And you’ll wind up with a list of datastores that your hosts can share.

But that’s not what we’re after, we want datastores that can’t be shared. Not too hard,


Once we have $localOnly, we pass that to Get-VM.

This gives us the list of VMs that can’t be shared amongst hosts.

Trying to dig into PowerCLI but don’t know PowerShell? Start here.

A lot of people see potential with PowerCLI, but struggle to get started
because PowerShell is a fairly new technology and not too many people know it
yet. If you’re getting started, here are my recommendations.

If you’re getting your hands on PowerShell for the very first time, I’d
recommend working through the first 3 or 4 sections of the PowerShell
owner’s manual
, this will get you to the point where you can launch
PowerShell and are ready to run some commands.

After that, to master PowerShell you’ll have to master these two separate
areas:

  1. The PowerShell language.
  2. PowerShell cmdlets.

Know how to get help.

First, if you’re an absolute beginner, start by mastering PowerShell help.
Spending a few hours here will save you a lot of time later on. PowerShell
cmdlets are self-documenting, so the online help is accurate and definitive. It
also shows you critical facts about cmdlets, such as what arguments they accept,
what input types are needed, and what can be piped in to the cmdlet. There’s a
great intro on using help over at concentratedtech, and
you should also check out MSDN.
Use these guides, play around with the cmdlets, then try these same
principles with a few cmdlets of your own, you won’t regret it.

The key things to remember are “help <cmdlet name> –full” to get all
information about a particular cmdlet (example “help get-vm”) or “help
<cmdlet name> –examples” to get just the examples. One of the great things
about PowerShell is that help cmdlet –examples almost always yields a useful
example or two.

This works great for cmdlets, but unfortunately the situation is not so
simple for things that are part of the PowerShell language itself, like the if
statement or comparison operators. We’ll talk about how to get a handle on those
later.

The top 25 cmdlets worth knowing.

Anyway there are a lot of cmdlets and only so much time in the day. You may
be wondering where to focus your attention. There’s a great online of repository
of PowerShell code over at poshcode.org.
Recently I analyzed all the code on poshcode to count the frequencies of various
cmdlets and operators.

So here are the top 25 cmdlets used by code over at This sample is drawn from a lot of different authors all trying to accomplish different goals, so it's a good cross section of the most useful cmdlets. Note however that I would recommend that beginners not learn add-member until later, even though it is obviously a very useful cmdlet for experts.

Poshcode.org most popular cmdlets.

Rank

Cmdlet
Name

Cmdlet
Frequency

1

add-member(*)

4862

2

foreach-object

4039

3

write-host

3696

4

write-output

2494

5

new-object

2139

6

out-file

1669

7

where-object

1550

8

sort-object

1473

9

get-content

1399

10

get-childitem

1187

11

get-command

805

12

select-object

754

13

read-host

693

14

get-history

693

15

set-variable

682

16

write-debug

657

17

invoke-expression

576

18

test-path

501

19

split-path

471

20

get-process

436

21

add-content

408

22

get-wmiobject

383

23

get-member

379

24

start-process

370

25

write-verbose

354

(*) This cmdlet ranks high due to the nature of the code on
poshcode.org, it’s a useful cmdlet to know but you should wait until you’re an
expert to really dig in to it.

If you’re starting out, you should begin by mastering most of these cmdlets
(you should skip add-member until you’ve become a master in your own right).
Remember, each of these has samples within its help that will get you on your
way. There’s one more cmdlet that VMware admins in particular get a lot of use
from, namely Measure-Object. If you’re mainly going to use PowerShell to manage
VMware, that one should be in your top 10 because our users use this cmdlet a
lot.

Operators and other language features.

As I mentioned, you’ll also need to master the PowerShell language. This
includes learning how to branch using if/then/else, how to build loops with
while and foreach, etc. For example, consider this code:

 So how, exactly, do if and else work? Help is available for all these things,
but in some cases it’s hard to track down. PowerShell’s help contains a lot of
topics in the form of about_something. For example, there is an
about_if, which talks about both the if and else statements. You can see all of
them by typing “help *about*”. The other thing to note in the statement above is
the use of “-eq”. This is a so-called comparison operator, and mastering these
is very important. Type “help about_comparison_operators” to learn all about
these. Also, here are the top comparison operators as found on poshcode.org:

 Poshcode.org most popular operators.

Rank

Operator

Frequency

1

-eq

4480

2

-gt

1329

3

-and

1320

4

-match

924

5

-ne

775

6

-contains

582

7

-not

514

8

-lt

506

9

-or

480

Lastly but not leastly, you’ll need to master the pipe. This tends to be
pretty easy because once you start piping it will seem very natural to you. Just
for reference, our trusty vertical friend appears 15491 times.

Getting really serious.

Another great little tutorial is the Top 10 Tips for Using Windows PowerShell. But if you’re really serious about upgrading your PowerShell skills,
you should consider getting a hold of either Hal Rotterberg’s Managing VMware Infrastructure
with Windows PowerShell TFM
and Bruce Payette’s Windows
PowerShell in Action
. Bruce’s book is the best from a theory perspective,
while Hal’s book lets you learn while you improve the way you manage VMware.

If you’re having trouble and need more help, don’t forget that help is
available in the PowerCLI
community
.

What to do when you can’t find what you’re looking for.

Ever had a problem to solve but not sure how to proceed? This little function might help you out, it searches through all available PowerShell commands looking for a keyword you specify, and will search the full help text, including the examples.

 

For example, let’s say that in PowerCLI you’re trying to set up or diagnose VMotion, but not sure what cmdlets might help. All you’ve got to do is load this function, then run Search-Help vmotion, which will return this list.

CommandType     Name                            Definition
———–     —-                            ———-
Cmdlet          Move-VM                         Move-VM [[-VM] <VirtualMachi…
Cmdlet          New-Cluster                     New-Cluster [-Name] <String>…
Cmdlet          New-VMHostNetworkAdapter        New-VMHostNetworkAdapter [[-…
Cmdlet          Set-Cluster                     Set-Cluster [[-Cluster] <Clu…
Cmdlet          Set-VMHost                      Set-VMHost [[-State] <Nullab…
Cmdlet          Set-VMHostNetworkAdapter        Set-VMHostNetworkAdapter -Ph…

You can then run help on any of the results to drill down further.