Home > Blogs > VMware PowerCLI Blog


Waiting for OS customization to complete

image
Posted by
Vitali Baruh
PowerCLI QE team

Provisioning of new virtual machines often includes Operating System (OS) customization.

A typical provisioning workflow could look like this:

1.  Clone a virtual machine from a template and specify a customization specification

2.  Start the VM. This triggers the customization process

3.  Wait for the customization to complete

4.  Perform additional setup steps

 

Step three is quite tricky, because there’s no easy way to find out when the customization has completed within the guest operating system and if it has succeeded or has failed.

This step could be implemented in an optimistic manner with sleep for 10 minutes, but it could also be implemented in a more robust and informative way. To do this, we could create a script with the following specification:

Input

  • List of virtual machines which should be monitored for the completion of a customization process
  • Timeout in seconds that the script should wait for the customization to end

Output

  • List of PSObjects where each object will hold a passed VM and its customization status – successful, failed, started, et.

The function is based on several virtual machine events defined by vCenter Server:

  • VmStartingEvent – posted when VM is powered on
  • CustomizationStartedEvent – posted when customization is started
  • CustomizationSucceeded – posted when customization succeeds
  • CustomizationFailed – posted when customization fails

Our script workflow is pretty simple:

  • 1. For each VM we will look for the last VmStarting event, because the customization process starts after the VM has been powered on
    • 1.1. If such an event is found, change the status to “CustomizationNotStarted”
    • 1.2. If such an event is not found, change the status to “VmNotStarted”
  • 2. Start a loop
    • 2.1. For each VM with status “CustomizationNotStarted”
      • 2.1.1. Check for posted CustomizationStartedEvent after the last power-on operation
        • 2.1.1.1. If such an event is found, change the status to “CustomizationStarted”
    • 2.2. For each VM with status “CustomizationStarted”
      • 2.2.1. Check for Succeded or Failed Event
        • 2.2.1.1. If such an event is found, change the status to the corresponding “CustomizationSucceeded” or “CustomizationFailed” values
    • 2.3. Check is there more VMs with status “CustomizationNotStarted” or “CustomizationStarted” (we should continue to monitor these VMs)
      • 2.3.1. If no, break the loop
    • 2.4. Check whether the specified timeout has elapsed
      • 2.4.1. If yes, break the loop
  • 3. Return result

Now let see the script in action!

Step 1 – retrieve template, OS customization spec, target vmhost and target datastore.

clip_image002

As result we have several variables that are needed to start cloning with customization tasks.

Step 2 – start clone operations for two VMs

clip_image004

As result we have $vms variable holding our virtual machines.

Step 3 – start virtual machines, which triggers customization process and monitor it with our script.

The script is started with two virtual machines and also we specify to wait maximum 600 seconds for OS customization to complete.

clip_image006

After the start the script reports each VM which will be monitored.

The next screen shows that after about 45 second customization start events have been posted for both VMs.

clip_image007

The last screen shot shows the moments when the two customization succeed events have been posted and also the result of the script.

clip_image009

Finally the $result variable contains two custom objects. Each object contains VM and CustomizationStatus properties.

Our script could iterate over the $result variable and to perform some additional setup actions – installing software not included in the template, applying specific software configurations, etc.

Of course for these additional setup actions we can use the Invoke-VMScript and Copy-VMGuestFile cmdlets.

Now let’s see the code of WaitVmCustomization.ps1 script:

<#

.SYNOPSIS
Waits customization process for list virtual machines to completes.

.DESCRIPTION
Waits customization process for list virtual machines to completes.
The script returns if customization process ends for all virtual machines or if the specified timeout elapses.
The script returns PSObject for each specified VM.
The output object has VM and CustomizationStatus properties.

.EXAMPLE
$vm = 1..10 | foreach { New-VM -Template WindowsXPTemplate -OSCustomizationSpec WindowsXPCustomizaionSpec -Name "winxp-$_" }
.\WaitVmCustomization.ps1 -vmList $vm -timeoutSeconds 600

.NOTES
The script is based on sveral vCenter events.
* VmStarting event – this event is posted on power on operation
* CustomizationStartedEvent event – this event is posted for VM when customiztion has started
* CustomizationSucceeded event – this event is posted for VM when customization has successfully completed
* CustomizationFailed – this event is posted for VM when customization has failed

Possible CustomizationStatus values are:
* "VmNotStarted" – if it was not found VmStarting event for specific VM.
* "CustomizationNotStarted" – if it was not found CustomizationStarterdEvent for specific VM.
* "CustomizationStarted" – CustomizationStartedEvent was found, but Succeeded or Failed event were not found
* "CustomizationSucceeded" – CustomizationSucceeded event was found for this VM
* "CustomizationFailed" – CustomizationFailed event wass found for this VM

#>
[CmdletBinding()]
param(
   # VMs to monitor for OS customization completion
   [Parameter(Mandatory=$true)]
   [ValidateNotNullOrEmpty()]
   [VMware.VimAutomation.ViCore.Types.V1.Inventory.VirtualMachine[]] $vmList,
  
   # timeout in seconds to wait
   [int] $timeoutSeconds = 600
)

# constants for status
      $STATUS_VM_NOT_STARTED = "VmNotStarted"
      $STATUS_CUSTOMIZATION_NOT_STARTED = "CustomizationNotStarted"
      $STATUS_STARTED = "CustomizationStarted"
      $STATUS_SUCCEEDED = "CustomizationSucceeded"
      $STATUS_FAILED = "CustomizationFailed"
     
      $STATUS_NOT_COMPLETED_LIST = @( $STATUS_CUSTOMIZATION_NOT_STARTED, $STATUS_STARTED )
     
# constants for event types     
      $EVENT_TYPE_CUSTOMIZATION_STARTED = "VMware.Vim.CustomizationStartedEvent"
      $EVENT_TYPE_CUSTOMIZATION_SUCCEEDED = "VMware.Vim.CustomizationSucceeded"
      $EVENT_TYPE_CUSTOMIZATION_FAILED = "VMware.Vim.CustomizationFailed"
      $EVENT_TYPE_VM_START = "VMware.Vim.VmStartingEvent"

# seconds to sleep before next loop iteration
      $WAIT_INTERVAL_SECONDS = 15
     
function main($vmList, $timeoutSeconds) {
   # the moment in which the script has started
   # the maximum time to wait is measured from this moment
   $startTime = Get-Date
  
   # we will check for "start vm" events 5 minutes before current moment
   $startTimeEventFilter = $startTime.AddMinutes(-5)
  
   # initializing list of helper objects
   # each object holds VM, customization status and the last VmStarting event
   $vmDescriptors = New-Object System.Collections.ArrayList
   foreach($vm in $vmList) {
      Write-Host "Start monitoring customization process for vm '$vm'"
      $obj = "" | select VM,CustomizationStatus,StartVMEvent
      $obj.VM = $vm
      # getting all events for the $vm,
      #  filter them by type,
      #  sort them by CreatedTime,
      #  get the last one
      $obj.StartVMEvent = Get-VIEvent -Entity $vm -Start $startTimeEventFilter |
         where { $_ -is $EVENT_TYPE_VM_START } |
         Sort CreatedTime |
         Select -Last 1
        
      if (-not $obj.StartVMEvent) {
         $obj.CustomizationStatus = $STATUS_VM_NOT_STARTED
      } else {
         $obj.CustomizationStatus = $STATUS_CUSTOMIZATION_NOT_STARTED
      }
     
      [void]($vmDescriptors.Add($obj))
   }        
  
   # declaring script block which will evaulate whether
   # to continue waiting for customization status update
   $shouldContinue = {
      # is there more virtual machines to wait for customization status update
      # we should wait for VMs with status $STATUS_STARTED or $STATUS_CUSTOMIZATION_NOT_STARTED
      $notCompletedVms = $vmDescriptors |

         where { $STATUS_NOT_COMPLETED_LIST -contains $_.CustomizationStatus }

      # evaulating the time that has elapsed since the script is running
      $currentTime = Get-Date
      $timeElapsed = $currentTime – $startTime
     
      $timoutNotElapsed = ($timeElapsed.TotalSeconds -lt $timeoutSeconds)
     
      # returns $true if there are more virtual machines to monitor
      # and the timeout is not elapsed
      return ( ($notCompletedVms -ne $null) -and ($timoutNotElapsed) )
   }
     
   while (& $shouldContinue) {
      foreach ($vmItem in $vmDescriptors) {
         $vmName = $vmItem.VM.Name
         switch ($vmItem.CustomizationStatus) {
            $STATUS_CUSTOMIZATION_NOT_STARTED {
               # we should check for customization started event
               $vmEvents = Get-VIEvent -Entity $vmItem.VM -Start $vmItem.StartVMEvent.CreatedTime
               $startEvent = $vmEvents | where { $_ -is $EVENT_TYPE_CUSTOMIZATION_STARTED }
               if ($startEvent) {
                  $vmItem.CustomizationStatus = $STATUS_STARTED
                  Write-Host "Customization for VM '$vmName' has started"
               }
               break;
            }
            $STATUS_STARTED {
               # we should check for customization succeeded or failed event
               $vmEvents = Get-VIEvent -Entity $vmItem.VM -Start $vmItem.StartVMEvent.CreatedTime
               $succeedEvent = $vmEvents | where { $_ -is $EVENT_TYPE_CUSTOMIZATION_SUCCEEDED }
               $failedEvent = $vmEvents | where { $_ -is $EVENT_TYPE_CUSTOMIZATION_FAILED }
               if ($succeedEvent) {
                  $vmItem.CustomizationStatus = $STATUS_SUCCEEDED
                  Write-Host "Customization for VM '$vmName' has successfully completed"
               }
               if ($failedEvent) {
                  $vmItem.CustomizationStatus = $STATUS_FAILED
                  Write-Host "Customization for VM '$vmName' has failed"
               }
               break;
            }
            default {
               # in all other cases there is nothing to do
               #    $STATUS_VM_NOT_STARTED -> if VM is not started, there's no point to look for customization events
               #    $STATUS_SUCCEEDED -> customization is already succeeded
               #    $STATUS_FAILED -> customization
               break;
            }
         } # enf of switch
      } # end of the freach loop
     
      Write-Host "Sleeping for $WAIT_INTERVAL_SECONDS seconds"
      Sleep $WAIT_INTERVAL_SECONDS
   } # end of while loop
  
   # preparing result, without the helper column StartVMEvent
   $result = $vmDescriptors | select VM,CustomizationStatus
   return $result
}

#calling the main function
main $vmList $timeoutSeconds

And here is the code for the scenario script:

# prepare necesary objects for clone operations
$template = Get-Template WindowsXPTemplate
$spec = Get-OSCustomizationSpec WindowsXPSpec
$vmhost = Get-VMHost | select -First 1
$datastore = Get-Datastore -VMHost $vmhost -Name Storage1

# Start cloning operations
$vms = 1..2 | % { New-VM -Name "WinXP-$_" -ResourcePool $vmhost -Template $template -Datastore $datastore -OSCustomizationSpec $spec }

# Trigger customization and wait its completion
$vms = $vms | Start-VM
$result = .\WaitVmCustomization.ps1 -vmList $vms -timeoutSeconds 600

Several notes about the script

  • It doesn’t accept pipeline input, because it checks the customization status for multiple virtual machines simultaneously. This won’t be possible if the process block of the script is executed for each object passed by the pipeline.
  • It also shows how to work with specific type of events.
  • Its VIEvent searches are optimized by specifying concrete entity and start time filters

Hope that this blog post is interesting and useful.

More about Vitali Baruh…

Vitali joined VMware and PowerCLI team in the beginning of 2009. He is member of the quality engineering part of the team and his main role is the functional verification of the vSphere, vCloud and License PowerCLI components.

As all members of the team he is working to deliver a good and valuable product. He is also working to improve all processes and tools involved in product development and validation.

This entry was posted in Uncategorized on by .
Alan Renouf

About Alan Renouf

Alan Renouf is a Product Line Manager at VMware focusing on API's, SDK's and CLI's, He is responsible for providing the architects and operators of private and public cloud infrastructure with the toolkits/frameworks and command-line interfaces they require to build a fully automated software-defined datacenter. Alan is a frequent blogger at http://blogs.vmware.com/PowerCLI a book author and has a personal blog at http://virtu-al.net. You can follow Alan on twitter as @alanrenouf.

6 thoughts on “Waiting for OS customization to complete

  1. Rajesh

    Great Post. Thanks. We have deployed an Self Service portal for VM deployment via VCO on VCD. Now we have specified an random sleep time before sending out email to requestor. Now I guess we can really test before sending out the email.

    Reply
  2. Joel Newton

    Hi,

    Thanks for the great post. I was wondering if you’ve come across the scenario where the customization never logs to VI-Events that it succeeds or fails, but for all intents and purposes, it looks complete. Is it possible that, if the new VM is vmotioned while the customization is happening, that that may interrupt the event logging?

    (No errors logged in C:\Windows\TEMP\vmware-imc\guestcust.log either)

    Thanks,
    Joel

    Reply
    1. Cory

      I’ve also run into the scenario where the customization never logs that it completed to the VI-Events, but it did actually complete without any errors. This also corresponded with the a VMotion occurring in the middle of the customization process. This seems like a bug.

      Reply
  3. Pingback: The ultimate VM batch deployment script. - VMwaremine - Mine of knowledge about virtualization

  4. Pingback: Waiting on virtual machine Guest OS customization to complete | Deans Blog

  5. Pingback: Auto-Deploy Virtual Domain Controller on VMware using PowerCli. -Virtualize & Automate-

Leave a Reply

Your email address will not be published. Required fields are marked *

*