Waiting for OS customization to complete

Posted by
Vitali Baruh
PowerCLI QE team

Provisioning of new virtual machines often includes Operating System (OS) customization.

A typical provisioning workflow could look like this:

1.  Clone a virtual machine from a template and specify a customization specification

2.  Start the VM. This triggers the customization process

3.  Wait for the customization to complete

4.  Perform additional setup steps

Step three is quite tricky, because there’s no easy way to find out when the customization has completed within the guest operating system and if it has succeeded or has failed.

This step could be implemented in an optimistic manner with sleep for 10 minutes, but it could also be implemented in a more robust and informative way. To do this, we could create a script with the following specification:

Input

• List of virtual machines which should be monitored for the completion of a customization process
• Timeout in seconds that the script should wait for the customization to end

Output

• List of PSObjects where each object will hold a passed VM and its customization status – successful, failed, started, et.

The function is based on several virtual machine events defined by vCenter Server:

• VmStartingEvent – posted when VM is powered on
• CustomizationStartedEvent – posted when customization is started
• CustomizationSucceeded – posted when customization succeeds
• CustomizationFailed – posted when customization fails

Our script workflow is pretty simple:

• 1. For each VM we will look for the last VmStarting event, because the customization process starts after the VM has been powered on
  • 1.1. If such an event is found, change the status to “CustomizationNotStarted”
  • 1.2. If such an event is not found, change the status to “VmNotStarted”
• 2. Start a loop
  • 2.1. For each VM with status “CustomizationNotStarted”
    • 2.1.1. Check for posted CustomizationStartedEvent after the last power-on operation
      • 2.1.1.1. If such an event is found, change the status to “CustomizationStarted”
  • 2.2. For each VM with status “CustomizationStarted”
    • 2.2.1. Check for Succeded or Failed Event
      • 2.2.1.1. If such an event is found, change the status to the corresponding “CustomizationSucceeded” or “CustomizationFailed” values
  • 2.3. Check is there more VMs with status “CustomizationNotStarted” or “CustomizationStarted” (we should continue to monitor these VMs)
    • 2.3.1. If no, break the loop
  • 2.4. Check whether the specified timeout has elapsed
    • 2.4.1. If yes, break the loop
• 3. Return result

Now let see the script in action!

Step 1 – retrieve template, OS customization spec, target vmhost and target datastore.

As result we have several variables that are needed to start cloning with customization tasks.

Step 2 – start clone operations for two VMs

As result we have $vms variable holding our virtual machines.

Step 3 – start virtual machines, which triggers customization process and monitor it with our script.

The script is started with two virtual machines and also we specify to wait maximum 600 seconds for OS customization to complete.

After the start the script reports each VM which will be monitored.

The next screen shows that after about 45 second customization start events have been posted for both VMs.

The last screen shot shows the moments when the two customization succeed events have been posted and also the result of the script.

Finally the $result variable contains two custom objects. Each object contains VM and CustomizationStatus properties.

Our script could iterate over the $result variable and to perform some additional setup actions – installing software not included in the template, applying specific software configurations, etc.

Of course for these additional setup actions we can use the Invoke-VMScript and Copy-VMGuestFile cmdlets.

Now let’s see the code of WaitVmCustomization.ps1 script:

<#

.SYNOPSIS
Waits customization process for list virtual machines to completes.

.DESCRIPTION
Waits customization process for list virtual machines to completes.
The script returns if customization process ends for all virtual machines or if the specified timeout elapses.
The script returns PSObject for each specified VM.
The output object has VM and CustomizationStatus properties.

.EXAMPLE
$vm = 1..10 | foreach { New-VM -Template WindowsXPTemplate -OSCustomizationSpec WindowsXPCustomizaionSpec -Name "winxp-$_" }
.\WaitVmCustomization.ps1 -vmList $vm -timeoutSeconds 600

.NOTES
The script is based on sveral vCenter events.
* VmStarting event – this event is posted on power on operation
* CustomizationStartedEvent event – this event is posted for VM when customiztion has started
* CustomizationSucceeded event – this event is posted for VM when customization has successfully completed
* CustomizationFailed – this event is posted for VM when customization has failed

Possible CustomizationStatus values are:
* "VmNotStarted" – if it was not found VmStarting event for specific VM.
* "CustomizationNotStarted" – if it was not found CustomizationStarterdEvent for specific VM.
* "CustomizationStarted" – CustomizationStartedEvent was found, but Succeeded or Failed event were not found
* "CustomizationSucceeded" – CustomizationSucceeded event was found for this VM
* "CustomizationFailed" – CustomizationFailed event wass found for this VM

#>
[CmdletBinding()]
param(
   # VMs to monitor for OS customization completion
   [Parameter(Mandatory=$true)]
   [ValidateNotNullOrEmpty()]
   [VMware.VimAutomation.ViCore.Types.V1.Inventory.VirtualMachine[]] $vmList,
  
   # timeout in seconds to wait
   [int] $timeoutSeconds = 600
)

# constants for status
      $STATUS_VM_NOT_STARTED = "VmNotStarted"
      $STATUS_CUSTOMIZATION_NOT_STARTED = "CustomizationNotStarted"
      $STATUS_STARTED = "CustomizationStarted"
      $STATUS_SUCCEEDED = "CustomizationSucceeded"
      $STATUS_FAILED = "CustomizationFailed"
     
      $STATUS_NOT_COMPLETED_LIST = @( $STATUS_CUSTOMIZATION_NOT_STARTED, $STATUS_STARTED )
     
# constants for event types     
      $EVENT_TYPE_CUSTOMIZATION_STARTED = "VMware.Vim.CustomizationStartedEvent"
      $EVENT_TYPE_CUSTOMIZATION_SUCCEEDED = "VMware.Vim.CustomizationSucceeded"
      $EVENT_TYPE_CUSTOMIZATION_FAILED = "VMware.Vim.CustomizationFailed"
      $EVENT_TYPE_VM_START = "VMware.Vim.VmStartingEvent"

# seconds to sleep before next loop iteration
      $WAIT_INTERVAL_SECONDS = 15
     
function main($vmList, $timeoutSeconds) {
   # the moment in which the script has started
   # the maximum time to wait is measured from this moment
   $startTime = Get-Date
  
   # we will check for "start vm" events 5 minutes before current moment
   $startTimeEventFilter = $startTime.AddMinutes(-5)
  
   # initializing list of helper objects
   # each object holds VM, customization status and the last VmStarting event
   $vmDescriptors = New-Object System.Collections.ArrayList
   foreach($vm in $vmList) {
      Write-Host "Start monitoring customization process for vm '$vm'"
      $obj = "" | select VM,CustomizationStatus,StartVMEvent
      $obj.VM = $vm
      # getting all events for the $vm,
      #  filter them by type,
      #  sort them by CreatedTime,
      #  get the last one
      $obj.StartVMEvent = Get-VIEvent -Entity $vm -Start $startTimeEventFilter |
         where { $_ -is $EVENT_TYPE_VM_START } |
         Sort CreatedTime |
         Select -Last 1
        
      if (-not $obj.StartVMEvent) {
         $obj.CustomizationStatus = $STATUS_VM_NOT_STARTED
      } else {
         $obj.CustomizationStatus = $STATUS_CUSTOMIZATION_NOT_STARTED
      }
     
      [void]($vmDescriptors.Add($obj))
   }        
  
   # declaring script block which will evaulate whether
   # to continue waiting for customization status update
   $shouldContinue = {
      # is there more virtual machines to wait for customization status update
      # we should wait for VMs with status $STATUS_STARTED or $STATUS_CUSTOMIZATION_NOT_STARTED
      $notCompletedVms = $vmDescriptors |
         where { $STATUS_NOT_COMPLETED_LIST -contains $_.CustomizationStatus }

      # evaulating the time that has elapsed since the script is running
      $currentTime = Get-Date
      $timeElapsed = $currentTime – $startTime
     
      $timoutNotElapsed = ($timeElapsed.TotalSeconds -lt $timeoutSeconds)
     
      # returns $true if there are more virtual machines to monitor
      # and the timeout is not elapsed
      return ( ($notCompletedVms -ne $null) -and ($timoutNotElapsed) )
   }
     
   while (& $shouldContinue) {
      foreach ($vmItem in $vmDescriptors) {
         $vmName = $vmItem.VM.Name
         switch ($vmItem.CustomizationStatus) {
            $STATUS_CUSTOMIZATION_NOT_STARTED {
               # we should check for customization started event
               $vmEvents = Get-VIEvent -Entity $vmItem.VM -Start $vmItem.StartVMEvent.CreatedTime
               $startEvent = $vmEvents | where { $_ -is $EVENT_TYPE_CUSTOMIZATION_STARTED }
               if ($startEvent) {
                  $vmItem.CustomizationStatus = $STATUS_STARTED
                  Write-Host "Customization for VM '$vmName' has started"
               }
               break;
            }
            $STATUS_STARTED {
               # we should check for customization succeeded or failed event
               $vmEvents = Get-VIEvent -Entity $vmItem.VM -Start $vmItem.StartVMEvent.CreatedTime
               $succeedEvent = $vmEvents | where { $_ -is $EVENT_TYPE_CUSTOMIZATION_SUCCEEDED }
               $failedEvent = $vmEvents | where { $_ -is $EVENT_TYPE_CUSTOMIZATION_FAILED }
               if ($succeedEvent) {
                  $vmItem.CustomizationStatus = $STATUS_SUCCEEDED
                  Write-Host "Customization for VM '$vmName' has successfully completed"
               }
               if ($failedEvent) {
                  $vmItem.CustomizationStatus = $STATUS_FAILED
                  Write-Host "Customization for VM '$vmName' has failed"
               }
               break;
            }
            default {
               # in all other cases there is nothing to do
               #    $STATUS_VM_NOT_STARTED -> if VM is not started, there's no point to look for customization events
               #    $STATUS_SUCCEEDED -> customization is already succeeded
               #    $STATUS_FAILED -> customization
               break;
            }
         } # enf of switch
      } # end of the freach loop
     
      Write-Host "Sleeping for $WAIT_INTERVAL_SECONDS seconds"
      Sleep $WAIT_INTERVAL_SECONDS
   } # end of while loop
  
   # preparing result, without the helper column StartVMEvent
   $result = $vmDescriptors | select VM,CustomizationStatus
   return $result
}

#calling the main function
main $vmList $timeoutSeconds

And here is the code for the scenario script:

# prepare necesary objects for clone operations
$template = Get-Template WindowsXPTemplate
$spec = Get-OSCustomizationSpec WindowsXPSpec
$vmhost = Get-VMHost | select -First 1
$datastore = Get-Datastore -VMHost $vmhost -Name Storage1

# Start cloning operations
$vms = 1..2 | % { New-VM -Name "WinXP-$_" -ResourcePool $vmhost -Template $template -Datastore $datastore -OSCustomizationSpec $spec }

# Trigger customization and wait its completion
$vms = $vms | Start-VM
$result = .\WaitVmCustomization.ps1 -vmList $vms -timeoutSeconds 600

Several notes about the script

• It doesn’t accept pipeline input, because it checks the customization status for multiple virtual machines simultaneously. This won’t be possible if the process block of the script is executed for each object passed by the pipeline.
• It also shows how to work with specific type of events.
• Its VIEvent searches are optimized by specifying concrete entity and start time filters

Hope that this blog post is interesting and useful.