Home > Blogs > VMware vCenter Orchestrator Blog


How to deal with long running external processes

When building vCenter Orchestrator Workflows, chances are that you eventually have to integrate external systems with longer running (multiple minutes, hours, maybe even days) processes.

Examples for that are:

  • Software Deployment processes
  • Clone tasks
  • When you have to rely on processes that only occur once a day (like an automatic discovery in a monitoring system)
  • “semi-automatic” processes: The workflow creates a ticket, that is going to be processed manually, and continues with the automation after the ticket is closed.

 While it’s quite easy to initiate/trigger the external system to start that long running task, it is more challenging to programmatically figure out when the external task completed (and the continue with the workflow).
vCenter Orchestrator External systemsThis article shows different strategies to wait for long running external tasks and their advantages and drawbacks.

“Synchronous” processing:

Here the workflow in vCenter Orchestrator triggers a long running process in an external system, and when this process finishes, the external system actively “tells” vCenter Orchestrator that it’s done.
vCenter Orchestrator Synchronous processing This can be achieved through two (and a half) different ways:

  1. The external system is integrated via a Plugin that provides Trigger elements or Events for vCenter Orchestrator Policies.
    If that is the case, you can use a “Waiting event” element in the workflow while waiting for the external task to complete.
    Example: The “Wait for Task” Workflow in the vCloud Director Plugin does exactly this. It is used by all the workflows of the vCloud Director Plugin that trigger long running tasks in vCloud Director.
    vCenter Orchestrator plugin waitinga “and a half”) If there is no special Plugin for the system you call out to, but this external system “speaks” AMQP or SNMP, you can use these plugins. The AMQP Plugin provides a Workflow “Wait for a message” for this for example.
  2. Use a “User Interaction” element, that is answered by the external system via vCenter Orchestrator’s SOAP or REST API.

The workflow stops at the “User interaction” element, and waits for the answer.

Here the external system of course needs the ability to somehow call the vCenter Orchestrator API to answer the user interaction.

This way has a nice side effect: If something goes wrong in the external process, a human administrator can fix it and easily answer the element manually.

Having the external system actively call back to vCenter Orchestrator doesn’t waste any time when the external process is done. And it is not necessary for the workflow itself to examine that process (as it will be necessary with the next strategy).

“Asynchronous” polling:

If the external system which runs the long running process can not actively call back to vCenter Orchestrator, then there is only one other way to examine when it is finished: Polling.
vCenter Orchestrator asynchronous pollingTo allow a polling in general, a workflow in vCenter Orchestrator must be able to figure out that the external process has finished. This sometimes can be done by checking the external system directly, e.g. because its API allows to monitor these processes, or you can check for a certain state of the external system.

In case it is not possible to check the external system directly if the process has finished (because it only provides a “fire-and-forget” sort of API for instance), then you might use some “helper”: Check if a certain file exists in a specified place, or if there is a certain entry in a database (remember the SQL Plugin for vCenter Orchestrator!), or even if there just is a new email in the mailbox vcoresponses@mycompany.com.

When you made sure how the workflow can figure out that the external process is completed, the next challenge is to organize to schedule the polling. Here again there are different strategies possible:

  1. A loop in a JavaScript element that uses System.sleep()While this is a simple solution for short running processes, it is not recommended for to monitor longer running processes: System.sleep() keeps the workflow in a running state. So it keeps using system resources on the vCenter Orchestrator server.Example: The widely used “vim3waitTaskEnd”-Action in the vCenter Library that monitors the (usually short running) tasks in vCenter which have been triggered by vCenter Orchestrator.
    To save resources (and to not risk reaching the maximum number of concurrent running workflows, 300 in vCenter Orchestrator 5.1), it’s better to have the polling workflow in a not “running” state while waiting for the next poll. This can be achieved using following strategies:
  2. Use a “Waiting Timer” element
    You first calculate the date & time when the workflow shall continue (with the next polling iteration), and then use a “Waiting timer” element. This will let the workflow get into a “waiting” state, so it’s state gets persisted in the vCenter Orchestrator database, and it does not use any runtime resources (“waiting” workflows also don’t count to the maximum number of running workflows!). When the calculated time is reached, vCenter Orchestrator will automatically “re-awake” the workflow again.
    vCenter Orchestrator use Waiting Timer
  3. Use the Workflow scheduler
    vCenter Orchestrator allows to schedule workflows to run automatically at a given date/time, optionally with a recurrence, e.g. for every hour. Now you even can schedule workflows programmatically, using the “Schedule workflow” element in the schema. This can be used nicely for the polling workflows, it is even possible that the polling workflow schedules itself for the next run.Find an example how to do this (in a slightly other scenario, but you’ll can transfer it easily) in the communities: http://communities.vmware.com/thread/318791

However, all these strategies generate a linear growing: For every external process started one workflow waiting/monitoring/polling this process is generated. If you now have a lot of these, and/or very long running processes, then you might want to avoid this growing. Even that is possible with vCenter Orchestrator, by using Configuration Elements in a creative way. (In case you need a short “refresh” about Configuration Elements, read this article: http://blogs.vmware.com/orchestrator/2012/02/configuration-elements-revisited.html)

The idea is that a workflow starts a new external process, stores information about this process to a Configuration Element and then just finishes. Another SINGLE polling workflow then reads this Configuration Element, checks all external processes, and in case some of them finished it calls workflow that follows the external process.

So it does not matter if 5 or 500 external processes have to be monitored, there is always only one polling workflow active.

You can see, vCenter Orchestrator provides a lot of mechanisms you can use to build reliable workflow solutions even for difficult integration situations.

Article Author: Joerg Lew