posted

0 Comments

The all new vRealize Orchestrator 7.0 Content Pack for Log Insight has just been released.  Anyone who has used the vRealize Automation 6.1+ Content Pack might recall there was a pretty basic dashboard for vRO workflow calls from the workflow stubs, but no in depth vRO queries.  This new content pack provides thorough coverage for both standalone/clustered vRO appliances and the embedded version on the vRA appliance, including:

  • Server Overview – get the status of all vRO hosts in your environment at a glance, including details of reboots and service state changes
  • Workflow Failures – details about failed workflows by host, by workflow and error log events
  • Troubleshooting – provides two sets of queries for investigating general log errors and warnings, and workflow-specific logs for troubleshooting
  • Workflow Logs – breaks down workflow info, warning and error logs by workflow name and token, user, and message type
  • Workflow Scripting Logs – breaks down workflow scripting info, warning and error logs by user, workflow name and workflow token
  • Configuration Audit – reports events for changes to boot, initial heap and max heap configurations
  • Authorization Audit – reports on number of failed logins, failed login details by user and message, and provides license auditing
  • Content Audit – tracks resource, workflow and configuration changes by host and over time
  • Content Timeline Audit – content modifications by element type and change time, and detailed list of changed items over time
  • Workflow Statistics – shows successfully completed workflows and workflow runs by host, workflow name, user name and combinations there of
  • Workflow Timeline Statistics – shows successfully completed workflows and workflow runs over time and by end state

Continue reading to learn more about each of these…

Server Overview

The Server Overview dashboard provides the status of vRO nodes and state changes for hosts.  It also shows server reboots and server state changes from standby to running, and vice versa.

vRO-CP-SvrOverview

vRO Content Pack – Server Overview

Workflow Failures

The Workflow Failures dashboard will help you track down problem workflows by host, and see which workflows repeatedly fail.  You will also find Error log events by workflow and over time.

vRO-CP-WFFailures

vRO Content Pack – Workflow Failures

Drilling into the FAILED workflows grouped by workflow name allows you to see the details of the log, but also gives you next steps for troubleshooting.  In the sample below, you can see that the workflow token (unique identifier for a specific workflow execution) is tagged in the error log. Adding this field/value pair to the query is a simple step: hover over the field (vmw_vro_wf_token) below the event and select Value is ‘xxx’.

vRO-CP-FailedWFToken

vRO Content Pack – Filter by Failed WF Token

After adding the token field to the filter and removing all other filters, as shown below, you can see all logs related to this specific workflow execution. It’s pretty easy to spot the No route to host exception now! As it turns out, I intentionally powered off the Puppet host in my lab to force a failure. You can see that this is far more valuable, from a troubleshooting perspective, than the error logs showing the execution stack in the vRO client.

vRO-CP-LogsByToken

vRO Content Pack – Logs by WF Token

Troubleshooting

While we just walked through some troubleshooting steps, here in the Troubleshooting dashboard you will find more queries for investigating issues.  The two sets of queries found here allow you to drill into general log events of different priorities and workflow runs. The Workflow run queries are set up with various parameters, ready for you to paste values in, after you open them in the Interactive Analytics tab. Context, for example, is used to trace vRealize Automation requests from the catalog through the various components, including vRO, for a more complete picture of logs related to a user request.

vRO-CP-Troubleshooting

vRO Content Pack – Troubleshooting

Workflow Logs

The Workflow Logs dashboard gives you a great view of workflow executions as it breaks down the logs by priority (info, warning, error) and by workflow name and workflow token (specific execution of a given workflow). In the example below, you can see that the Puppet workflows have had some issues (because I powered off the Puppet host in my lab).  You will also see a breakdown of workflow warnings and errors per user and per message type.

vRO-CP-WFLogs

vRO Content Pack – Workflow logs

Workflow Scripting Logs

The Workflow Scripting logs are those explicitly written into the workflows by the workflow author, using the System.log  (or .warn or .error or .debug) command.  When thoughtful logging is applied during workflow authoring, these queries can be used to root cause intermittent issues.

vRO-CP-WFScriptLogs

vRO Content Pack – Workflow scripting logs

Configuration Audit

The Configuration Audit dashboard allows you to trace changes to key system properties and changes to them. This can be helpful in troubleshooting your vRO environment or finding possible changes that have impacted it, such as vRA integration details (found by searching for “vro.cafe” in the results of the BOOT configuration query).

vRO-CP-ConfigAudit

vRO Content Pack – Configuration Audit

Authorization Audit

As the name implies, this dashboard focuses on Authorization issues (i.e. failed logins).  In the examples below, the failure for administrator@vsphere.local was due to an incorrect password while the failure for kim@vsphere.local was for an invalid user.  You can also check all vRO nodes to see if any are using an evaluation license using the provided query.

vRO-CP-AuthAudit

vRO Content Pack – Authorization Audit

Content Audit

The Content Audit dashboard has great value in tracking modifications to resources, workflows and configurations, all per host and over time.  This can be helpful, in a clustered vRO environment, to ensure updates such as workflow changes are made to all nodes. You can also use these queries to track down changes if a workflow suddenly starts to throw errors.

vRO-CP-ContentAudit

vRO Content Pack – Content Audit

Content Timeline Audit

Tracking changes to workflows as well as resource elements (image files, scripts, XML templates, HTML files, etc used by any workflows or web views) can help root cause issues that suddenly appear or happen intermittently. The queries on this dashboard provide a view of changes to both workflows and resource elements on the same timelines.

vRO-CP-ContentTLAudit

vRO Content Pack – Content Timeline Audit

Workflow Statistics

The Workflow Statistics dashboard has queries to track successfully completed workflows (with final state of successful) as well as workflow runs (workflows initiated, regardless of final state). These are grouped by various combinations of host, by workflow name, and user, and give you a sense of overall volume of execution.

vRO-CP-WFStats

vRO Content Pack – Workflow Statistics

Workflow Timeline Statistics

This dashboard shows similar information to the Workflow Statistics, but with different views so that you can compare, for example, the volume of workflow runs over time by workflow name, or the volume of workflow runs by end state (completed or failed).

vRO-CP-WFTLStats

vRO Content Pack – Workflow Timeline Statistics

Downloading and Configuring

The vRealize Orchestrator 7.0 Content Pack for Log Insight is available from the in-product Log Insight Marketplace or on VMware Solution Exchange. Note that this release of the vRO 7.0 Content Pack is NOT compatible for use with the Log Insight agent. You should use the vRO Control Center (<hostname>:8283/vco-controlcenter) to set up the log forwarding (found under Log, Syslog Integration).  Be sure to set the Threshold to INFO so proper logs are forwarded to Log Insight.

vRO-CP-CtrlCenterLog

Control Center – Log Forwarding

I hope you find this new vRealize Orchestrator Content Pack helpful in monitoring and managing your vRO and/or vRA environment! As always, feel free to leave a comment here or reach out to me via Twitter with feedback or enhancements. Happy Troubleshooting!