Cloud Automation Cloud Operations vRealize Operations vRealize Orchestrator Web/Tech

The Self-Healing Datacenter – Using Webhooks to Automate Remediation

Recently, Steven Flanders wrote a nice blog post explaining what Webhook Shims are and how they can be used.  In summary, both vRealize Log Insight and vRealize Operations support webhooks.  This is a capability that allows for integration with any other solution that has a REST API available.  Ticketing, notification, chat and other capabilities can be leveraged by vRealize Intelligent Operations.  Extending that capability can make your datacenter a self-healing datacenter!

If you are a user of vRealize Operations, you know that it can monitor your infrastructure, server OS, applications and more.  But as you know, monitoring is only part of the answer.  Wouldn’t it be much better to have vRealize Operations attempt some simple fixes before giving up and calling for human intervention?

In this blog post series, I will explain how to activate automatically remediate a problem detected by vRealize Operations using the vRealize Orchestrator Webhook shim to launch workflow based on an alert.

REST Notification Plugin

Currently, vRealize Operations alerting can trigger notifications via email, SNMP trap or REST notification.  The REST notification plugin allows you to interact with REST APIs from practically any third-party system – almost!

Setting up an Alert notification rule in vR Ops

The truth is the REST Notification Plugin is not very robust and does not provide a lot of control over format of the REST call.  You plug in the URI and some credentials and vRealize Operations will try a PUT and POST against that URI.  By the way, it will append the alert ID onto the end of the URI.

Adding a REST Notification Plugin instance in vR Ops. Note the limited options available; there is nowhere to format the request body or control the method used.

This means is that unless your REST API is expecting the request in the way vR Ops would like to send it – it will fail.

For example, Orchestrator has a robust REST API.  To launch a workflow via that API, I would send:

curl -X POST -H "Authorization: Basic dmNvYWRtaW46dmNvYWRtaW4=" -H "Accept: application/json" -d '{
  "parameters": [
      "value": {},
      "name": "",
      "type": "",
      "description": "",
      "scope": "",
      "updated": false
}' ""

Enter Webhook Shims!

As you can see, Orchestrator expects a body for this request that includes any inputs.  Orchestrator expects this payload even if there are no inputs.  My vRealize Operations REST call to Orchestrator will fail because it is not making a valid request.  How can we fix this?

One way is to translate the REST call from vRealize Operations into a REST call that can be consumed by Orchestrator.  Enter the Webhook Shims for Log Insight and vRealize Operations!  This diagram shows how it works:

Webhook Shims enables self-healing with vRealize Operations through an Orchestrator shim.

The Orchestrator shim takes the simplistic input from vR Ops REST notification and rebuilds the request in the format expected by Orchestrator.

In the next blog post, I will show you how to implement this solution to automatically attempt remediation based on alert notifications from vRealize Operations to create your own self-healing datacenter!


6 comments have been added so far

  1. hi john
    thanks for your great article
    i have a problem
    when i wanted to install PIP it gave me an error that indicated it doesn’t know python command
    i installed it using ” tdnf install python git pycrypto python-zmq ”
    now it knows python as well as python2
    but after running command python get-pip-py it gives me this error:
    root@photon-machine [ / ]# root@photon-machine [ / ]# python2
    Traceback (most recent call last):
    File “”, line 20890, in
    File “”, line 197, in main
    File “”, line 82, in bootstrap
    import pip._internal
    File “/tmp/tmp7jCRL3/”, line 40, in
    File “/tmp/tmp7jCRL3/”, line 8, in
    File “/tmp/tmp7jCRL3/”, line 12, in
    File “/tmp/tmp7jCRL3/”, line 6, in
    File “/tmp/tmp7jCRL3/”, line 6, in
    File “/tmp/tmp7jCRL3/”, line 23, in
    File “/tmp/tmp7jCRL3/”, line 14, in
    File “/tmp/tmp7jCRL3/”, line 25, in
    File “/tmp/tmp7jCRL3/”, line 7, in
    File “/tmp/tmp7jCRL3/”, line 13, in
    File “/tmp/tmp7jCRL3/”, line 10, in
    ImportError: No module named xml.etree.ElementTree
    -bash: root@photon-machine: command not found

    what should i do? thanks

Leave a Reply

Your email address will not be published. Required fields are marked *