In this post, I would like to drill into the new webhook alert option available in Log Insight 3.3. Read on to learn more!
Defining webhooks
My colleague Alan Castonguay put it well when he said, “Webhooks are user-defined HTTP callbacks. Something happens in an application (e.g. alert fired, change made, commit merged, etc) that some remote service cares about. It doesn’t matter why they care, they just want to be told that some event happened. It could be for auditing, or to start a workflow, or something else.”
Given how generic the webhook definition is, webhooks provide a lot of flexibility, but also incur some administrative overhead — more on this in a minute.
Configuring webhooks in LI
There are two places in LI where webhooks can be configured:
- For system notification: under the General page in the Administration section
- For user alerts: while creating a new user alert or by editing an existing user alert
In both cases, a space separated list of webhook URLs should be specified. Note that authenticated webhooks are not supported in the LI UI as of version 3.3. You can of course send traffic over HTTPS, but the destination cannot require authentication.
Output of a LI webhook
The output of a LI webhook depends on the type of webhook (i.e. user or system) and type of query (user only). Let’s see an example for each:
System
{
"AlertName":" Admin Alert: Worker node has returned to service (Host = 127.0.0.2)",
"messages":[
{
"text":"This notification was generated from Log Insight node (Host = 127.0.0.2, Node Identifier = a31cad22-65c2-4131-8e6c-27790892a1f9).nnA worker node has returned to service after having been in maintenance mode.nnThe Log Insight master node (Host: <a href='https://10.113.236.182:9443/'>https://10.113.236.182:9443/</a>, Node Identifier: 88fc9956-bf9a-428b-806a-22ff07636273) reports that worker node has finished maintenance and exited maintenance mode. The node will resume receiving configuration changes and serving queries. The node is also now ready to start receiving incoming log messages. If an external load balancer is configured to distribute messages among workers, the administrator should add this node back to the pool of nodes receiving incoming messages.nnThis message was generated by your Log Insight installation, visit the <a href='https://www.vmware.com/support/pubs/log-insight-pubs.html'>Documentation Center</a> for more information.",
"timestamp":1458665320514,"fields":[]
}
]
}
User: Message Query
{
"AlertType":1,
"AlertName":"Hello World Alert",
"SearchPeriod":300000,
"HitCount":0.0,
"HitOperator":2,
"messages":[
{
"text":"hello world 1",
"timestamp":1451940578545,
"fields":[
{
"name":"Field_1",
"content":"Content 1"
},
{
"name":"Field_2",
"content":"Content 2"
}
]
},
{
"text":"hello world 2",
"timestamp":1451940561008,
"fields":[
{
"name":"Field_1",
"content":"Content 1_2"
},
{
"name":"Field_2",
"content":"Content 2_2"
}
]
}
],
"HasMoreResults":false,
"Url":"https://10.11.12.13/s/8pgzq6",
"EditUrl":"https://10.11.12.13/s/56monr",
"Info":"This is an alert for all the 'Hello World' messages",
"NumHits":2
}
User: Aggregation Query
{
"AlertType":2,
"AlertName":"field_1 aggregated alert",
"SearchPeriod":300000,
"HitCount":2.0,
"HitOperator":2,
"messages":[
{
"fields":[
{
"name":"Field_1",
"content":"Content 1"
}
]
}
],
"HasMoreResults":false,
"Url":"https://10.11.12.13/s/r25g3s",
"EditUrl":"https://10.11.12.13/s/n3gsed",
"Info":null,
"NumHits":1
}
Sending a LI webhook to a 3rd party
If you read the release notes, you will see mention of third-party applications like Slack. If you go ahead and enable an incoming webhook in Slack and then leverage the new test alert option for user alerts in LI, you will receive the following error:
Wait, I thought LI webhooks allow me to send notifications to third-party solutions? They do, but since LI sends a webhook in its own proprietary format and third-party solutions expect incoming webhooks to be in their proprietary format we have a problem. The solution is to write shim between LI and the third-party solution which translates LI format to third-party format. I will talk about how to write shims in a future post.
Requesting more LI webhook functionality
You might look at the existing LI webhook capability and wish for some additional functionality (e.g. built-in shims, authentication support, etc). If so, please head over to the Log Insight Community and file or vote for a feature. Note the LI team wanted to get this feature out early so feedback could be gathered and more functionality could be added (welcome to agile development!) so please be sure to participate in the community.
Summary
LI webhooks provide further extensibility for alerts in LI. They are simple to enable, but as of LI 3.3 do not support authentication and require a shim be written in order to translate the outputted format into a third-party understandable format. For suggestions on future enhancements to the webhook functionality, be sure to head over to the Log Insight community.
