Agents make it possible to collect events from log files on Linux and Windows devices and forward them to a vRealize Log Insight server or a third-party logging system. Well … we all know that!! A LOT and yes I mean a LOT of folks from the field have asked for the agent to be able to send logs to multiple destinations and possibly over different protocols and perhaps also have filtering on the logs before they get sent as the cherry on the cake. With vRealize Log Insight v4.5 we can!
vRealize Log Insight v4.5 agent allows you to deliver collected events to multiple destinations via Ingestion API or syslog protocols simultaneously. It also has the ability to define multi-destination servers and allow filtering of events per destination based on event collecting source and the events field’s values.
You could want to do this for several reasons like deliver the same events to multiple vRLI clusters for backup or data recovery purposes. Or you may want to send different kind of events to different IT department logging systems, for example audit/system logs to the security team’s server, application logs to DevOps team’s server and system metric logs to the IT team’s management system.
So let’s look at how you can actually use it…..
Multiple destination connections could be defined through the [server|<destination_id>] section, where <destination_id> is unique per configuration connection id ( i.e. per liagent.ini). For backward compatibility there could be an unnamed [server] section which will be treated as master connection and the current implementation assumes that it always exists by default. Only the master connection will be able to ask configuration from server and only if cfapi is used as connection protocol. All server sections will use all existing configurations options to define connection properties, i.e. hostname, proto, port, ssl, etc. The only exception is that ‘hostname’ option will not have a default value for non-master connections. For [server] section (without the destination_id) the default value for hostname option remains “loginsight”.
The following parameters can be defined in server sections: hostname, proto, port, ssl, filter.
Note:
- Default value for ‘filter’ option is {;.*;}, which means accept all events.
- The {;;} filter could be used to deny any event transmission to the destination, for example the connection could be used only for configuration and auto-update purposes.
- Example #1 of multi-destination use in liagent.ini
– # First (default) destination will receive all collected events.
[server]
hostname=prod1.team.vmware.com
– # Second destination will receive just syslog events through the syslog protocol.
[server|syslog-audit]
hostname=third_party_audit_management.vmware.com
proto=syslog
filter={filelog; syslog; }
– # Third destination will receive vrops events if they have the level field equal to “error” or “warning” and they are collected by agent config sections which name begin with “vrops-“.
[server|team-prod1]
hostname=vrops-errors.team.vmware.com
filter={; vrops-.*; level == “error” || level == “warning”}
– # The filelog section below is collecting syslog messages that are sent to [server] and [server|syslog-audit]
– [filelog|syslog]
directory=/var/log
include=messages
– # For various vRops logs. Note that all section names begin with “vrops-” prefix, which is used in third server destination filter above. The logs in these sections are sent to [server] and [server|team-prod1]
– [filelog|vrops-ANALYTICS-analytics]
directory=/data/vcops/log
include=analytics*.log*
exclude=analytics*-gc.log*
parser=auto
– [filelog|vrops-COLLECTOR-collector]
directory=/data/vcops/log
include=collector.log*
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\,\d{3}
parser=auto
– [filelog|vrops-COLLECTOR-collector_wrapper]
directory=/data/vcops/log
include=collector-wrapper.log*
event_marker=^\d{4}-\d{2}-\d{2}[\s]\d{2}:\d{2}:\d{2}\.\d{3}
parser=auto
Here all logs from [filelog|syslog] section go to server third_party_audit_management.vmware.com over the syslog protocol. The vROps logs from the three filelog sections go to server vrops-errors.team.vmware.com if they are of level error or warning. And the server prod1.team.vmware.com is receiving all events.
- Example #2 of multi-destination use in liagent.ini
[server]
hostname=10.11.12.13
ssl=no
[server|desktop]
hostname=10.21.22.23
filter={winlog; System; }
[winlog|System]
channel=System
[winlog|Application]
channel=Application
[winlog|Security]
channel=Security
In this example all events (from all three channels) go to hostname 10.11.12.13 and server 10.21.22.23 only receives events that are received on the System channel. ssl option is not defined for [server|desktop] section therefore the default will be used which is ssl=yes.
Note: The defaults for the options in the [server] sections are still the same as before.
; Protocol can be cfapi (Log Insight REST API), syslog. Default:
;proto=cfapi
; Log Insight server port to connect to. Default ports for protocols (all TCP):
; syslog: 514; syslog with ssl: 6514; cfapi: 9000; cfapi with ssl: 9543. Default:
;port=9000
; SSL usage. Default: (Starting vRLI 4.0 default ssl value is yes)
;ssl=yes
It is important to note that sending logs to multiple agent destination will not send duplicate events. No events duplication will be seen at the destination servers, i.e, if more than 1 filter_tuple matches the same event, then the event will be sent just once. So the following filter definitions are equal:
- filter = {filelog; sample.*; facility > 7}, {filelog; sample.*; level == “error”}
- filter = {filelog; sample.*; facility >7 || level == “error”}
The events with facility=8 and level=”error” will be sent just once.
Some additional details about Filter criteria
The filter format for an agent destination looks like { collector_type ; collector_filter ; event_filter }
The filter selects only logs collected by sections where collector type (i.e. filelog or winlog) matches to provided collector_type and name of collector_filter (identifier which goes after pipe sign e.g. filelog|syslog) matches to collector_filter regular expression, in addition if there is an event_filter defined then it selects only events for which the expression evaluates to True.
collector_filter expression should not contain ;{} characters. In case of name-spaced collectors the collector_filter matches only the name part ignoring the namespace, i.e “ana.*” filter will match the following section names: “com.vmware.vrops.analitics” and “anaconda” . The regular expression collector_filter value could be an empty string which doesn’t match to any collector name, i.e no events will be sent. The {;;} filter could be used to deny any event transmission to the destination, for example the connection could be used only for configuration and auto-update purposes.
event_filter is an event fields expression working in the same way as collectors ‘whitelist’ option and evaluates based on event fields values. It could be an empty string which treated as True expression.
Default value for ‘filter’ option is {;.*;}, which means accept all events.
For every collected event the connection evaluates the event properties to match the filter. All filter_tuples in the list are concatenated by logical ‘or’ operation. If event passed any filter_tuple then it will be sent to the destination server. Every filter_tuple evaluated by the following steps:
1. If collector_type_list is empty or contains the collector type of the event then proceeds to the next step, otherwise drop the event.
2. If event collector name (i.e. part of the section name after pipe sign) of the event matches the collector_filter then proceed to the next step, otherwise drop the event.
3. If event_filter is empty or evaluates to True then send the event to the destination, otherwise drop the event (including the case when expression could not be evaluated because the event doesn’t have field(s) used in the expression).
Invalid values of filter options in server sections
Invalid values are generally skipped/ ignored and defaults are used which will result in all events being collected. Some examples of invalid values for the filter option are:
- filter = {filelog; samplez.*; facility >7 || level == “error”} – If there is no log files called samplez.* events will not be collected
- filter = {filelog; sample.*; facility <0 || level == “error”} – There are no events with facility less than zero and no events will be collected.
- filter = {filelog; sample.*; facility >7 && level == “nolevel”} – There are no events with a level called nolevel (assuming nolevel is an invalid value) AND facility greater than 7 so no events will be collected.
- filter = {winlog; Security;. } – ‘.’ Is invalid option for event_filter and all events collected by agent will be sent to the server and filter is ignored.
- filter = {mylog; sample.*; facility >7 || level == “error”} – although the collector filter and event filter are valid ; mylog is not a valid collector_type; so all events collected by agent will be sent to the server and filter is ignored.
- filter = {winlog; NoName; } – If there is no winlog section called ‘NoName’ then events are not collected and sent to the destination.
- filter = {winlog; System } – there is no ‘;’ after system this is not a valid format and all log events collected by agent will be sent to the server.
And last but not the least all errors will be reported in agent log file. Consult the log file if an unexpected behavior is encountered and fix all errors by reported agent.
Note:
- All filter options case sensitive.
- debug_level = 2 gives verbose information about errors as in earlier versions of the agent.
- These options cannot be used to send importer events to multiple destinations via the Importer tool.
- GUI does not support applying of multi-destination in server options to an agent yet in vRealize Log Inisight v4.5
