Simplifying agent parser configuration with Global Common Options in vRealize Log Insight

This blog post is about using the new global common options to simplify your agent parser configuration in vRealize Log Insight.

Parsers in vRealize Log Insight

Agents make it possible to specify an optional parser that operates on the log messages and extracts fields from the message. Parsing results in a map of field names to their corresponding values, where “value” means the matched sub-string and its position in the original log message. Parsers as you know can be used on static data as well as live data. Log Insight can then further the experience of data mining by allowing you to query, alert and display data of your interest.

You can configure parsers for both FileLog and WinLog collectors. There are global options that you can configure for all winlog & filelog sections that produce named fields (except the timestamp parser). The list of common global options are tags, parser, directory, include, exclude, exclude_fields and debug.

NOTE: All earlier rules for field names and options are still applicable as before. Field names are restricted and certain field names are reserved and cannot be used as field names. For e.g.

• event_type
• hostname
• source
• text
• timestamp

However as is often seen by administrators when using parser configurations; most of these common options are applicable to all or most of the FileLog and WinLog collectors. Putting them in for each section not only makes the agent config long but also difficult to read and error prone.

So let’s look at a case where the user wants the agent config to have a configuration section where he/she can specify parameters applicable to everything on the agent, i.e. to override the default values of agent configuration. Alternatively they can also have such sections for only FileLog or only WinLog sections to be overridden.

NOTE: At this time this is a client-side only feature. It works only via modification of agent configuration file. There is no way to enable this from server side via content packs or elsewhere.

All this is fine and dandy but how exactly does this work?

The following parameters can be defined in global common sections:   tags, include, exclude, event_marker, charset, exclude_fields and parser.

• eg #1
  • [common|global]
  • tags = {“log_source_vm”:”win-2008r2-64″}
  • exclude_fields = test_tag;some_other_tag
  • parser = auto
  • [common|filelog]
  • tags = {“collector_type”:”filelog”}
  • exclude = *.trc
  • [filelog|channel_1]
  • directory = C:\app\log
  • include = *.log

Here all logs from filelog sections will have both log_source_vm and collector_type tags with their corresponding values. test_tag and some_other_tag tags will be excluded from all logs sent. auto parser will be applied to all collected logs. And all filelog collectors will by default exclude *.trc files from monitoring. Options in [common|global] will also be applied to all winlog sections.

• eg #2
  • [common|winlog]
  • tags={“app”:”global-test”}
  • exclude_fields = test_tag;some_other_tag
  • parser = auto
  • [winlog|wlg_test_section]
  • tags={“app”:”local-test”,”section”:”wlg_test_section”}
  • exclude = *.trc

Note:  If you have tags={“app”:”global-test”} in [common|winlog] and tags={“app”:”local-test”,”section”:”wlg_test_section”} in [winlog|wlg_test_section] then value of “app” tag from [winlog|wlg_test_section] section will override the value from [common|winlog].

Options in [common|winlog] will also be applied to all other winlog sections.

Sample example:

Merging/Overriding criteria

When there are options which are defined in more than one section their values will be merged or overridden and the section with a smaller scope will have a higher priority when merging/overriding. For e.g. value from [common|global] will be merged with or overridden by value from [common|filelog], which in turn will be combined with or overridden by value from [filelog|sample_section]. So, for example, if you have tags={“app”:”global-test”} in [common|filelog] and tags={“app”:”local-test”,”section”:”flg_test_section”} in [filelog|flg_test_section] then value of “app” tag from [filelog|flg_test_section] section will override the value from [common|filelog]. So all logs collected via this filelog section will have additional “app” tag with “local-test” value and “section” tag with “flg_test_section” value.

For winlog sections the chain of priority is the same, with any [winlog|…] section having the highest priority and [common|global] having the lowest priority.

Rules for merging/overriding options are the following:

• Options whose values represent list of values (tags, include, exclude and exclude_fields)will be merged with values for that option from a section with a higher priority. And in case of tags, values of tags from sections with a higher priority will override value of that same tag from a section with a lower priority, as described above.
• Value of options which can have single value (event_marker, charset and parser) will be overridden by values of those options from sections with higher priority.

So value of charset=UTF-8 from [filelog|sample_section] will override global value of charset= UTF-16LE from [common|global].

Invalid values of options in common sections

Invalid values are generally skipped and not merged with values from prior and corresponding filelog/winlog sections. In case of invalid values for tags or exclude_fields options, agent will try to extract as much valid data as it can and skip the rest once invalid data is encountered. All anomalies are reported in the agent log file. Consult the log file if an unexpected behavior is encountered and fix all reported errors by agent.

Invalid values of options in filelog/winlog sections

If the agent detects an invalid value for an option in a filelog or winlog section then it will not merge the option values from that section with option values from common sections and will not enable that section. This is done so that valid value from common sections will not render the section invalid and enable collection from it. This is done to be consistent with prior behavior of the agent; where in the agent disables a section if it has an invalid value for an option. All errors will be reported in agent log file. Consult the log file if an unexpected behavior is encountered and fix all reported errors by agent.

Happy Parsing!!