Create Parsers for VMware Aria Operations for Logs Agent

VMware Aria Operations for Logs Agent is a powerful tool for collecting infrastructure and application logs. There are three ways to collect logs:

• Windows Event Logs
• File Logs
• Journal Logs

This article provides information about File Logs and shows how to create parsers.

You need to specify multiple parameters in the File Log configuration, and one of them is event_marker. The default value of event_marker is the new line (\n), which means that the Agent separates the logs by new lines, and you see them in the Operations for Logs server (or any other log management tool) as separate log events. This is good for applications with a one-line logging mechanism, but in practice, there are multiple examples of a log containing multiple lines, for example, the exceptions:

[2023-02-15 09:49:12.478+0000] ["netty-event-loop-77"/10.10.10.10 ERROR] [play.core.server.netty.PlayRequestHandler] [Exception caught in Netty]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-codec-4.1.72.Final.jar:4.1.72.Final]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
	at sun.security.ssl.Alert.createSSLException(Alert.java:68) ~[?:1.8.0_351]

In this case,  define the event_marker as follows to avoid getting each line of an exception trace as a separate log event:

         event_marker=\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}

So, every time the Agent hits the log with this pattern, it identifies it as a new log. As a result, you need to use the following config:

         [filelog|netty]
         directory=/my/directory/
         include=my_file.log*
         event_marker=\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}

You can leave this config as it is, but you can also use the power of parsers.

The next parameter in Agent configuration is the parser. 

Agent-side log parsers extract structured data from raw logs, and in the Operations for Logs server, you have the logs with correspondingly parsed fields instead of extracting the fields on the server side. If the syslog protocol is used, fields extracted by parsers are part of STRUCTURED-DATA according to RFC5424. Agent-side log parsers extract structured data from raw logs. There are multiple types of parsers supported by VMware Aria Operations for Logs Agent.

Add a parser to the previous example:

         [filelog|netty]
         directory=/my/directory/
         include=my_file.log*
         event_marker=\[\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}
         parser=my_parser_netty

Now you need to identify my_parser_netty. In this case,  you can use CLF (Common Log Format) as a base parser.

         [parser|my_parser_netty]
         base_parser=clf
         format=[%t] [“%{thread}i”/%{ip}i %{netty_severity}i] %M

Here, thread, ip, and netty_severity fields are defined and appear under each corresponding log in the Operations for Logs server. For this example the values of the fields are as follows:

         %t=2023-02-15 09:49:12.478+0000 
         %{thread}i=netty-event-loop-77 
         %{ip}i=10.10.10.10 
         %{netty_severity}i=ERROR

The parser gives you an opportunity to parse these fields too, for example, the timestamp (%t). You can easily define the type of timestamp used in the log. In this example, you can use the following:

         [parser|my_parser_netty]
         base_parser=clf
         format=[%{%Y-%m-%d %H:%M:%S.%f%z}t] [“%{thread}i”/%{ip}i  {netty_severity}i] %M

For the timestamps with the following type:

###<Feb 15, 2023 6:18:10 AM UTC> <Notice> <Stdout> kafka.consumer.offset.OffsetMonitor:Processing message from topic

Use this definition:

         format=###<%{%b %e, %Y %H:%M:%S %p %z}t> %M

The logs with standardized timestamp:

2023-01-05T09:03:33.789214-06:00 prexesapp34 Calling PDP Hierarchy endpoint

Will have a parser with the following format:

         format=<%{%Y-%m-%dT%H:%M:%S.%f%z}t> %M

When the timestamp is parsed correctly in the Agent’s side, the logs appear in the Operations for Logs Server side with exactly the same timestamp. Note that in the Operations for Logs server, you see two timestamps (see Fig. 1):

Consider the first example from Fig. 1.

A Operations for Logs user has created a parser for the timestamp and generated the log at 10:21(UTC+4) their time. Note that the user used the log’s timestamp with a 3 minute difference from the real time – 6:18:10 (UTC).
As a result, Operations for Logs places the generated log exactly with the same timestamp (see VMware Aria Operations for Logs’s time) – 8 Feb 2023 10:18:10.000 (UTC+4)

In the Agent’s log, the user got these two lines as a confirmation that their created parser for the timestamp is working fine:

2023-02-08 10:21:21.000636 0x00007f50fbfff700 <trace> FieldParser:50     | Running parser 'my_parser_netty' on input msg "###<Feb 8, 2023 6:18:10 AM -0000> <Notice> <Stdout> time is 10:21"

2023-02-08 10:21:21.000811 0x00007f50fbfff700 <trace> FieldParser:83     | Parser 'my_parser_netty' returned the following fields: timestamp=2023-02-08T06:18:10, log_message="<Notice> <Stdout> time is 10:21"

If the parser was not correct, the user would get their log with the 8 Feb 2023 10:21 timestamp.

Do you want to manage the ingestion time of your infra logs? Create right parsers.
Do you want to manage the fields on flight before ingesting logs into VMware Aria Operations for Logs? Create cool parsers.   

You can also learn more about other VMware Aria Operations for Logs On-Prem features on TechZone !

Got feedback? We’d love to hear it! You can leave a review in comments.

Happy logging!