12 Reasons Why You Should Use The Log Insight Agent

When the Log Insight Windows agent was released in version 2.0, the decision to use the agent was easy because Windows does not natively support syslog. Given the release of the Log Insight Linux agent, I have been asked a few times why the agent should be used over already available syslog agents like Rsyslog and Syslog-NG for sending events to a remote destination like Log Insight. I would like to cover 12 reasons in this post.

1. The agent supports sending events over syslog and over Log Insight’s ingestion API – this means the agent will work with any remote syslog destination.
2. The agent is free and fully supported by VMware – if you have a problem with a third-party agent then you will need to get support from them. With the Log Insight agent you get support from VMware. This provides an end-to-end solution from client to server and at no additional cost.
3. The agent properly handles rotated log files – based on tests I have performed, non-enterprise third-party syslog agents like the free versions of Rsyslog and Syslog-NG have several limitations around file rotation. As you may know, logs get written to a file and over time the file gets rotated. The problem arises when the file gets rotated as it is possible that the syslog agent has not collected the most recent logs since the rotation. The result is that some logs never get sent to the remote syslog destination. As long as the include option in the Log Insight agent configuration includes rotated log files, then the Log Insight agent does not suffer from this issue as it keeps track of where it left off as well as all rotated files. For example, you should use something like:

   [filelog|vCenterMain]
   directory=C:ProgramDataVMwareVMware VirtualCenterLogs
   include=vpxd-*.log*
   exclude=vpxd-alert-*.log;vpxd-profiler-*.log
   event_marker=^d{4}-d{2}-d{2}[A-Z]d{2}:d{2}:d{2}.d{3}

   instead of this (note the missing asterisk at the end of the include option):

   [filelog|vCenterMain]
   directory=C:ProgramDataVMwareVMware VirtualCenterLogs
   include=vpxd-*.log
   exclude=vpxd-alert-*.log;vpxd-profiler-*.log
   event_marker=^d{4}-d{2}-d{2}[A-Z]d{2}:d{2}:d{2}.d{3}

   Note: The include option defaults to all files within a directory (*), which means it will handle rotated files automatically.

4. The agent supports multi-line messages – the syslog RFC does not provide a standard on how to support multi-line messages. Many syslog agents provide a solution to this problem (see this post for more information), but they require newer versions of the agent and can be difficult to configure. The Log Insight agent offers an event_marker option and supports Perl-based regular expressions to define what an event looks like. For example:

   [filelog|vCenterMain]
   directory=C:ProgramDataVMwareVMware VirtualCenterLogs
   include=vpxd-*.log*
   exclude=vpxd-alert-*.log;vpxd-profiler-*.log
   event_marker=^d{4}-d{2}-d{2}[A-Z]d{2}:d{2}:d{2}.d{3}

   Note: The event_marker options defaults to newline (n).

5. The agent is extremely resource efficient – the agent was built for performance. In testing I have done, the agent never consumes more than 6% of the available CPU of the system and uses less than 300 MB of memory. For typical clients (<1000 EPS) resource utilization is normally under 2% of the available CPU and less than 50 MB of memory. I have seen syslog agents bring down systems before so this is huge!
6. The ingestion API provides client-side compression – In testing I have done, this can lead to up to 16 times less traffic being sent over the wire, with 5 times less traffic being the average. Even though log network bandwidth is low, this is huge!
   

   Note: Free syslog agents like Rsyslog and Syslog-NG do not offer compression.

7. The ingestion API provides encryption over TCP/9543 – because security matters.
8. The ingestion API provides the ability to add metadata to events – sometimes you may wish to add additional information to an event to make it easier to query for or correlate over. While many syslog agents provide a solution to this problem (see this post for more information), they do so by manipulating the actual event instead of providing supplemental or metadata information. The Log Insight agent offers a tags option and supports JSON key/value pairs for specifying tags. For example:

   [filelog|vra]
   directory=/var/log/vmware/vcac
   event_marker=^[^s]
   tags={"vmw_product":"vra","vmw_product_component":"cafe"}

   Note: You can also specify a tag key of “appname”, which will override the appname of the event in Log Insight, but only to the static field and not the actual event.

9. The ingestion API provides the ability for client-side timestamps to be used (assuming they are within 10 minutes of the server time) – with the syslog protocol, Log Insight uses the server ingestion time as the client event may not contain a parsable timestamp. Since the agent can be trusted to provide time in a specific format, the time within the event or the time on the client device if the event does not have a timestamp can be sent and used by Log Insight. In the case of significant clock drift between the client and server (greater than 10 minutes), the client time will be ignored and the server time during ingestion will be used just like with the syslog protocol.
10. The ingestion API provides the ability for the server to throttle events from the client (i.e. back pressure) and for the client to ensure no message loss during intermittent connection issues – back pressure is handled by sending a 503 HTTP status code while connection issues can be detected and prevented against through REST calls. The Log Insight agent provides a configurable disk-backed queue used to save events until the server has acknowledged ingestion. This disk-backed queue prevents drops of ingestion API events during intermittent connection issues where syslog events already sent, but not acknowledged by the server, can be lost after the TCP ACK window expires. Free versions of syslog agents like Rsyslog and Syslog-NG do not provide back pressure functionality nor do they provide a disk-based queue (some do provide a configurable in-memory queue). In addition, these agents rely only on the retry abilities of the TCP protocol (e.g. TCP ACK window) if the TCP protocol is being used to send the traffic–none of this is available for UDP.
11. The ingestion API provides the ability to collect and see agent statistics from the Log Insight UI – giving you centralized monitoring.
12. The ingestion API provides the ability to push agent configuration from the Log Insight UI – giving you centralized management.

Summary

As you can see, the Log Insight agents and the Log Insight ingestion API both offer significant benefits over other syslog agents and the syslog RFC. In my opinion, if you are already using the Log Insight server, then you should also be leveraging the Log Insight agent for all Windows and Linux devices.