12 Reasons Why You Should Use The Log Insight Forwarder

In Log Insight 2.5, a forwarding option was introduced, which provided similar capabilities to a syslog aggregator. I have been asked a few times why you should use the Log Insight forwarding option. I would like to cover 12 reasons in this post.

1. Forwarders support sending events over syslog (TCP only) and over Log Insight’s ingestion API: this means the forwarder will work with any remote syslog destination. A classic example of the value of supporting syslog is for an existing SIEM product used to analyze security events.
2. Forwarders using the ingestion API send compressed events: In testing I have done, this can lead to up to 16 times less traffic being sent over the wire, with 5 times less traffic being the average. Even though log network bandwidth is low, this is huge! This is a compelling reason to deploy forwarders in every datacenter.
   

   Note: Free syslog agents like Rsyslog and Syslog-NG do not offer compression.

3. Forwarders using the ingestion API support sending encrypted events: because security matters.
4. Forwarders minimize the attack vector of the primary Log Insight instance / remote syslog destination: because security matters. In short, if the only devices allowed to send events to the primary Log Insight / remote syslog destination are the forwarders then security becomes easier. This is a compelling reason to deploy forwarders in every datacenter.
5. Forwarders using the ingestion API can add metadata to events – sometimes you may wish to add additional information to an event to make it easier to query for or correlate over. While many syslog agents provide a solution to this problem (see this post for more information), they do so by manipulating the actual event instead of providing supplemental or metadata information. The Log Insight forwarder offers a tags option and supports JSON key/value pairs for specifying tags. This makes it possible to add information such as which datacenter the event came from.
6. Forwarders using the ingestion API can respond to throttle requests (i.e. back pressure) and can ensure no message loss during intermittent connection issues – back pressure is handled by sending a 503 HTTP status code while connection issues can be detected and prevented against through REST calls. The Log Insight forwarder provides a configurable disk-backed queue used to save events until the server has acknowledged ingestion. This disk-backed queue prevents drops of forwarded events during intermittent connection issues where syslog events already sent, but not acknowledged by the server, can be lost after the TCP ACK window expires. Free versions of syslog agents like Rsyslog and Syslog-NG do not provide back pressure functionality nor do they provide a disk-based queue (some do provide a configurable in-memory queue). In addition, these agents rely only on the retry abilities of the TCP protocol (e.g. TCP ACK window) if the TCP protocol is being used to send the traffic–none of this is available for UDP.
7. Forwarders can server as a backup: events that are forwarded are also stored on the forwarder. In addition, forwarders are complete Log Insight instances and offer the same UI as regular Log Insight instances.This means if the central Log Insight instance / remote syslog destination becomes temporarily or permanently unavailable the events still exist on the forwarder, users can be given direct access to forwarders, alerts can be configured on forwarders, etc. The forwarder local capacity can be configured to store events for as long as desired.
8. Forwarders can be configured to integrate with different vSphere and/or vR Ops instances: forwarders are complete Log Insight instances and offer the same options as regular Log Insight instances. This means that vSphere integration can be configured on forwarders keeping traffic local to vSphere / forwarder. The same applies to vR Ops and makes it possible to support environments running more than on vR Ops instances.
9. Forwarders make it easier to support disaster recovery: again, the forwarder is a Log Insight instance; it contains the events it forwarded and supports querying. In addition, forwarders can send events to up to 10 different destinations. This make it easy to forward events to two different remote destinations (e.g. Log Insight instances) to handle complete disaster recovery. This is important as many devices still only support sending events to a single destination.
10. Forwarders support filtering of events: since forwarders support sending filters events it is possible to send security events to an existing SIEM and all other logs to a central Log Insight instance. In addition, filtering makes it possible to ensure “noise” is not ingested by the central Log Insight instance.
11. Forwarders make it possible to support variable retention periods: again, since forwarded events can be filtered, it is possible to send security events to one Log Insight instance where events are kept for 90 days on the remote Log Insight instance and all other events to a separate Log Insight instance where events are kept for 30 days.
12. Forwarders can be used as an alternative to archiving: forwarders store events locally, which is similar to having an archive. In addition, forwarders can send some/all events to a remote destination that acts as an archive. The benefits to this approach is that the “archive” is online and searchable immediately if needed. The downside to this approach is that it requires additional resources or more disk space than just archiving. In order to properly handle the archiving use-case this means the forwarder or the secondary instance has local capacity greater than the primary instance.

Summary

As you can see, the Log Insight forwarder feature, especially in conjunction with the Log Insight ingestion API, provides several benefits. In my opinion, if you are already using the Log Insight server, then you should also be leveraging the Log Insight forwarder in front of it.