Happy New Year! You may have noticed in late December some new Log Insight content packs were published to the marketplace. In this post I would like to talk about which content packs were released and the value they provide.
Background
A total of 5 content packs were released:
- Apache – CLF
- Apache – HTTP Server
- Apache – Tomcat
- HAProxy
- NGINX
Of the ones listed, the only one you may not recognize is the Apache – CLF content pack. What is CLF and why should you care? CLF stands for Common Log Format and is a logging standard established by the Apache Foundation. While the standard was created for Apache web applications, the standard has been adopted by most other web applications including HAProxy, NGINX and Microsoft IIS.
Given that most web servers follow CLF the options for creating content packs for web applications were to either duplicate the same dashboards, queries, alerts and extracted fields for each application or create a central content pack that could be used by all. Having a single content pack makes the most sense and thus the Apache – CLF content pack was born. But how would you know the Apache – CLF content pack works for NGINX web traffic logs just be searching the marketplace? The solution was to list each web application’s content pack individually, but have them all indicate that the Apache – CLF content pack is needed.
For example, select the Apache – Tomcat content pack and you will see the following description:
Go ahead and install the content pack and you will see the following directions — assuming you are running Log Insight 3.0 or newer:
While you can install the content pack, you will see it does not contain any content:
Instead, you need to install the Apache – CLF content pack for content:
Installation and Configuration
Now that you know the Apache – CLF content pack is needed for all web applications that follow the CLF log format, you can go ahead and install the Apache – CLF content pack from the marketplace. Once this is done, assuming you are running Log Insight 3.0 or newer, you will be presented with directions on how to configure the content pack:
The directions indicate that the use of the Log Insight agent is required on web applications for which events are to be collected and monitored. Once you have installed the agent and configured it to point to you Log Insight instance you can then configure the agent using the agent groups included in the Apache – CLF content pack — again assuming you are running Log Insight 3.0 or newer. To do this you would go to the Administration > Agents, page as described in the instructions, and select the All Agents drop-down menu. You will see the Apache – CLF content pack contains agent groups for different web applications running on different operating systems or with specific log formats. Select the Copy Template button for the ones that are applicable to you environment one at a time:
Rename as desired and select the Copy button:
Add dashboard filters to restrict which agents received the configuration. For example, perhaps I want all of my agents running on Linux to receive the configuration so I will use the OS filter. Be sure to select the Refresh button (the background of the button will be blue if changes have been made, but not applied) and confirm the agents you expect are returned as they will be the ones that receive the configuration:
Go through the Agent Configuration and search for comments which start with a semicolon and are colored in brown. Comments indicate information that you need to know and changes you may need to make in order to properly collect logs:
Note while the above example talks about changing the format of some parsers, the most common comment is to adjust the directory option of filelog configuration sections. This is because it is common for logs to be written to dynamic directories names which are often created during installation or configuration. Be sure agent configuration comments are validated and adjusted as appropriate or the dashboards and alerts may not work as expected.
Finally, you need to select the Save New Group button to ensure the agents that match the filters specified receive the configuration.
Usage
Now that you have the content pack installed and configured you may be wondering what value the content pack brings. Most of the value comes in the form of different dashboards. Given the primary focus of the content pack is on CLF events let me quickly cover analytics, access and error events.
First, the Overview dashboard provides a Google Analytics like view. It provides you with information such as pageviews and popular pages.
In addition, the Visitors dashboard tells you more about the demographics of the people visiting your site. Most of this information is parsed from the user agent information contained within web events.
The Errors dashboard is specific to the error log found on most web applications. It is a good indication of problems within your environment.
Finally, and possibly the most interesting, is the Status dashboard which contains information about HTTP status codes returned. Like Errors, this dashboard is a good indication of problems within your environment.
Summary
As you can see, installing and configuring the Apache – CLF content pack is quick and easy. The content pack works for a variety of different web and load balancing applications and requires the use of the Log Insight agent. Once everything is in place, you will immediately get rich analytics from your web application events. This content pack is available for free from the in-product marketplace so be sure to try it out today!
