This post was co-authored by Nico Guerrera, Sr. Technical Account Manager and Chris McClanahan, Sr. Technical Marketing Manager.
If you haven’t heard of Logstash, it’s a powerful open source log and event collection tool. It collects events via a set of configurable plugins. Once input is collected by a plugin, it can be processed and then sent to any number of external log aggregation programs that support syslog (and many other kinds of) ingestion, such as our own vRealize Log Insight and Log Intelligence.
In this blog we will learn to configure Logstash to send Apache access logs to a VMware Log Intelligence SaaS logging service using both our basic syslog protocol and TCP.
Installing Logstash on Linux
As mentioned above, Logstash is configured through a number of plugins that support different output types like stdout, http, tcp, and syslog. We will be using syslog in this example as it’s an open standard supported by our Log Intelligence collector appliance.
Logstash is a simple install and the instructions will depend on what version of Linux you are running. Elastic has a very good Logstash install page here for you to follow if necessary.
Once installed, we will want to download and install the syslog output plugin for Logstash:
Installing the plugin simply involves running logstash-plugin install logstash-output-syslog in Logstash’s bin directory.
Configuring Logstash to Forward Events via Syslog
Now that we have our syslog output plugin installed, we can write a basic configuration file to send our apache access logs to a VMware Log Intelligence collector for forwarding to our cloud instance of Log Intelligence. If you want to read up on how the Log Intelligence remote collector works, this blog on VMware Log Intelligence is a good place to start.
Next, let’s look at a basic configuration file we use to forward events on to our collector.
input {
file {
path => “/var/log/apache2/access.log”
start_position => “beginning”
}
}
filter {
if [path] =~ “access” {
mutate { replace => { “type” => “apache_access” } }
grok {
match => { “message” => “%{COMBINEDAPACHELOG}” }
}
}
date {
match => [ “timestamp” , “dd/MMM/yyyy:HH:mm:ss Z” ]
}
}
output {
syslog {
#Log Intelligence forwarder
host => [“192.168.1.111”]
port=> [“514”]
}
stdout { codec => rubydebug }
}
In this configuration, we are parsing /var/log/apache2/access.log from the beginning and then filtering to break events down into readable and searchable pieces. We are then forwarding the entries from the logfile to Log Intelligence. In my lab I created a configuration file called logstash-syslog.conf with the above configuration.
Once we have the config file build, we can start Logstash and call the config file with /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-syslog.conf
Now that the agent is running, we can generate some logs by opening a web browser to our local Apache default page. This will generate some access events that will then show on your Linux console if you didn’t run the agent as a background process.
Every time your refresh the page you should see the parsed access log output in your console.
Checking vLI for Events
Now that we have created some logs, let’s check Log Intelligence to make sure they’ve been properly forwarded. We should be able to see the events in LI once we start forwarding to the LI collector.
The screenshot above shows that our events are forwarding, and we have them available to query and alert on in Log Intelligence. This process should work for our standard VMware private cloud deployments.
Forwarding to VMware Log Intelligence from a Cloud Server Running Logstash
What if we are running AWS, Azure, or some other Linux workloads hosted by a public cloud provider, and we want to forward events to Log Intelligence? Since the Log Intelligence collector might not be an option, we can forward our events from Logstash via the http output plugin to Log Intelligence using the Log Intelligence API. This will require us to modify our Logstash configuration file a bit, and request an API token from Log Intelligence for basic authentication by the Logstash forwarder.
First thing we need to do is install the http output plugin for Logstash, so we can send events to Log Intelligence via API. We install the plugin just like our syslog one in the exercise above.
Now that we have our plugin, we need to create a new configuration file:
input {
file {
path => “/var/log/apache2/access.log”
start_position => “beginning”
}
}
filter {
if [path] =~ “access” {
mutate { replace => { “type” => “apache_access” } }
grok {
match => { “message” => “%{COMBINEDAPACHELOG}” }
}
}
date {
match => [ “timestamp” , “dd/MMM/yyyy:HH:mm:ss Z” ]
}
}
output {
stdout { codec => “rubydebug” }
http {
url => “https://data.mgmt.cloud.vmware.com/le-mans/v1/streams/ingestion-pipeline-stream”
headers => {“Authorization” => “Bearer ouDo4uxxavnFvZFbqYSkphduDCBn4oZw“}
http_method => “post”
format => “json”
}
}
This is basically the same config file as our syslog config above, but the output section uses http instead of the syslog protocol. We use post as the http method, and json as the format. For the url and API key (highlighted) we need to request those from Log Intelligence.
Start by logging into Log Intelligence and navigating to the API Key section on the left.
Click on ‘New API Key’ to start generating a key for our configuration file.
We name the key something relevant.
Then we receive a URL and key that we want to copy and paste into our Logstash configuration file. Once we fill in the URL and key, we should be able to start the Logstash agent with our new configuration file and start generating Apache access logs to send to Log Intelligence with no collector needed.
Start your Logstash agent with /usr/share/logstash/bin/logstash -f /etc/logstash/logstash-http.conf – I used logstash-http.conf as the name of my configuration file.
We can then refresh our Apache default page a few times, and we should see the parsed Logstash output on our console, just like in the last example with syslog.
Now if we check Log Intelligence for apache access events from the client IP, 192.168.1.110, we should see our events were forwarded via the Log Intelligence API via the Logstash http output plugin.
With our events now in Log Intelligence we can create queries, dashboards and alerts to report on.
Conclusion
Logstash is a very powerful log aggregator with robust filtering and forwarding capabilities. Together with VMware Log Intelligence, we can paint a complete picture of our environment for troubleshooting and root cause analysis purposes, all the way from the application down to the networking and storage. Try it out for yourself, and please comment if you come up with any additional helpful tips or tricks!
Request access to Log Intelligence today and try it for yourself!
