Just imagine you are an IT guy, working hard in an air conditioned windowless (home-) office – your social life is limited to slack, skype, socialcast or some other social media plattform. Wouldn’t it be nice to pretend, that you participate in the life outside and chat about something not work related – for example about the weather.
In this small example, I will show how to retrieve world wide weather data for any city and visualise it in Log Insight without leaving the windowless office. So in the next chat with your colleague in Seattle you may surprise him with an accurate rain prediction.
I will gather data from the excellent and partially free openweathermap.org service – and we will do it for an array of cities. This data will be written to a log file and collected by the Log Insight agent on a Linux system. In Log Insight we will create example dashboard for the “World wide current weather” and a second dashboard for the next day forecast. If you decide to take a shortcut and import the provided example content pack, you can skip the field extraction and dashboard building, i.e. steps 6 and 7.
Step 1: Get your own personal api key from http://openweathermap.org/appid – just create a free account (sufficient for the example below) and get an API key.
Step 2: In the next step we use the search on the openweathermap.org and write down a list of locations we will collect weather data. You may use geo coordinates or city codes, but in most cases a name the format “CityName,CountryCode” will be sufficient. My example list look like this: “Yerevan,AM Seattle,US PaloAlto,US Sofia,BG London,UK Frankfurt,DE Sydney,AU”
Step 3: Now we will test the API key with a curl request. Please replace the api key in the command below with the key you retrieved in step 1. In the US (for Fahrenheit) you may prefer to use “units=imperial” instead of “units=metric” (for Celsius), if the units= is ommitted, Kelvin will be used. If you receive an error, check the doublequotes and check if you are using your own api key retrieved in step 1.
root@ugurke:~# curl -s "http://api.openweathermap.org/data/2.5/weather?q=Frankfurt,DE&mode=xml&units=metric&cnt=7&APPID=905730596699e6cd7d4c70a5eb8abb86"
<current><city id="2925533" name="Frankfurt am Main"><coord lon="8.68" lat="50.12"></coord><country>DE</country><sun rise="2016-06-22T03:15:43" set="2016-06-22T19:39:05"></sun></city><temperature value="19.58" min="16.67" max="22.22" unit="metric"></temperature><humidity value="83" unit="%"></humidity><pressure value="1005" unit="hPa"></pressure><wind><speed value="1.03" name="Calm"></speed><gusts value="2.57"></gusts><direction value="288" code="WNW" name="West-northwest"></direction></wind><clouds value="12" name="few clouds"></clouds><visibility></visibility><precipitation value="19.56" mode="rain" unit="1h"></precipitation><weather number="503" value="very heavy rain" icon="10d"></weather><lastupdate value="2016-06-22T06:16:02"></lastupdate></current>
Step 4: We will automate the process of data retrieving in a simple shell script. This script below is just a very basic example. You may want to tweak it. You will need to provide your API key. You may want to use “imperial” units for Fahrenheit. This script will do a separate curl call for each city, gather data in XML and add the output to the file /var/log/current.log and /var/log/forecast.log.
#!/bin/bash # Add your own API key from http://openweathermap.org/appid APIKEY=905730596699e6cd7d4c70a5eb8abb86 MODE=xml #json or xml UNITS=metric #metric or imperial COUNT=1 #forecast days # Add City,CountryCode to the list of cities for CITY in Yerevan,AM Seattle,US PaloAlto,US Sofia,BG London,UK Frankfurt,DE Sydney,AU do CURRENT=`curl -s "http://api.openweathermap.org/data/2.5/weather?q=$CITY&mode=$MODE&units=$UNITS&cnt=7&APPID=$APIKEY"` FORECAST=`curl -s "http://api.openweathermap.org/data/2.5/forecast/daily?q=$CITY&mode=$MODE&units=$UNITS&cnt=$COUNT&APPID=$APIKEY"` echo $CURRENT >> /var/log/current.log echo $FORECAST >> /var/log/forecast.log done
Copy the script to any linux box with log insight agent installed, make it executable (chmod +x) and add it to your crontab (crontab -e). I am running the collection every 10 minutes, so the crontab entry looks like this:
*/10 * * * * /usr/local/bin/getweatherdata.sh
Step 5: Now that we have a working cron job logging the weather data, we now need to configure the Log Insight server to collect this data.
On Log Insight Server: create a group or just add the following file directive to the agent configuration and check the results in interactive analytics. (You may skip this step and go directly to step 8 and import the content pack)
[filelog|weather]
directory=/var/log/
include=current.log
tags={"weather":"current"}
[filelog|weather-forecast]
directory=/var/log/
include=forecast.log
tags={"weather":"forecast"}
Step 6: In Log Insight “Interactive Analysis” I used the semi-automated feature “field extraction” to mark, extract and name the fields used in the weather dashboards. XML is not a log format, so some manual tweaking was required. With the fields extracted and most of them containing numeric data it is easy to create colourfull and meaningfull dashboards. The following screenshot shows the automatic extraction of temperature value. (You may skip this step and go directly to step 8 and import the content pack)
Step 7: Now, I can take any extracted data field and visualise it in the dashboard. This example shows the extraction and visualisation of the cloudiness data field.
And finally I have put this world weather station dashboard together:
Step 8: If you want to have a shortcut: feel free to use my example dashboard and fields included in this importable content pack. This will work with evaluation and full versions of Log Insight. It will not work with Log Insight “vCenter edition”, because in this edition the import button is missing.
After the import, you will just need to clone the agent group and apply it to the Linux system running the getweatherdata.sh script.
If everything works as expected, you should get two dashboards: one for the current weather and one for the next days forecast:
Some ideas for further enhancements:
- You could create an alert for certain forecasted weather conditions, e.g. “Forecast: A sunny day in Seattle” and automate your time off requests through a webhook.
- Enhance the forecast for up to 16 days, you may need to redesign the dashboard and re-define some fields
References: you will find information on how to build your own content packs here: https://developercenter.vmware.com/web/loginsight I also recommend Steve Flanders blog: http://sflanders.net/log-insight/
Download link for the unofficial World Weather Content Pack
