vRealize Log Insight is VMware’s log analytics tool which allows IT administrators to rapidly troubleshoot their applications and infrastructure. Intuitive search capabilities combined with dashboards makes data extraction easier and dare I say, a little fun! vRealize Log Insight is about to celebrate its sixth birthday since version 1.0 went GA on July 9th 2013. And the latest 4.8 version marks the 26th release since then!
vRealize Log Insight 4.8 introduces several new features including data retention, new agent features, new vROps dashboards, and UI enhancements. As always, be sure to check out the release notes for a complete list of new features. Here are just a few of my favorites!
New Data Retention Feature
A common request we often get from customers is “can vRealize Log Insight be set up to store x number of days’ worth of logs?” In previous versions, the answer was that you can get fairly close by sizing your environment appropriately. This roughly meant figuring out what your daily ingestion rate is and multiplying by the number of days to calculate your storage requirements. In vRealize Log Insight 4.8, this is no longer the case as we’ve introduced Data Retention to address these requests.
The Basics of Log Insight Storage
Before we dive into this exciting new feature, let’s take a look at the fundamentals of how vRealize Log Insight data retention works.
Log Insight ingests log data via Syslog or its own ingestion API known as CFAPI. When a new log message is received, it gets parsed and stored in a bucket.
Once a bucket reaches 500 MB in size, it gets marked as read-only and a brand-new bucket is created for new messages. If archiving is enabled, then a copy of the full bucket gets written to the NFS share at this time. This process repeats until Log Insight runs out of storage.
Without the new data retention feature, Log Insight will continue to fill buckets until it runs out of storage. At that point, the oldest bucket is deleted to make room for a new bucket. If archiving is enabled, then the archive copy will remain untouched on the external NFS share.
A Deeper Dive into Data Retention
Now that we have a basic understanding of how log messages are stored, let’s dive into data retention! This new feature is configured in the General Configuration page and is turned off by default. This means that logs will be saved until Log Insight needs to make room for new buckets. Once enabled, the default value is 12 months, but this can be configured as low as one day. Note that Data Retention applies to the entire Log Insight instance/cluster. It cannot be selectively enabled for a few logs.
With Data Retention enabled, this does not mean that individual log messages will be deleted once they exceed the retention period. Instead, this feature works at the bucket level. Let’s take a deeper look into how this works.
When Data Retention is enabled, it will do an initial cleanup within the first hour and subsequent checks will be run daily. When Log Insight runs a cleanup, it takes an inventory of the most recent log message in each bucket. If the most recent log message in a read-only bucket is older than the retention period, then that bucket gets deleted. Any data that has been archived to an external NFS share will remain untouched.
For example, let’s say we have Data Retention enabled and set for seven days. Bucket 0, which is our oldest bucket, contains log messages that are eight days or older. This bucket will be deleted. However, bucket 2 contains log messages that are as old as eight days and as recent as four days. Since the newest logs in bucket 2 are newer than our seven-day retention period, it will not be deleted. Bucket 3 is still open as it has not reached 500 MB and will not be considered for deletion regardless of the age of its log messages.
As you can see, depending on your log ingestion rate it’s possible for log messages to persist days or even weeks beyond the configured retention period. The higher the ingestion rate, the faster the buckets fill. And the faster log messages get aged out.
vRealize Operations User Audit Dashboards
vRealize Operations 7.5 introduced new user audit logs which track things such as logins and content modifications. In keeping up with the tight integration between vRealize Log Insight and Operations, 4.8 includes two dashboards to visualize these new logs.
The first dashboard is the Activity Audit dashboard. This provides a breakdown of activity by object types such as Dashboards and Alerts. In the example above, you can see that someone created, deleted, and exported dashboards in the last 24 hours. We can also see that new user accounts were created and old ones were deleted.
The next new addition is the Delete Activity Audit dashboard. This provides a breakdown over time of anything that was deleted such as adapters, alerts, dashboards, super metrics, etc. There’s also a breakdown by user.
Also included in this dashboard are pre-built queries which will take you to Log Insight’s Interactive Analytics and show you the pertinent log messages.
Improved vROps Permission Details
Speaking of vRealize Operations integrations, there is another small but very useful feature. When configuring Log Insight to work with vRealize Operations, there are a number of permissions that are required for the vROps service account (documented here). Previous versions of Log Insight performed a simple validation and either return a pass or failure. Now in 4.8, Log Insight will tell you the exact permissions that are missing for that account! Simple, but very helpful indeed!
Exclude Field Extractions from Searches
Another great enhancement in vRealize Log Insight 4.8 is the ability to exclude field extractions for content packs during searches. By narrowing down the number of fields that Log Insight has to extract during a search, results can be returned much faster. For example, if you’re troubleshooting an issue with your vSphere environment, and you need to pull data for a long period of time such as a week. Since we know we’re just looking for vSphere logs, we can click on the Content Packs dropdown and disable iDRAC, PAN-OS, and other field extractions to return results faster!
New Agent Features – JSON Parsing and Conditional Parsers
The last new feature we’ll be taking a look at is the ability to parse JSON files with the vRealize Log Insight Agent. This gives us the ability to search through JSON files in the same way we search for logs. JSON files get ingested and parsed by the Log Insight Agent and appear in Log Insight complete with extracted fields!
Since not all JSON files are created equal, conditional parsers can be used to extract fields from different types of JSON files! In fact, conditional parsers can be used for any type of parser such as log files. And best of all, this can be configured from within the Log Insight UI! No need to open VI and change config files directly!
Conclusion
vRealize Log Insight 4.8 brings many welcome enhancements to an already great product that’s robust and easy to use. This is just a sampling of some of the biggest new features. Be sure to check out the release notes for more details! Upgrading is simple, so why not do it today?
