We can all recognize and agree on the benefits of operations management, allowing you to become more proactive and to quickly address and resolve building performance, configuration, and capacity problems across physical, virtual, and cloud environments. So what added value would log analysis provide, and why isn’t everyone using this data proactively today? In this post we will identify the “gaps” and challenges that organizations face by not analyzing machine data, as well as the clear benefits gained by adding a centralized log management solution.
Structured vs. Unstructured Data
It’s important to understand the difference between “Structured” vs. “Unstructured” data. Traditional monitoring tools are focused on consuming and analyzing “structured” data: raw performance and capacity metrics, for example. However ignoring logs will effectively leave you with visibility into a subset of IT data available. This leaves a large gap by not analyzing “unstructured” data: the plethora of application and system logs that contain a goldmine of information useful for identifying and troubleshooting problems proactively, identifying root-cause analysis, and quickly resolving issues.
Log Analysis: The Last Mile
The importance of analyzing unstructured data is undeniable. So why is this practice not ubiquitous? The answer to this question lies in the fact that in many cases manually digging through logs is simply overwhelming! Machine-generated log data is massive in scale, and difficult to capture and manage. An average vSphere Host generates roughly 250 MB of logs per day. Now take a common application like Microsoft Exchange: larger Exchange deployments generate upwards of 1GB of logs per day! Do you recall that classic image of Bill Gates sitting on a mountain of paper a mile high, holding a 650MB CD-ROM disk? Now extrapolate that out to tens, hundreds, and even thousands of applications, operating systems, storage, and network devices – all generating enormous amounts of log data. According to Lexisnexis, 1GB is the equivalent of about 677,963 pages of text! Simply put: manually digging through logs is simply time-consuming, complex, and often not worth the hassle.
When faced with troubleshooting, an IT Admin is sometimes forced to remote into 1 or more systems and spend hours digging through and correlating logs, trying to find the needle in the haystack, troubleshooting “the last mile” to pinpoint the root cause and gain insight into specifics of what is going on. How many man hours are wasted that could be better spent achieving IT or business objectives? How are these organizations ever going to achieve being strategic in their IT initiatives or drive competitiveness when IT Ops are wasting dozens of man hours every week manually reviewing and archiving logs, hunting down system events when facing unexpected application and system downtime, and playing the blame game, finger-pointing at other IT silos? In reality these types of issues unfortunately compound even further with increased pressure to reduce costs, improve ROI, and reduce downtime using existing point-solutions. There is often a result of poor end-user satisfaction with the IT organization as IT Admins face lack of time, skills, or toolset to properly manage service health and IT policies. It’s counterintuitive.
Use Cases
Let’s evaluate a few use cases of log analysis that would transform today’s existing pain points into achieving operations nirvana:
IT Operations Management
- Gain end-to-end visibility, speed up MTTR with troubleshooting across consolidated logs from applications, network, storage, infrastructure, cloud.
- Align IT with the business by freeing up lost time spent manually digging through logs, focus on strategic projects.
- Reduce unexpected system downtime, drive end user satisfaction with IT organization.
- Facilitate troubleshooting and Root Cause Analysis
- Avoid finger-pointing across IT silos
Compliance Benefits
- Free up time consuming, manual processes of manual auditing & reviewing logs
- Avoid costly fines and restrictions imposed by failing audits
- Adhere to internal best practices and mandated regulatory compliance standards
- Some Examples:
- Regularly reviewing log files at specified intervals:
- PCI-DSS 3.0 Rule 10.6
- HIPAA Requirement 164.308(a)(3)
- SOX Requirements Sec 302(a)(4)(C) and (D)
- ISO 27001 Requirement A.12.4.1
- GLBA Section 501
- Availability of audit trail analysis and specified historical log data retention periods:
- PCI-DSS 3.0 Rule 10.7
- HIPAA Requirement 164.308 (a)(5)
- SOX Requirement Sec 302(a)(5)
- Admin/User access to systems must be recorded and monitored for possible abuse:
- PCI-DSS 3.0 Rule 10.2.5
- HIPAA Requirement 164.308(a)(5)
- SOX Requirements Sec 302(a)(4)(C) and (D), 302 (a)(5), and 302(a)(6)
- ISO 27001 Requirement A.9.2.5 and Requirement A.12.4.3
- GLBA Section 501
- Regularly reviewing log files at specified intervals:
DevOps Benefits
- Free up developer time to focus on adding new functionality and increase the value of the app rather than wasting time troubleshooting latency/performance
- Accelerates releases, avoids delays due to unexpected issues
Security Benefits
- Real-time search across consolidated data for security events
- Visibility to unauthorized logins, credential misuse, privilege escalations, and anomalies.
- Easily identify anomalous and potentially malicious activity
- Consolidated historical system logs allow for forensics activities
The Solution:
Ideally, logs should be used proactively to provide early warning and prevent future downtime. An effective log management solution should not require a team of dedicated resources to understand and use, and should not be cost-prohibitive for environments of any size. VMware’s vRealize Log Insight delivers real-time log management, with machine learning-based analytics, and high performance search, enabling faster troubleshooting and better operational analytics across physical, virtual, and cloud environments. It can analyze terabytes of logs, discover structure in unstructured data, and deliver enterprise-wide visibility. Combined with vRealize Operations platform, you are able to bring both structured and unstructured data together, offering a significantly enhanced solution for end-to-end operations management.
Added benefits, Differentiators
A major differentiator of vRealize Log Insight is the rich integration with vRealize Operations Manager, combining the power of Log analytics with structured data analysis to provide you with that comprehensive 360-degree view, extending operational visibility and proactive management across infrastructure and applications in a dynamic, hybrid cloud environment. Tight integration between Log Insight and vR Ops also facilitates seamless transition from monitoring to troubleshooting, with unique integration benefits such as inventory mapping, 2-way launch in context between Log Insight and vR Ops, virtual object relationship event tagging, and even surfacing alerts up into vR Ops – enabling you to respond to issues quicker, significantly improving MTTI/MTTR.
Packaging:
Competitors’ pricing based on log data volume actually leads customers to purposely leave out specific logs or even omit monitoring entire systems in order to keep the price down – effectively crippling their ability of being comprehensive in their monitoring strategies. On the contrary, Log Insight is available in simple, predictable pricing and packaging options, based on environment size – not log data volume.
vRealize Log Insight (standalone)
VMware vRealize Log Insight delivers real-time log management, with machine learning-based Intelligent Grouping, high performance search, and better troubleshooting across physical, virtual, and cloud environments.
Licensing available in a “Per Operating System” or a “Per CPU” basis.
Purchase Log Insight licenses *direct* using the VMware online store.
VMware vRealize Operations Insight delivers intelligent operations from applications to storage – across vSphere and physical hardware – for businesses of all sizes.
An add-on offering to VMware vSphere with Operations Management. Available on a “per CPU basis.”
VMware vRealize Suite is a cloud management platform purpose-built for the hybrid cloud. It provides a comprehensive management stack for IT services on vSphere and other hypervisors, physical infrastructure and external clouds, all with a unified management experience.
