By Bahubali Shetti, Director of Public Cloud Solutions for VMware Cloud Services at VMware
Observability is one aspect of managing Kubernetes clusters. Observability involves gaining insight into multiple data points/sets from the cluster and analyzing this data to resolve issues. Observability covers three main data sets:
- Metrics — This includes metrics from both the cluster, through cAdvisor, metrics server, and/or Prometheus, along with application data.
- Logs — Whether it’s cluster logs, or application log information like Syslog, these data sets are important analysis.
- Tracing — generally obtained with tools like Zipkin, Jaeger, etc. and provide detailed flow information about the application.
Logs are of particular interest since there is an abundance of these, and a significant amount of information can be analyzed from both cluster and application logs. One issue in analyzing logs, is properly aggregating log into a singular location making it easier to cross reference and correlate logs from multiple nodes, pods, containers, and even between multiple clusters.
There are two main solutions:
- Custom built – A popular solution is to use a singular Prometheus aggregation instance, gathering data from all the individual Prometheus cluster instances
- Commercial SaaS service – Splunk, Logzio, AWS Elasticsearch, etc.
Custom built solutions are a great option due to the controls they provide, but more than likely it’s easier to use a SaaS based solution.
I explore how AWS Elasticsearch can be used as a SaaS based log aggregation solution using two different yet similar data collectors:
- fluentd – https://medium.com/@bahubalishetti/aggregating-application-logs-from-kubernetes-clusters-using-fluentd-to-log-intelligence-91da5f536692
- fluentbit – https://medium.com/@bahubalishetti/configuring-fluentbit-on-kubernetes-for-aws-elasticsearch-bec486bcc727
In each of these two blogs I describe how to properly configure AWS Elasticsearch and configure Fluentd and Fluent Bit on VMware Cloud PKS to properly forward logs.
- Fluentd configuration for VMware Cloud PKS – https://github.com/bshetti/fluentd-helm-vke
- Fluent Bit configuration for VMware Cloud PKS – https://github.com/bshetti/fluentbit-setup-vke
Please feel free to reach out to me @Shetti