There are several quoted variations of “You can’t ___ what you can’t measure” being referenced ad nauseum in both business and technological contexts across the internet. Among the adherents to this shifting mantra appears to be the Cloud Native Computing Foundation, which began its existence in 2017 with Kubernetes and three monitoring projects under its banner, including Prometheus. This post will detail the process to deploy Prometheus as a metrics monitoring framework for Kubernetes, in conjunction with Grafana as a visualization engine for those metrics.
Kubernetes
Kubernetes is an open-source system for automating the deployment, scaling and management of containerized applications. Kubernetes is based on technology which was originally designed by Google and is now maintained by the Cloud Native Computing Foundation. Between deployments from the open-source codebase and the commercial offerings based on that code, Kubernetes is becoming a dominant industry platform for containers in production. The items pertaining to extracting and analyzing metrics from the containers themselves may apply to other container ecosystems as well.
How Is Monitoring Kubernetes Different?
Containerized systems such as Kubernetes environments present new monitoring challenges as compared to virtual machine-based compute environments. These differences include:
- The ephemeral nature of containers
- An increasing density of objects, services, and metrics within a given node
- The focus moves to services, rather than machines
- The consumers of monitoring data have become more diverse
- Changes in the software development life cycle
Prometheus is an open source monitoring and alerting tool-kit which is designed to address the shifts in monitoring detailed above. As is frequently the case in the Kubernetes ecosystem, Prometheus will deployed in conjunction with Grafana as a visualization engine for the metrics extracted from the example Kubernetes environment.
VMware Cloud PKSTM
VMware Cloud PKSTM (CPKS) is an enterprise-grade Kubernetes-as-a-Service offering in the VMware Cloud Services portfolio that provides easy to use, secure, cost effective, and fully managed Kubernetes clusters. CPKS enables users to run containerized applications without the cost and complexity of implementing and operating Kubernetes.
At the heart of CPKS is VMware Smart Cluster™. Smart Cluster automates the selection of compute resources to constantly optimize resource usage, provide high availability, and reduce cost. This construct removes the need for educated guesses around cluster definition and sizing for optimal compute resources. It also enables the management of cost-effective, scalable Kubernetes clusters that are optimized for application requirements. These clusters provide built-in high availability with multi-master deployment, routine health checks and self-healing capabilities for Kubernetes clusters. CPKS allows users to run applications in a highly available environment without manual infrastructure configuration and maintenance.
Prometheus
Prometheus is an open-source systems monitoring and alerting toolkit. It is now a standalone open source project and maintained independently of any company. Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted project, after Kubernetes.
Prometheus is well suited to recording time series metrics. It fits both machine-centric monitoring as well as monitoring of highly dynamic service-oriented architectures. In a Kubernetes environment, its support for multi-dimensional data collection and querying is a particular strength. Each Prometheus server is standalone, not depending on network storage or other remote services.
Grafana
Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB. It is a commonly used open source visualization tool for time series metrics.
Visualizing metrics in Grafana revolves around the Dashboard construct. A dashboard is a set of one or more panels arranged into row(s). Each panel is the result of a query run against a data source (such as Pormetheus).
Helm
In the examples detailed here, both Prometheus and Grafana are deployed using the tool Helm. The Helm project describes itself as ‘The package manager for Kubernetes’ and is a Cloud Native Computing Foundation project in its own right. Helm charts represent an easy way to define, install, and upgrade Kubernetes based applications, regardless of the number of services they contain. For a more detail on Helm, please visit the project page.
Building and Preparing the Kubernetes Cluster
In CPKS, the process of creating a new cluster is simple and straightforward. Having already authenticated to the service, creating a new Kubernetes cluster can be easily accomplished via the command line package. In order to create the cluster used for this article, the following command was run:
$vke cluster create -n vcs-demo-cluster -r us-west-2
This command uses two flagged options. The ‘-n’ flag allows the cluster to be named (vcs-demo-cluster) and the ‘-r’ flag allows the region of the cluster deployment to be specified.
Following the successful creation of the cluster, command line access to Kuberenetes will need to be established. To do so, another command will need to be executed:
$vke cluster merge-kubectl-auth vcs-dem-cluster
Assuming the ‘kubectl’ command line utility is already installed for interacting with Kubernetes clusters, this command will merge the active user’s CPKS service identity with the necessary credentials to directly interface with the specified cluster (in this case ‘vcs-demo-cluster’) and change to the kubectl context to reflect that intent.
Once kubectl access has been established, the component of Helm which resides within the Kubernetes cluster (Tiller) will need to be initialized. If Helm is installed on the user’s system, the following command will accomplish this:
$helm init
If Helm is not currently installed, instructions to do so can be found here.
Installing Prometheus
As mentioned above, for the purposes of this article Prometheus was installed on a CPKS cluster via a helm chart. In the previous section, a new Kubernetes cluster was created on CPKS and the Tiller component of Helm was initialized on it. At this point, the cluster is prepared for the installation of the Prometheus Helm chart.
A repository of stable Helm charts is maintained by the Helm project in Github. For the purposes of this article, this chart was used to install Prometheus. The installation was accomplished via this command:
$helm install --name prometheus-release stable/promethues --set server.service.type=NodePort
This single command has several options defined in it. First, all Helm deployments require a release name. Here, the name ‘prometheus-release’ has been chosen. If no release name is specified, one will be generated based on the name of the selected chart. Next, the location of the chart itself is specified (stable/prometheus). In this case, stable is the branch within the repository, while ‘prometheus’ is the chart name.
Finally, the ‘–set’ flag is used to override one of the chart’s default values. In this example, the Kubernetes service deployment type of the Prometheus server is being changed from ‘Cluster IP’ (the default value), to ‘NodePort’. This was done in order to make the Prometheus server easily accessible from via the CPKS ingress controller. Altering the default values in a Helm chart can be accomplished via the command line in this manner, or multiple settings can be altered by referencing a ‘values.yaml’ file. To specify this file, the ‘-f’ flag is used in the ‘helm install’ command, and the default values themselves can be found in the documentation for the Helm chart in question.
Verification that the correct settings were applied may be accomplished with this command:
$ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE api-fitcycle NodePort 10.0.0.19 <none> 5000:30431/TCP 124d fitcycle-mysql ClusterIP None <none> 3306/TCP 124d grafana-release NodePort 10.0.0.68 <none> 80:30401/TCP 141d kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 141d prometheus-release-alertmanager ClusterIP 10.0.0.165 <none> 80/TCP 141d prometheus-release-kube-state-metrics ClusterIP None <none> 80/TCP 141d prometheus-release-node-exporter ClusterIP None <none> 9100/TCP 141d prometheus-release-pushgateway ClusterIP 10.0.0.159 <none> 9091/TCP 141d prometheus-release-server NodePort 10.0.0.84 <none> 80:30402/TCP 141d wavefront-proxy ClusterIP 10.0.0.132 <none> 2878/TCP,4242/TCP 124d web-server NodePort 10.0.0.225 <none> 8000:30440/TCP 124d
In order to access the Prometheus interface, we take the output in the ‘Address:’ line of another CLI command:
$ vke cluster show vcs-demo-cluster | grep Address: Address: vcs-demo-cluster-2ca804a0-8a4e-11e8-80fc-0a365bcc232c.bfd7a751-b2f5-43a3-b2b3-0c82d6896aee.vke-user.com
And combine it with the ‘NodePort’ value of the ‘prometheus-release-server’ service to access the Prometheus UI via a web browser.
Navigating through Prometheus ‘Status’ menu to the ‘Targets’ page, the various endpoints available from which Prometheus can scrape metrics can be viewed.
Installing Grafana
The Grafana installation in this environment was also accomplished via a Helm chart, and the chart in question is significantly more simple than the Prometheus one discussed above.
To install the Grafana chart, the following command was run:
$helm install --name grafana-release stable/grafana --set service.type=NodePort
Similarly to the Prometheus example, the Grafana UI can then be reached by combining the CPKS cluster ingress URL with the service NodePort value (30401).
After accessing the Grafana UI, some standard dashboards will be available, or others can be created and/or imported. One such example is an imported Kubernetes dashboard, which can be seen in Figure 4.
Conclusion
Utilizing open source monitoring tools such as Prometheus and Grafana provides a straightforward path to visibility and insight based on time series metrics. These concepts are applicable to multiple application environments, from the more traditional virtual machine or instance based domains to the rapidly expanding kubernetes platform.
