Charlie Black and Bob Glithero contributed to this post.
Today we’re introducing VMware Tanzu GemFire for Kubernetes, an in-memory database for fast response and high consistency adapted to the ephemeral nature of resources in Kubernetes.
VMware Tanzu GemFire is a distributed in-memory key value store for a variety of caching, session-state, data-store, and geo-distributed mission-critical workloads. Hundreds of enterprise customers rely on Tanzu GemFire: in hospitality, logistics, ticketing, insurance, government, and almost every financial service sector. Increasingly we see enterprises adopting Kubernetes as a universal control plane for containerized applications. Why? Because with Kubernetes, developers can manage their database systems in the same way as the rest of the application stack and achieve the same benefits: rapid, consistent deployment and management across environments.
In-memory data consistency with Kubernetes is hard
Kubernetes uses both declarative configuration and automation to simplify complex management of containerized workloads and services. It’s designed to automatically reconcile node state to match the desired configuration. However, the idea that Kubernetes will independently scale, restart, or destroy nodes—and their associated data—is a nightmare scenario for data integrity for most in-memory products. When a pod is destroyed, all that memory is lost, and any ongoing operations may become inconsistent within your distributed system.
Consider the following scenario, where an in-memory system prioritizes data availability over data consistency. A customer record of a financial deposit is coming from a client application into your data store. Your cluster is already replicating data between nodes A and B for high availability; however, these updates are asynchronous (i.e., we defer replication until the transaction processing is done). Node A sends an acknowledgement of the transaction back to the client application. However, assume node B fails at that same moment, just prior to node A replicating this transaction to node B. As it’s designed to, Kubernetes will bring node B back online automatically. However, if the data store doesn’t guarantee consistency, node B will have no information about the transaction that node A has acknowledged. Kubernetes itself has no native mechanism to track or replicate the deposit from node A to node B. Therefore, the states of nodes A and B are inconsistent, and there may be a significant amount of time before they are reconciled. Until then, any reads from node B can return stale data.
However, if a data system uses synchronous replication of transactions to guarantee consistency, node A will not acknowledge the client transaction until node B has been updated. Your data reads are protected from getting stale or inconsistent data if a node is lost and then restored. So it's important to consider your workload when choosing an in-memory or caching layer to run in Kubernetes. The minimal consistency issues you might find running on durable virtual machines will be greatly exaggerated in a Kubernetes cluster due to the ephemeral nature of pods.
Fast, consistent data on Kubernetes with Tanzu GemFire
Tanzu GemFire emphasizes data consistency with fast performance, which makes it an excellent choice for in-memory data workloads. Now we’ve created VMware Tanzu GemFire for Kubernetes with optimized defaults, single-command cluster creation for rapid, self-service deployments, resilience and elasticity out of the box, and simpler fleet management on any certified Kubernetes distribution. The key to Tanzu GemFire for Kubernetes is an operator that extends the default capabilities of Kubernetes to provide the services needed by Tanzu GemFire. The operator is a combination of a custom resource definition (CRD)—a way to define a custom object that Kubernetes can act upon—and one or more controllers that operate different facets of that object within Kubernetes. Tanzu GemFire for Kubernetes currently creates a CRD and a single controller, written in GoLang.
We also used Helm 3, with a Helm chart, as a simple way to install our operator onto any certified Kubernetes cluster. Below is an image of what happens when you run the ‘helm install gemfire-operator’ command from our documentation. The CRD is added to the Kubernetes main node, and a controller application is deployed to a pod on a worker node within the namespace you specify.
Installing the Tanzu GemFire operator
Once you have our operator installed, you can then create a Tanzu GemFire cluster with one other command to the kube API-server, using ‘kubectl’, Kubernetes’ command-line interface. By applying the necessary yaml file, Kubernetes will create a Tanzu GemFire cluster to service any data needs your workload may have.
Creating a Tanzu GemFire cluster
See how to install and create a Tanzu GemFire cluster in the video below.
The Tanzu GemFire operator makes it easy for developers and platform operators to easily spin up and manage a cluster. The controller takes care of provisioning storage, RAM, and compute, configuring network connectivity, accounting for how to horizontally and vertically scale as well as how to properly balance data in the cluster.
Here’s another video to demonstrate the scaling of Tanzu GemFire for Kubernetes.
In addition to scaling, the Tanzu GemFire for Kubernetes operator enables overriding defaults for special use cases, specifying vertical scaling of pods, rolling upgrades to the cluster, and adjusting metrics emission—all while still allowing the flexibility of complex Kubernetes and Tanzu GemFire deployment topologies.
Better with Spring and the Tanzu portfolio
Tanzu GemFire for Kubernetes works with many different languages and offers first-class Spring support, so developers to get up and running easily. See how easily you can connect your Spring app in the video below.
Tanzu GemFire is also a first-class citizen in the Tanzu Portfolio. For example, we’ve created an integration with VMware Tanzu Observability by Wavefront. This enables users to quickly establish a dashboard to monitor key metrics for any given Tanzu GemFire cluster on Kubernetes. You can watch a demo of Tanzu GemFire and Observability in this video.
In-memory backing services on Kubernetes now, with Tanzu GemFire!
Tanzu GemFire for Kubernetes is ready for you today! You can download it from the Tanzu Network and have a look at the documentation. Tanzu GemFire is based on open-source Apache Geode (watch a video demo of Geode on Kubernetes, and all the Geode track videos, at ApacheCon 2020). Finally, see the Tanzu GemFire product page for more about features and use cases.
