Yimeng Liu, Solutions Architect in the Solutions Architecture team of the Cloud Platform Business Unit
VMware vSphere® with Tanzu™ includes everything an enterprise needs to make the best use of Kubernetes as part of its VMware vSphere-based infrastructure. The Confluent Platform is an enterprise-ready platform that complements Apache Kafka with advanced capabilities designed for microservices and event-driven applications.
Using the Confluent Operator, customers can deploy and manage the Confluent Platform as a cloud-native, stateful container application on VMware vSphere with Tanzu to deliver a simplified operational environment that enables developer velocity, agility, and innovation.
Confluent Kafka
Apache Kafka is a popular open-source tool for real-time publish/subscribe messaging. It uses a scalable, fault-tolerant cluster for message storage, and it can also be integrated with other open-source data-oriented solutions such as Spark or Elasticsearch for real-time analysis and rendering of streaming data.
Founded by the original developers of Apache Kafka, Confluent delivers the most complete distribution of Kafka with the Confluent Platform. The Confluent Platform improves Kafka with additional community and commercial features designed to enhance the streaming experience of both operators and developers in production at a massive scale.
Why Confluent Kafka on vSphere with Tanzu?
vSphere with Tanzu is the best solution that puts together the virtualization and Kubernetes into the same platform. It can simplify the adoption of cloud-native constructs through familiar tools. VI admins can provision Kubernetes clusters using the same processes they already use to provision virtual machines. For Developers, it provides self-service access to resources and empowers deploy and manage Kafka using Confluent Operator as the event streaming platform. Confluent Platform automates provisioning Kafka on VMware vSphere with Tanzu in minutes and scales needed. Build modern applications on-prem or in any cloud and leverage your data, deploying Confluent Platform on VMware vSphere with Tanzu gives you the freedom to run applications on any cloud. vSphere with Tanzu also provides the Tanzu Kubernetes Grid extension that you can deploy Prometheus for monitoring the metrics for Confluent Kafka cluster and Grafana for visualization.
How to Deploy Confluent Kafka on vSphere with Tanzu?
You may follow the vSphere with Tanzu Quick Start Guide to set up the supervisor cluster and create a TKG cluster that will be used for the Confluent Kafka cluster deployment.
We used Confluent Operator to deploy Confluent platform as a cloud-native, stateful container application on the TKG cluster. Confluent Operator makes it easy to bootstrap the ZooKeeper, Kafka cluster, and other components in a minute.
After deploying Kafka cluster successfully on vSphere with Tanzu, we can validate performance, monitoring, and use cases. For more details, visit the Running Modern Applications with VMware vSphere with Tanzu reference architecture.
Confluent Kafka on vSphere with Tanzu Use Cases
Streaming processing Architecture
We used Kafka, Spark, and Solr on vSphere with Tanzu to provide data analytics platform as a service. Collect, store, and process the streaming data in real-time with the industry’s enterprise-ready event streaming platform that complements Kafka with Spark streaming application to simplify enterprise operations at scale and enable event transformations through stream processing. Spark is a high-performance engine for large-scale computing tasks, such as data processing, machine learning, and real-time data streaming. Similarly, Solr is an extremely powerful and open-source enterprise searching platform optimized to search large volumes of text-centric data.
Log Analytics Platform
Kafka is the most common broker solution deployed with Elasticsearch, Kibana Beats, and Logstash (also known as the ELK stack). Generally, Kafka is used as the entry point for collecting data from your existing data stores. ELK Stack retrieves data from Kafka and then searches, analyzes, and visualizes the data in real-time.
