VMware Spring Cloud Data Flow Adds Real-time Message Tracing, Redesigned Dashboard

The minor releases of VMware Spring Cloud Data Flow for Kubernetes (v1.3) and VMware Spring Cloud Data Flow for VMware Tanzu (v1.11) are now generally available. These commercial offerings build on the v2.8 release of open source Spring Cloud Data Flow, and add enterprise-ready features to boost developer productivity.

The core value proposition of Spring Cloud Data Flow is about providing a lightweight and interactive developer experience to design, develop, and deploy data pipelines made of event-streaming and batch microservices. By relying on these developer experiences, our customers have clocked significant mileage with large-scale event-streaming and batch data pipeline footprints running on both Kubernetes and VMware Tanzu Application Service.

The theme of these two new minor releases is to stabilize orchestration semantics in Spring Cloud Data Flow by pushing scale-out limits across supported platforms and compatible versions. There are also new features that improve the user experiences of application developers, data engineers, and application operators.

So let’s dig into the details.

Stability

Our customers orchestrate large-scale event-streaming and batch data pipelines through Spring Cloud Data Flow on various IaaS platforms running either Tanzu Application Service or Kubernetes, if not both. And there is increasing evidence that most deployments are orchestrated using Spring Cloud Data Flow’s Java DSL to programmatically build and initiate the deployment of data pipelines using CI/CD or GitOps automation.

To provide a stable foundation for our customers, irrespective of the scale of their data processing operations, our automation incrementally runs integration and end-to-end acceptance tests against several versions of Tanzu Application Service and Kubernetes. As a result, based on VMware’s internal testing, we have increased test harnessing by more than 200 percent, and deployments to different platform versions have increased by a factor of 5x!

Real-time traces and observability

The existing integrations of Spring Cloud Data Flow (event-streaming and batch applications with observability tooling), such as with Tanzu Observability, Prometheus with Grafana, and others, now all get a makeover. In addition, we are extending observability mechanics to include message tracing support for event-streaming data pipelines.

With message tracing support, users can inspect and remediate hotspots in real-time streaming data pipelines. Having this instrumentation is critical given the large-scale data processing required to drive downstream business outcomes. Additionally, tracing support can be used to architect message latency-driven triggers, which helps autoscale event-stream processors based on continuously changing throughput demands.

New Task and Batch dashboard

While event-streaming has seen significant adoption in the enterprise, batch data-processing continues to run strong if you look at specific use cases.

Businesses migrate from mainframes and database-stored procedure-driven ETL flows, to microservices-based batch data pipelines, that are triggered either by real-time event streams or through scheduled recurring runs. The choreography and orchestration of these batch-style data processing pipelines is where Spring Cloud Data Flow can help add significant value to our customers’ modernization journey.

The most important feature of this June 2021 release is the complete overhaul and redesign of both task/job workflows as well as the associated user interface and experience of using the Spring Cloud Data Flow dashboard. The tooling improvements can help significantly boost the productivity of application developers, data engineers, and application operations teams as they design, launch, interact with, and troubleshoot or monitor task/job data pipelines deployed by Spring Cloud Data Flow. 

Among our most requested features are the ability to filter based on task/job definition and to clean up/recover from a corrupted state of batch data processing and execution. We’re delighted to include these features in this release.

The screen capture below walks through both the new UX improvements and the workflow automation, showcasing guardrails that help to reduce guesswork and minimize mistakes.