A minor release of VMware Spring Cloud Data Flow for Kubernetes (v1.4.0) and Spring Cloud Data Flow for VMware Tanzu (v1.12.0) is now generally available. These commercial products build on the v2.9.x release of open source Spring Cloud Data Flow and additional commercial features.
The theme for this release is to improve the developer experiences in building, designing, and deploying data pipelines composed of event-streaming and batch-style data processing microservice applications. Let's dig in.
Single-step jobs dashboard
Most batch processing can be described in its simplest form as "reading" large amounts of data, performing some custom computation or transformation, and then "writing" the computed results. Spring Batch provides support for bulk reading and writing through `ItemReader` and `ItemWriter` abstractions.
By building on this abstraction, Michael Minella and Glenn Renfro designed and implemented the Spring Boot-based "autoconfiguration" capabilities in Spring Cloud Task, so the rich collection of ItemReaders and ItemWriters in Spring Batch can be made available out of the box as production-ready jobs for data movement.
Further building upon Spring Batch and Spring Cloud Task capabilities, this new commercial release of Spring Cloud Data Flow introduces a brand-new user experience in the dashboard. We are launching four "readers" and "writers," including data movements between file systems, any supported relational databases, Apache Kafka, and RabbitMQ. It is possible to also mix and match these "readers and writers," so there are several combinations of data movement possibilities without needing to write any code! This new dashboard will help application developers and data professionals discover single-step batch jobs, read documentation about their capabilities, and quickly assemble configuration properties—all from the dashboard—to customize execution of data movement between external systems.
The remaining set of readers and writers from Spring Batch will make their way into Spring Cloud Data Flow incrementally. Please follow our progress on GitHub, and help us expand this collection with your favorite readers and writers!
Other notable announcements
In addition to this release, there are exciting things added to the Spring Cloud Data Flow ecosystem.
-
Spring Cloud Data Flow and related components are upgraded to Spring Boot 2.5.x compatibility. Spring Native support is being explored at the Spring Cloud Stream binder level, and for applications deployed by Spring Cloud Data Flow to a target platform like Kubernetes or VMware Tanzu Application Service.
-
The screenshot below highlights the redesigned user experience for application properties in the Spring Cloud Data Flow dashboard. With grouping, filtering, and search features, it is more natural to dig into each of the application’s features and customize the desired behavior, as needed for an individual use case.
-
The TaskLauncher sink application is moved natively into Spring Cloud Data Flow distribution. With this application sitting close to Spring Cloud Data Flow, it will be easy for users to configure and use it to trigger batch jobs and tasks based on continuous upstream events.
-
We have ongoing efforts to increase test coverage and Spring Cloud Data Flow's feature compatibility across multiple Kubernetes versions and distributions and on Tanzu Application Service. With this release, test coverage has increased by 10 percent, to now running 55+ end-to-end acceptance tests that replicate real-world user scenarios.
-
The recently released 2021.0.0 release train brings Spring Cloud Stream 3.1.x and Spring Boot 2.5.x compatibility, for the out-of-the-box event-streaming applications. Big thanks to Dan Frey, who has contributed to the newly released ZeroMQ source and sink event-streaming functions!
Try it out
Ready to try out the latest Spring Cloud Data Flow on Kubernetes or Tanzu Application Service? Download, review the documentation, and install VMware Spring Cloud Data for Kubernetes and Spring Cloud Data Flow for VMware Tanzu. Please reach out in VMware Support or in the open source StackOverflow forum for questions and feedback.
