We are pleased to announce the general availability of Spring Cloud Data Flow 2.3.
The primary themes of this release are the addition of a new native capability to auto-scale streaming applications based on unpredictable traffic patterns; continuous delivery support for task applications; batch jobs; and composed tasks. Lastly, the new release also includes a foundational redesign of metrics and monitoring capabilities to reason through what's going on and troubleshoot the data pipelines.
-
Metrics-based autoscaling: The foundation for metrics-based auto-scaling requires the ability to scale the application instances independently and in isolation. A platform-agnostic new `scale()` API in Spring Cloud Data Flow (SCDF) 2.3 opens up this possibility.
-
CI/CD for tasks: The story of short-lived and transient task microservices going through a CI/CD lifecycle was the highlight of the Batch Processing in 2019 presentation. The continuation of that work extends to simple tasks, batch jobs, and composed tasks, as well, and it is ready for primetime adoption.
-
Prometheus monitoring: A redesigned monitoring foundation for Prometheus is now available. The user experience is consistent and repeatable on local, Cloud Foundry, or Kubernetes.
Let’s dive into the details.
Stream processing and auto-scaling
Metrics-driven auto-scaling is not a new topic for many. In fact, doing this for real-time streaming applications is a mandatory requirement for mission-critical businesses. For example, imagine a credit card fraud-detection use case where real-time transactions are captured and used to predict fraud, and then block fraudulent ones as they occur.
For this predictive analytics scenario, when transaction volumes surge, an intelligent autoscaling design must be in place to keep up with incoming traffic bursts. The application that is doing the most work must adapt to the increase in traffic and continue to operate with the same service capacity. To achieve this, the application would have to scale-up dynamically, and when the traffic fades away, it has to be scaled back down.
In Spring Cloud Data Flow 2.3, the newly added `scale()` API is used in conjunction with metrics such as message latency, offset-lag in Apache Kafka, or queue-depth in RabbitMQ to intelligently decide when and how and to scale the downstream applications. A thorough walkthrough of this design and the reference architecture was presented at SpringOne 2019.
Continuous delivery of short-lived tasks
A CI/CD for short-lived and ephemeral Task applications?!
Yes, you read it right!
From a developer's point of view, the tasks/batch-job development goes through the typical workflow involving writing code, testing, and iterating. However, from a deployment perspective, the incremental improvement to the task/batch-job application is often performed offline. That would mean for bug-fixes, CVE patches, application/deployer property changes, or any dependency upgrades, a developer would have to adjust the version manually, so the subsequent launches take the new improvements into account.
The Spring Batch tech lead, Michael Minella, initially conceptualized the CI/CD thinking for the ephemeral task/batch applications to help improve developer productivity and deployment experiences. Quickly, the concept turned into a spike, and the team added the following features to SCDF:
-
The foundation begins with the resolution of a specific application version from SCDF's App Registry.
-
Newly added persistence layer to capture application and deployment properties along with the task-launch arguments.
-
Intelligence in task-launch workflows to automatically infer and resolve the latest version of the application (if there is one) upon subsequent task-launches.
-
Addition of a new scheduling component that can smartly determine the most recent application version (if there is one) and reuse the existing metadata footprint of the task/batch jobs — again — upon subsequent launches.
-
Ability to restart the task or a composed task definition with newer versions of task/batch-job applications.
-
Automate all of the above using programmatic access to SCDF's RESTful APIs
You can read more about the new feature capabilities and see an end-to-end demonstration in the reference guide.
Prometheus monitoring
Spring Cloud Stream and Spring Cloud Task applications natively integrate with Micrometer for instrumentation and the tracking of the runtime metrics, including message latency, send/receive, and error counts, Building on the time-dimensional metrics from these applications, Spring Cloud Data Flow supports an easy getting-started experience for Prometheus and InfluxDB, while other monitoring systems are available as customization possibilities.
To bring a consistent developer and deployment experience to local, Cloud Foundry, and Kubernetes environments, we have made it easier to use Prometheus with streaming and batch-data pipelines in SCDF. The Prometheus RSocket Proxy removes the service-discovery requirement by opening a bi-directional connection between the applications and the Prometheus system for metrics scrapping.
The out-of-the-box monitoring dashboards for applications, streaming, and batch data pipelines are available as a getting-started guide, which, of course, can be extended for the desired monitoring dashboard customizations.
Import/Export data pipelines
There is increasing evidence of teams migrating streaming and batch data pipelines to different environments, and they are embarking on the multi-platform strategy with either active-active or active-passive deployments. Given these deployment patterns, a need for a utility to export streaming and batch data-pipeline definitions from one environment and import to another came in as a requirement from the community and customers.
We are glad to announce that the export/import utility is available in SCDF's dashboard, for both streaming and batch data pipelines.
Try it out and let us know what you think!
Security improvements
Newer and cleaner security integration is available in SCDF and Skipper.
We’ve redesigned the security components with the latest Spring Boot 2.2 and Spring Security 5.2 upstream releases. Here are the newer security capabilities in SCDF 2.3:
-
Ability to configure and rely on multiple OpenID identity provider backends for independent SCDF and Skipper instances.
-
Granular scope configurations for a given identity provider to drill-down and govern the authorization workflows.
-
A logout mechanism that logs you out from the associated `TokenStore` implementation.
Security integration is cleaner mainly due to what happened in Spring Security 5.2 itself. There have been a lot of changes, improvements, and deprecations, which we inherited to further optimize the security integration possibilities in SCDF and Skipper. The underlying code is cleaner, and there is a lot less duplicated code as a result.
Ecosystem update
Generally available: Spring Cloud Stream Horsham/3.0
As a framework to build event-driven Spring Boot microservices for real-time data processing, Spring Cloud Stream 3.0 includes new feature additions and improvements to amplify the developer productivity. The following new capabilities in Spring Cloud Stream are ready to be used in streaming data pipelines in SCDF 2.3.
Functional and Reactive
-
Elevating the business logic as Plain-Old-Java-Functions or Kotlin lambdas. The framework handles the rest of the boilerplate to bind the business logic with the source and the target data sources.
-
Developers can implement functions either in imperative or reactive programming styles or in combination.
What’s new
-
Representation of Kafka Streams processors as Plain Old Java Functions.
-
Micrometer integration in Kafka Streams applications.
-
Dynamic routing of outbound messages based on
`KafkaHeaders.TOPIC`configuration. -
Batching support in Apache Kafka and RabbitMQ consumers.
-
Support for Quorum Queues in RabbitMQ.
-
Schema Registry as a standalone project — stay tuned for the exciting improvements in the near future!
A deep-dive walkthrough of the various feature improvements is available in a blog series: Part 1 – Demystified and Simplified; Part 2 – Functional and Reactive; Part 3 – Interoperability with Spring Integration; and Part 4 – Event Routing.
Generally available: Spring Cloud Task 2.2
As a framework to build short-lived Spring Boot microservices for batch-style data processing, Spring Cloud Task 2.2 brings improved lifecycle capabilities, as well as the native integration with Micrometer to instrument and track the Task/Batch-job metrics. These new capabilities add to the batch data-pipeline toolchain in Spring Cloud Data Flow.
Deeper Integration with Pivotal Cloud Foundry and Kubernetes
The commercial version of SCDF for Pivotal Cloud Foundry will build upon the 2.3 GA release in the coming weeks to optimize for the rapid and consistent getting-started experience and to deliver the operational automation of SCDF as a service in the Pivotal Application Service.
Likewise, the SCDF helm-chart for Kubernetes will update to 2.3 GA shortly to bring the latest features to Kubernetes.
Join the Community!
We can’t wait for you to try out Spring Cloud Data Flow 2.3. If you want to ask questions or give us feedback, please reach out to us on Gitter, StackOverflow, or GitHub.
