Thanks to the wonderful Spring Cloud Data Flow community, in our recent survey, we received lots of suggestions, comments, and feedback with regards to limitations in the reference guides, documentation, and samples for Spring Cloud Data Flow.
You asked us to make it easier to find solutions to your Spring Cloud Data Flow problems!
Consequently, the theme for the 2.1 GA release became “helping the community and users,” and today we’re announcing a brand new Spring Cloud Data Flow Microsite to help developers get more from the product.
Background
All the projects under the Spring umbrella have a significant presence in the community forums including StackOverflow, GitHub, Slack, and Gitter channels.
The projects in the Spring Cloud Data Flow (SCDF) ecosystem are well represented in all these community platforms. On any given day, you’ll find the team busy answering community questions. We are proud to regularly engage with the community.
In answering community questions and customer support tickets, we noticed a high degree of context-switching was required between different projects, their reference guides, and the samples, in order to build streaming and batch processing solutions. This was consistent with the feedback you gave us in the survey results as well.
The main reason for all the context-switching is that there are various projects in the SCDF ecosystem, each of which evolves with different features and release cadences. SCDF brings them all together into a coherent set of developer tools to build, deploy, and manage streaming and batch data pipelines. Therefore, answering community questions typically involves pointing to various project-specific resources.
Is this a side effect to building SCDF as microservices based architecture? Perhaps. This architecture has brought us enormous new benefits due to the modular design. If you count Spring Cloud Stream, Spring Cloud Task, Spring Cloud Skipper, and the other projects in the SCDF ecosystem, it’d easily encompass over 100 releases in the last 1 year!
To minimize this context-switching and to promote the easy discovery of product capabilities, we realized that we needed to deliver a step-by-step developer guide to delve into new features, enhancements, and use-case possibilities.
The Spring Cloud Dataflow Microsite was born…
Our Goals
The first and foremost goal was to go deeper as opposed to just highlighting the breadth of the features. Ultimately, we wanted to answer the questions that a developer would face when building streaming and batch processing solutions using SCDF. Building upon that thinking, we chose to start with some ‘simple-to-complex’ scenarios in an incremental style so that folks can switch to the level that matches their familiarity. Lastly, we focused heavily on repeatability — code fragments, end-to-end samples, and recipes that a developer could learn in order to become proficient with Spring Cloud Data Flow in less time.
Other Notable Updates
While the significant focus of this release was on the new Microsite, there were a few other notable improvements within the ecosystem…
-
Spring Cloud Stream 2.2 GA: The primary goal was to elevate the Spring Cloud Function programming model in Spring Cloud Stream. A developed focused Spring IO blog is in the works, so stay tuned. The RabbitMQ, Apache Kafka, and Kafka Streams binders have had new feature improvements and validation through community contributions. Lastly, Kinesis binder recently also added the support for KCL/KPL and DynamoDB streams.
-
Moving away from bit.ly for Bulk App Registration: For convenience, we have shipped bit.ly based shortened URLs to redirect to the bulk-registration property files that we host in Spring repository. This had brought a lot of complexity over the years, and we are happy to announce that we are moving away from bit.ly to replace that with URLs served from the Spring IO domain instead. Look out for the updated URLs in reference guide.
-
Deployment Properties at Fingertips: The discovery of platform-specific deployment properties for the supported platforms were tedious. We have seen multiple reports of folks having trouble understanding how to override the deployment experience. It is now improved by a new dropdown in the UI to populate the deployer properties for the selected platform automatically. No more hunting down for properties from the docs/code, you can now choose from the dropdown.
-
Concurrent Task Launching Optimizations: Given the number of customers with filed-ingest use-cases in the cloud, we published a reference architecture to describe the end-to-end solution for Cloud Foundry and Kubernetes. One of the critical pieces in this architecture is the ability to rate-limit and throttle the concurrent task launches to avoid saturating the available resources. While the initial implementation worked for the new dataset, we received feedback on the checkpointing mechanism towards the historical datasets. We have an improved solution that now measures the actual running tasks to make downstream decisions.
-
Docker Compose / Helm Chart Customizations: With growing demands for quick getting-started experience using SCDF’s Docker Compose and the Helm Chart, we have added customizations to switch to different monitoring solutions and messaging middlewares.
It’s just the Beginning
We are barely getting started on the Spring Cloud Data Flow Microsite. We are determined to continue investing in these learning materials in future releases. We wouldn’t be able to do this without you, the community, customers, and the users, so please reach out to us in StackOverflow, GitHub, or Gitter if you see issues or if you have a new topic to include in the Microsite. Please note, everything is plain markdown files, and they get automatically generated as static web resources. It is set up to add and update by everyone quickly — we are looking forward to your contributions!
