products Edge Computing Event-Driven Architecture Spring Tanzu RabbitMQ

Event Streaming Using RabbitMQ with Spring

Today, enterprises are using event streaming technologies to become software companies with data-driven businesses. Integrating existing data systems and modernization efforts are keys to meeting this goal. Event streaming technologies such as RabbitMQ and Spring enable organizations to discover new ways to integrate applications, IoT/Edge Computing data flow, machine learning, artificial intelligence, and data science workloads.

RabbitMQ is a well-known reliable messaging, and now event streaming, solution for applications. Spring allows you to easily introduce RabbitMQ for applications written in Java Virtual Machine languages such as Java and Kotlin. Spring projects such as Spring AMQP encapsulate RabbitMQ best practices to manage connections, multi-threading, error handling, and more. All of these details are provided out of the box using the power of Spring. Developers do not need to be concerned with low-level details to build scalable resilient publisher and consumer applications using RabbitMQ.

Spring and RabbitMQ benefits 

  • Simplified sending and receiving messages with conversions to business domain objects. 
  • Automatically declare consumer application queues and publisher exchange topics with the needed binding routing rules. 
  • Auto support for best practice reliable messaging to confirm the publishing or acknowledgment of messages within an application transaction. 
  • Out-of-the-box replay capabilities based on configuration settings for a number of retries and delays between retries. 
  • Built-in error detection for non-retryable errors to be sent to dead letter routed queues for review or reprocessing.

Spring Integration

Spring Integration allows you to build Spring Boot applications that connect disparate systems. It implements best practice integration patterns such as messaging, routing, and transformation based on the very popular enterprise application integration book. Spring Integration simplifies your application’s reliable messaging delivery enterprise integration needs that are backed by RabbitMQ’s Advanced Message Queuing Protocol (AMQP).

Spring Cloud Stream

Spring Cloud Stream simplifies microservice management and configuration of Spring Boot messaging patterns such as publish/subscribe, worker consumer groups, and partitioning. The following is a Spring Cloud Stream example based on a spring-modern-data-architecture demo. 

@Component
public record SaveProductConsumer(ProductRepository repository) 
        implements Consumer<List<Product>> {
    @Override
    public void accept(List<Product> products) {
        repository.saveProducts(products);
    }
}

In this example, the accept method received a list of products from RabbitMQ to save into a repository. The use of RabbitMQ is totally abstracted.

RabbitMQ stream support

Spring supports the RabbitMQ Stream Plugin that communicates with the Java Client, while RabbitMQ supports streaming event log use cases with high throughput. With Spring Boot, developers only need to include the Rabbit Stream dependency.

<dependency>
   <groupId>org.springframework.amqp</groupId>
   <artifactId>spring-rabbit-stream</artifactId>
</dependency>

With Spring Cloud Stream, you can configure how applications use RabbitMQ. In this example, SaveProductConsumer is configured as a cloud function that uses the RabbitMQ “stream” as a queue container type. These details are all defined in a Spring application.yml file. 

spring:
  application:
    name: my-app 
  cloud:
    function:
      definition: saveProductConsumer 
    stream:
      bindings:
        saveProductConsumer-in-0:
          destination: retail.products 
          group: ${spring.application.name}
          consumer:
            concurrency: 1
      rabbit:
        bindings:
          saveProductConsumer-in-0:
            consumer:
              containerType: stream
              prefetch: 10000

Replay event streams 

RabbitMQ Streams allows applications to replay messages that have been consumed. Streams will allow consumers to replay all events or just the events from a particular point in time based on an offset. The following is an example of how Spring Cloud Stream makes it easy to replay from the first message when the property rabbitmq.streaming.replay equals true.

@Bean
@ConditionalOnProperty(name = "rabbitmq.streaming.replay",havingValue = "true")
ListenerContainerCustomizer<MessageListenerContainer> customizer() {
    return ( MessageListenerContainer cont, String dest, String group) ->
    {
        final var container = StreamListenerContainer.class.cast(cont);

        log.info("Replaying from the first record in the stream");
        container.setConsumerCustomizer( (name, builder) -> {
            builder.offset(OffsetSpecification.first());
        });
    };
}

Spring Cloud Data Flow

Spring Cloud Data Flow (SCDF) simplifies the orchestration of Spring Cloud Stream applications on platforms such as Kubernetes and Cloud Foundry. It allows you to build a library of Spring Cloud Stream applications (similar to connectors) for extract, transform, and load (ETL) data pipelines. You can execute data pipelines for change data capture, transformations, filtering, splitting, and more with reliable delivery that use RabbitMQ. 

Spring comes with out-of-box application connectors (e.g., file, JDBC, HTTP, MQTT, etc.) for data integration into various data stores (e.g., GemFire, MySQL/Postgres, and Greenplum) using Spring Cloud Stream Applications by VMware.

Users can deploy data pipelines in minutes using a drag-and-drop user interface. Spring Cloud Data Flow simplifies the management of applications and backing services for microservices cloud native data. Spring Cloud Data Flow eliminates the need for YAML file definitions for multiple applications. The drag and drop capabilities of the Spring Cloud Data Flow dashboard enables non-developers to create ETL data pipelines dynamically with little or no coding. 

SCDF allows you to define data pipelines using language similar to UNIX commands. In the following example, the file application reads from the local filesystem and each file’s content is sent to the Amazon Simple Storage Service (Amazon S3) application to save into an Amazon S3 bucket.

Graphical user interfaceDescription automatically generated

You can manage and monitor all data pipeline applications using the Spring Cloud Dataflow dashboard. You can access applications, logs, review environment settings, scale up instances of the applications, and access observability information conveniently from the SCDF dashboard. 

Graphical user interface, applicationDescription automatically generated

Learn more

Spring provides seamless event streaming over RabbitMQ. Spring allows developers to focus on business logic without dealing with the details of the RabbitMQ infrastructure. Customers such as Charles Schwab, Discover Financial, and Synchrony Financial are utilizing these technologies for digital transformation and modernization. 

VMware is the primary manager of open source RabbitMQ and Spring. Visit us at VMware to learn more about technologies like Spring and VMware Data Solutions  RabbitMQ, GemFire, Postgres, MySQL, and Greenplum.

Related Articles