Jan 31, 2020 · by Chester Pressler

Spring Cloud Stream with Kafka

A practical example to understand the processing and transaction technology

Spring Cloud Stream is a great technology to use for modern applications that process events and transactions in your web applications. Adding the ability to interface to many different stream interfaces allows Spring Cloud Stream to adapt to new system interfaces and new 3rd party technologies such as Kafka Message Broker.

Kafka is known for being able to handle large amounts of transactions per minute. There are many uses for this high level of transactions and Kafka has seen a recent surge in adoption at many organizations.

Spring Cloud Stream allows interfacing with Kafka and other stream services such as RabbitMQ, IBM MQ and others. Spring Cloud Stream uses a concept of Binders that handle the abstraction to the specific vendor. Most if not all the interfacing can then be handled the same, regardless of the vendor chosen. Configuration via application.yml files in Spring Boot handle all the interfacing needed.

Below is an overview example:

A Practice Example

What is a Spring Cloud Stream Binder?

  • Spring Cloud Stream provides a Binder abstraction for use in connecting to physical destinations at the external middleware
  • The Binder SPI (Service Provider Interface) consists of a number of interfaces, out-of-the box utility classes and discovery strategies that provide a pluggable mechanism for connecting to external middleware
  • Binder selection is configured in the application.yml and also by adding the proper binder dependency into the maven project pom.xml

Source: https://docs.spring.io/spring-cloud-stream/docs/Brooklyn.RELEASE/reference/html/_binders.html 

Spring Cloud Stream Binder Benefits

  • Provides abstraction to the physical connection and vendor implementation
    • Such as RabbitMQ, Redis or Kafka
  • Allows use of a standard tested codebase included in Spring Cloud Stream for handling Inputs and Outputs
  • Configuration and Annotations in code handles the connection between the code and the external systems
  • Developer focuses on the application handling of the data streams only

Example demo Spring Boot application with Kafka

Channels are used to send and receive data to the stream interface which is in this case, a Kafka message broker.

  • What is a Channel?
    • A channel is an input (Sink.class) or output (Source.class) that connects to a message broker topic
    • A topic can be subscribed to by multiple consumers 
  • Default Channel Names
    • output
    • input
  • Custom Channel Names
    • Can be any name you choose that is annotated on the class handling the code
  • Multiple Custom Channels can be defined in one application
  • Configuration is matched to the custom named channels in the spring boot application.yml file

Takeaways of the Benefits of Spring Cloud Stream with Kafka

  • Spring Cloud stream allows you to be vendor neutral in your code base.
  • Code is portable to new additional streams with only new channel names and configuration
  • Custom binder for new stream interfaces can be developed isolated from application logic.


Here are some conclusions to consider going forward.

Chester Pressler

Chester Pressler

Technical Director / Studio Manager
Chester Pressler

Latest posts by Chester Pressler

Share This Article

Post A Comment