Kotlin Ktor Kafka: Producer and Consumer Example

Here’s a step-by-step explanation of the sequence diagram for message flow:

1. User Sends a Request

  • The user initiates the process by sending a POST request to the Ktor server at the endpoint /kafka/send.
  • This request contains the message payload that the user wants to send.

2. Ktor Server Processes the Request

  • Upon receiving the request, the Ktor server calls the method sendMessage("my-topic", message).
  • The method is responsible for sending the user's message to the specified Kafka topic (my-topic).

3. Kafka Producer Sends the Message

  • The Kafka Producer, configured within the Ktor server, takes the message and sends it to the Kafka Broker.
  • The Kafka Producer ensures that the message is sent to the topic "my-topic" on the Kafka cluster.

4. Kafka Broker Receives the Message

  • The Kafka Broker receives the message from the Kafka Producer.
  • The Kafka Broker stores the message in the appropriate partition of the topic "my-topic", making it available for consumers.

5. Kafka Consumer Retrieves the Message

  • The Kafka Consumer, which is subscribed to the topic "my-topic", retrieves the message from the Kafka Broker.
  • The message delivery to the consumer happens asynchronously.

6. Ktor Server Handles the Consumed Message

  • Once the Kafka Consumer processes the message, it sends the consumed message back to the Ktor server (if required for further processing).
  • The Ktor server then responds to the user or processes the message further based on the application logic.

Here's a simple example of setting up Kafka producer and consumer with Ktor:

1. Setup Kafka

To run Kafka locally using Docker:

docker run -d --name kafka -p 9093:9093 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9093 -e KAFKA_LISTENER_SECURITY_PROTOCOL=PLAINTEXT -e KAFKA_LISTENER_PORT=9093 -e KAFKA_LISTENER_NAME=PLAINTEXT -e KAFKA_LISTENER_MODE=host -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e KAFKA_LISTENER_NAMES=PLAINTEXT -e KAFKA_LISTENER_INTERNAL=PLAINTEXT://localhost:9093 wurstmeister/kafka

Ensure Kafka and Zookeeper are running, or use cloud services like Confluent Cloud.

2. Dependencies in build.gradle.kts

Add the required dependencies for Ktor and Kafka:

plugins {
    kotlin("jvm") version "1.8.10"
    application
}

repositories {
    mavenCentral()
}

dependencies {
    implementation("io.ktor:ktor-server-core:2.3.0")
    implementation("io.ktor:ktor-server-netty:2.3.0")
    implementation("org.apache.kafka:kafka-clients:3.1.0")
    implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.6.4")
}

application {
    mainClass.set("MainKt")
}

3. Kafka Producer Example

The Kafka producer sends messages to a Kafka topic:

import org.apache.kafka.clients.producer.KafkaProducer
import org.apache.kafka.clients.producer.ProducerConfig
import org.apache.kafka.clients.producer.ProducerRecord
import java.util.Properties

class KafkaProducerService {
    private val producer: KafkaProducer<String, String>

    init {
        val props = Properties()
        props[ProducerConfig.BOOTSTRAP_SERVERS_CONFIG] = "localhost:9093"
        props[ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG] = "org.apache.kafka.common.serialization.StringSerializer"
        props[ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG] = "org.apache.kafka.common.serialization.StringSerializer"
        producer = KafkaProducer(props)
    }

    fun sendMessage(topic: String, message: String) {
        val record = ProducerRecord<String, String>(topic, null, message)
        producer.send(record) { metadata, exception ->
            if (exception != null) {
                println("Error sending message: ${exception.message}")
            } else {
                println("Message sent to ${metadata.topic()} with offset ${metadata.offset()}")
            }
        }
    }

    fun close() {
        producer.close()
    }
}

4. Kafka Consumer Example

The Kafka consumer listens for messages on a topic:

import org.apache.kafka.clients.consumer.KafkaConsumer
import org.apache.kafka.common.serialization.StringDeserializer
import java.util.Properties

class KafkaConsumerService {
    private val consumer: KafkaConsumer<String, String>

    init {
        val props = Properties()
        props["bootstrap.servers"] = "localhost:9093"
        props["group.id"] = "test-group"
        props["key.deserializer"] = StringDeserializer::class.java.name
        props["value.deserializer"] = StringDeserializer::class.java.name
        consumer = KafkaConsumer<String, String>(props)
    }

    fun consumeMessages(topic: String) {
        consumer.subscribe(listOf(topic))
        while (true) {
            val records = consumer.poll(1000)
            for (record in records) {
                println("Consumed message: ${record.value()} from partition ${record.partition()}")
            }
        }
    }

    fun close() {
        consumer.close()
    }
}

5. Ktor Server to Trigger Kafka Producer

Set up a Ktor server to trigger the Kafka producer from HTTP requests:

import io.ktor.application.*
import io.ktor.features.ContentNegotiation
import io.ktor.http.HttpStatusCode
import io.ktor.jackson.jackson
import io.ktor.request.receiveText
import io.ktor.response.respond
import io.ktor.routing.*
import io.ktor.server.engine.embeddedServer
import io.ktor.server.netty.Netty
import io.ktor.sessions.cookie
import io.ktor.sessions.sessions
import io.ktor.util.KtorExperimentalAPI

fun main() {
    embeddedServer(Netty, port = 8080) {
        install(ContentNegotiation) {
            jackson { }
        }
        routing {
            route("/kafka") {
                post("/send") {
                    val message = call.receiveText()
                    val producer = KafkaProducerService()
                    producer.sendMessage("my-topic", message)
                    producer.close()
                    call.respond(HttpStatusCode.OK, "Message sent")
                }
            }
        }
    }.start(wait = true)
}

This exposes a POST endpoint at /kafka/send where you can send a message to Kafka.

6. Running the Kafka Consumer

In another terminal, run the Kafka consumer:

fun main() {
    val consumerService = KafkaConsumerService()
    consumerService.consumeMessages("my-topic")
}

7. Testing the End-to-End Flow

  1. Start Kafka (if not already running).
  2. Start the Ktor application by running Application.kt.
  3. Start the Kafka consumer by running ConsumerMain.kt.
  4. Send a message to Kafka using Postman or curl:
curl -X POST http://localhost:8080/kafka/send -d "Hello Kafka"
  1. Observe the Consumer Output: The message should be consumed by the Kafka consumer.

Notes:

  • Kafka Producer: Sends a message to the my-topic Kafka topic.
  • Kafka Consumer: Continuously consumes messages from the my-topic Kafka topic.
  • Ktor Server: Acts as an API to trigger the Kafka producer.

You can expand this example by adding error handling, logging, and more advanced configurations.


🌟 Master Kotlin with Kotlin in Action, Second Edition ðŸŒŸ

Unlock the power of Kotlin and elevate your programming skills! This comprehensive guide offers practical tutorials, real-world examples, and the latest Kotlin features to help you build efficient, modern applications like a pro. 🚀

🔥 Exclusive Deal Alert! Save a massive 34% today! 🎉 Don’t miss out on this chance to learn Kotlin while saving big.

👇 Click now to grab your discount and start your Kotlin journey! 👇


👉 Claim Your 34% Discount Now!

Hurry—this offer won't last forever! ⏳

Popular posts from this blog

Learn Java 8 streams with an example - print odd/even numbers from Array and List

Image
Example 1: Java 8 program to print odd numbers from a List   import java.util.Arrays; import java.util.List; import java.util.stream.Collectors; /*Java 8 Program to find Odd Numbers from a List*/ public class DriverClass { public static void main( String [] args) { List < Integer > numbers = Arrays. asList( 1 , 4 , 8 , 40 , 11 , 22 , 33 , 99 ); List < Integer > oddNumbers = numbers.stream(). filter(o -> o % 2 != 0 ). collect(Collectors.toList()); System.out.println(oddNumbers); } } 1. Initialize List using Arrays.asList() method. Java's built-in Arrays.asList() method converts an array into a list. This method takes an array as an argument and returns a List object. 2.  Java List interface provides stream() method which returns a sequential Stream with list of Integer as its source. 3. The Java stream filter(...
Read more

Java Stream API - How to convert List of objects to another List of objects using Java streams?

Image
In this section, we will show you how to convert a List of objects to another List of objects in Java using the Java streams map(). The ‘map’ method maps each element to its corresponding result. Java Stream API   The Java Stream API provides a functional programming approach over the object oriented capabilities of Java. It is used for processing collections of objects. The Stream in Java can be defined as a sequence of elements from a source List, Set or Array. Most of the stream operations return a Stream. This avails engender a chain of stream operations(stream pipe-lining). The streams withal support the aggregate or terminal operations on the elements. for example, finding the sum or maximum element or finding the average etc...Stream operations can either be executed sequentially or parallel. when performed parallelly, it is called a parallel stream.   Stream map() Method   The Java 8 Stream map() is an intermediate operation. It converts Stream<obj1> to Str...
Read more

Registration and Login with Spring Boot + Spring Security + Thymeleaf

Image
In this section, we will learn how to create user registration and login using Spring boot, Spring security, Thymeleaf, JPA, and H2DB. The GitHub repository link is provided at the end of this tutorial. You can download the source code. Technologies used: Spring Boot 2.5.5  Spring Data JPA  Spring Security  Thymeleaf  Maven 3+  Java 17  H2 Database   User Interface User Registration Authentication Failed Authentication success Sign out Final Project Structure: Complete pom.xml: <? xml version ="1.0" encoding ="UTF-8" ?> < project xmlns ="http://maven.apache.org/POM/4.0.0" xmlns: xsi ="http://www.w3.org/2001/XMLSchema-instance" xsi :schemaLocation ="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" > < modelVersion >4.0.0</ modelVersion > < parent > < groupId >org.springframework.boot</ groupId > < artifactId >spring-boot-starter-pare...
Read more

Java, Spring Boot Mini Project - Library Management System - Download

Image
Hello everyone, Hope you are doing well. Here, we provide a Library Management System mini-project for beginners, it's free for download. Maybe you can build more on top of this project and create your own product, that's all about you. The source code download link is provided at the end of this post. 1. Benefit of this project You could cover the following technologies and features by learning and debugging this mini project. Java 17 Java Stream Java Lamda Expressions Java Records Functional programming Object-oriented programming var Keyword in Java Generics in Java Java Switch Spring boot 2.6.4 Spring Security Thymeleaf template engine OpenCSV Spring Data JPA Server Side Pagination H2 Database  Bootstrap HTML CSS and more... 2. Objective Of Spring Boot On Library Management System The main objective of this project is to manage the details of the books, author, category, and publisher. Only Admin/Librarian will manage all these activities. Also, they can export the data in ...
Read more

ReactJS, Spring Boot JWT Authentication Example

Image
In this tutorial, we’ll create a user registration & login example using ReactJS, Spring Boot, Spring Security, and JWT authentication. You could download the source code from our Github repository, the download link is provided at the end of this tutorial. Technologies used Backend Technologies: Java 17 Spring Boot 2.7.0 Spring Security Spring Data JPA JWT H2 Database Frontend Technologies: React 17.0.1 Axios 0.27.2 Redux 4.0.5 Bootstrap 4.5.2 ReactJS - SpringBoot - JWT - Flow Backend Project Directory: Frontend Project Directory: Following is the screenshot of our application - User Registration: User Signin: Profile View: Access Resource: We will build two projects:  1. Backend:   spring-boot-security-jwt 2. Frontend: react-redux-jwt Project 1: spring-boot-security-jwt Pom.xml <?xml version = "1.0" encoding = "UTF-8" ?> <project xmlns = "http://maven.apache.org/POM/4.0.0" xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance"...
Read more

Top 5 Java ORM tools - 2024

Image
  The term ORM refers to object-relational mapping, which uses objects to adapt programming languages to database systems while allowing them to function with SQL and object-oriented programming ideas. Any kind of database management system that can accomplish object mapping to the table in the virtual system is suitable for the implementation of ORM. Java has a wide variety of ORM tools, some of which are mentioned below: 1. Hibernate Hibernate is a lightweight, open-source, non-invasive Java ORM (Object-Relational Mapping) framework that allows developers to create objects that function independently of database software and create persistence logic in any Java or JavaEE application. It offers an abstraction layer, so developers don't have to bother about implementations; Hibernate handles these tasks internally, such as connecting to databases and creating queries to carry out CRUD operations, among other things. Persistence logic development is done with it. Persistence logic r...
Read more

Java - Blowfish Encryption and decryption Example

Image
Blowfish is an encryption method developed by Bruce Schneier in 1993 as an alternative to the DES encryption method. It is significantly faster than DES and provides good encryption speed, although no effective cryptanalysis technique has been found to date. It is one of the first secure block ciphers that is not protected by any patents and is therefore freely available for anyone to use. This is a symmetric block cipher algorithm. Block size: 64 bits Key size: variable size from 32 to 448 bits number of subsections: 18 [P-array] number of rounds: 16 number of substitution blocks: 4 [each with 512 records of 32 bits each] develop  Example 1: Encryption and decryption using Blowfish import java.io.UnsupportedEncodingException; import java.nio.charset.Charset; import java.security.InvalidKeyException; import java.security.NoSuchAlgorithmException; import java.util.Base64; import javax.crypto.BadPaddingException; import javax.crypto.Cipher; import javax.crypto.IllegalBlockSiz...
Read more

Spring boot video streaming example-HTML5

Image
Hello everyone, Today we will learn how to stream MP4 video using Spring Boot and Spring Web Flux.  The Source code download link is provided at the end of this post. New, Quarkus Practice User Interface   Project Structure: G radle(build.gradle) plugins { id 'org.springframework.boot' version '2.5.4' id 'io.spring.dependency-management' version '1.0.11.RELEASE' id 'java' } group = 'com.knf.demo' version = '0.0.1-SNAPSHOT' sourceCompatibility = '11' repositories { mavenCentral() } dependencies { implementation 'org.springframework.boot:spring-boot-starter-webflux' testImplementation 'org.springframework.boot:spring-boot-starter-test' testImplementation 'io.projectreactor:reactor-test' } test { useJUnitPlatform() } Service (VideoStreamingService.java) package  com.knf.demo.service ; import org.springframework.beans.factory.annotation. Autowired ; import org.springframework...
Read more

Google Cloud Storage + Spring Boot - File Upload, Download, and Delete

Image
In this section, we will learn  how to  create a simple spring boot application to perform different file operations such as upload, download, list, and delete files from the Google Cloud Storage. 1.  A little bit of Background Google Cloud Storage Cloud Storage is a managed service for storing unstructured data. Store any amount of data and retrieve it as often as you like. More Info - click here! Spring Boot Spring Boot makes it easy to create stand-alone, production-grade Spring-based Applications that you can "just run".  More Info - click here! 2. Create a GCP Project First, Sign into the Google console at  https://console.cloud.google.com . You can create a new project by first selecting the project dropdown in the top left and selecting " New Project ". Next, specify your GCP  Project name  and  Project ID . Then  Click on the " CREATE " button. Copy " Project ID " and keep it for future purposes. 3. Create a Google Cloud Storage bucke...
Read more