Complete Guide to Setting Up ELK Stack with Kafka, Redis, Beats, and Spring Boot for Microservices Logging

Architecture diagram:

Components Breakdown:

  1. Microservices 1, 2, and 3:

    • These are the core applications responsible for generating logs.

  2. Logstash:

    • Acts as a central log processor. It collects logs from the microservices, transforms them into a common format (usually JSON), and sends them to Elasticsearch.

  3. Elasticsearch:

    • A powerful search engine that stores and indexes the logs received from Logstash. It allows for fast retrieval and analysis of log data.

  4. Kafka:

    • A distributed streaming platform used to buffer and distribute logs from the microservices to Logstash. Kafka ensures reliability and fault tolerance in log handling.

  5. Redis:

    • An in-memory data store used for caching, session management, and other purposes. Redis can help improve the performance and scalability of the application.

  6. Beats:

    • A lightweight data shipper that can forward logs from the microservices to Logstash or Kafka. Beats is typically used for collecting logs from application files or other sources.

  7. Kibana:

    • A data visualization and exploration tool that allows users to query, visualize, and analyze the logs stored in Elasticsearch. Kibana provides an intuitive interface for monitoring and troubleshooting the logs.

Overall Flow:

  1. Log Generation:

    • The microservices generate application logs.

  2. Log Collection:

    • Beats or Kafka collects logs from the microservices and sends them to Logstash.

  3. Log Processing:

    • Logstash receives logs from Beats or Kafka, transforms them, and sends them to Elasticsearch.

  4. Log Storage:

    • Elasticsearch stores and indexes the logs, enabling fast search and retrieval.

  5. Log Analysis:

    • Kibana provides a user interface for querying, visualizing, and analyzing the logs stored in Elasticsearch. This allows users to gain insights and monitor the health of their microservices.

Additional Notes:

  • The use of Kafka enhances scalability and fault tolerance by decoupling log producers (microservices) from log consumers (Logstash, Elasticsearch). This makes the system more resilient and capable of handling large volumes of log data.

  • Redis can be used for caching, session management, and other performance-enhancing purposes. It can reduce latency by storing frequently accessed data in memory.


To set up a complete logging pipeline for a microservices architecture with Elasticsearch, Logstash, Kibana (ELK) stack, Kafka, Redis, Beats, and Spring Boot Microservices (Microservices 1, 2, and 3), follow the detailed steps below.

1. Install and Configure Elasticsearch

Install Elasticsearch:

  • Download and install Elasticsearch from the official website.
  • Extract and start the Elasticsearch service:
    bin/elasticsearch

Configure Elasticsearch:

  • Edit elasticsearch.yml to bind the network and configure cluster settings (e.g., network.host, cluster.name, etc.):
    network.host: 0.0.0.0
cluster.name: "log-cluster"

2. Install and Configure Logstash

Install Logstash:

  • Download and install Logstash from Logstash Download Page.
  • Install Kafka and Redis plugins for Logstash:
    bin/logstash-plugin install logstash-input-kafka
bin/logstash-plugin install logstash-output-elasticsearch

Create Logstash Configuration (logstash.conf):

  • Define input, filter, and output configuration in the Logstash configuration file:
    input {
  kafka {
    bootstrap_servers => "localhost:9092"
    topics => ["microservices-logs"]
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "microservices-logs-%{+YYYY.MM.dd}"
  }
}
  • This configuration will collect logs from Kafka and push them to Elasticsearch.

Start Logstash:

bin/logstash -f logstash.conf

3. Install and Configure Kibana

Install Kibana:

Configure Kibana:

  • Edit the kibana.yml configuration file to set the Elasticsearch host:
    server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

Start Kibana:

bin/kibana

4. Install and Configure Kafka

Install Kafka:

  • Download and install Kafka from Kafka Download Page.

  • Start Zookeeper (Kafka depends on Zookeeper):

    bin/zookeeper-server-start.sh config/zookeeper.properties

  • Start Kafka:

    bin/kafka-server-start.sh config/server.properties

Create Kafka Topic:

bin/kafka-topics.sh --create --topic microservices-logs --bootstrap-server localhost:9092 --partitions 1 --replication-factor 1

5. Install and Configure Redis (Optional for Buffering)

Install Redis:

Use Redis as a Buffer:

  • Redis can be used to buffer logs before sending them to Kafka or Elasticsearch. Configure your Spring Boot microservices to send logs to Redis first if needed.

6. Install and Configure Beats (Filebeat)

Install Filebeat:

  • Download and install Filebeat from Filebeat Download Page.
  • Configure Filebeat to monitor the log files of Spring Boot microservices and forward them to Logstash.

Configure Filebeat: Edit the filebeat.yml file:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/microservices-logs/*.log

output.logstash:
  hosts: ["localhost:5044"]

Start Filebeat:

sudo filebeat -e

7. Configure Spring Boot Microservices

For each Spring Boot microservice (Microservice 1, 2, and 3), you need to configure logging so that logs are sent to Kafka or directly to Redis.

Add Dependencies to pom.xml:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-logging</artifactId>
</dependency>

Configure Logback (or Log4j2) to Send Logs to Kafka:

Example for logback-spring.xml:

<appender name="KAFKA" class="org.apache.kafka.log4jappender.KafkaAppender">
    <param name="kafka.topic" value="microservices-logs"/>
    <param name="kafka.bootstrap.servers" value="localhost:9092"/>
    <param name="layout" value="org.apache.log4j.PatternLayout"/>
    <param name="Pattern" value="%d{ISO8601} [%-5p] %c{1}:%L - %m%n"/>
</appender>

<root level="info">
    <appender-ref ref="KAFKA"/>
</root>

In this configuration, log messages from Spring Boot microservices will be sent to Kafka in the microservices-logs topic.

8. Monitor Logs in Kibana

Once everything is set up, you can use Kibana to monitor the logs. Kibana will show logs indexed by Elasticsearch, and you can query logs using the search bar and create visualizations or dashboards.

  1. Go to http://localhost:5601 to access the Kibana UI.
  2. Create an index pattern for microservices-logs-*.
  3. Go to the "Discover" section to view incoming logs in real-time.

9. Test and Verify the Pipeline

  • Start your Spring Boot microservices (1, 2, and 3).
  • Generate some log data by making requests to your microservices.
  • Check if logs are visible in Kibana in real-time.
  • If using Filebeat, verify the log files are being sent to Logstash and processed correctly.
  • Ensure Kafka is receiving logs and Logstash is forwarding them to Elasticsearch.

10. Optional Enhancements

  • Kafka Connect: If you have other data sources, use Kafka Connect to connect other systems with Kafka.
  • Redis: Use Redis for buffering logs or for caching log data before processing.
  • Monitoring & Alerts: Set up monitoring and alerting in Kibana to notify you about certain log events (e.g., errors).

Summary of Components:

  1. Elasticsearch: Stores and indexes logs.
  2. Logstash: Processes logs from Kafka, formats them, and forwards them to Elasticsearch.
  3. Kibana: Visualizes and analyzes logs in real-time.
  4. Kafka: Acts as the central log aggregator.
  5. Redis: Optionally used for buffering or caching logs.
  6. Beats (Filebeat): Collects and forwards log files from microservices.
  7. Spring Boot Microservices: Generate and send logs to Kafka or Redis.

This setup ensures that logs from your Spring Boot microservices are captured, processed, and displayed effectively for monitoring and troubleshooting.

When to Use This Architecture

  1. High Traffic Microservices: Ideal for large systems with many microservices generating high log volumes.

  2. Real-time Monitoring: Needed for real-time log analysis and quick issue detection.

  3. Scalable Infrastructure: Useful when the system grows and requires scalable logging.

  4. Distributed Systems: Aggregates logs from multiple microservices for easier debugging.

  5. Structured Logs: Efficient when logs are in structured formats (e.g., JSON) that need fast querying.

  6. Improved Debugging: Helps trace requests across services for distributed debugging.

When NOT to Use This Architecture

  1. Small Applications: Overkill for small or monolithic apps.

  2. Low Log Volume: Unnecessary for applications with minimal log traffic.

  3. Short-term Projects: Not needed for prototypes or projects without scalability demands.

  4. Limited Resources: Too resource-intensive for teams with limited capacity.

  5. Simple Debugging: Unnecessary for apps with basic logging needs.


Buy Now – Unlock Your Microservices Mastery for Only $9!

Get your copy now for just $9! and start building resilient and scalable microservices with the help of Microservices with Spring Boot 3 and Spring Cloud.

Popular posts from this blog

Learn Java 8 streams with an example - print odd/even numbers from Array and List

Image
Example 1: Java 8 program to print odd numbers from a List   import java.util.Arrays; import java.util.List; import java.util.stream.Collectors; /*Java 8 Program to find Odd Numbers from a List*/ public class DriverClass { public static void main( String [] args) { List < Integer > numbers = Arrays. asList( 1 , 4 , 8 , 40 , 11 , 22 , 33 , 99 ); List < Integer > oddNumbers = numbers.stream(). filter(o -> o % 2 != 0 ). collect(Collectors.toList()); System.out.println(oddNumbers); } } 1. Initialize List using Arrays.asList() method. Java's built-in Arrays.asList() method converts an array into a list. This method takes an array as an argument and returns a List object. 2.  Java List interface provides stream() method which returns a sequential Stream with list of Integer as its source. 3. The Java stream filter(...
Read more

Java Stream API - How to convert List of objects to another List of objects using Java streams?

Image
In this section, we will show you how to convert a List of objects to another List of objects in Java using the Java streams map(). The ‘map’ method maps each element to its corresponding result. Java Stream API   The Java Stream API provides a functional programming approach over the object oriented capabilities of Java. It is used for processing collections of objects. The Stream in Java can be defined as a sequence of elements from a source List, Set or Array. Most of the stream operations return a Stream. This avails engender a chain of stream operations(stream pipe-lining). The streams withal support the aggregate or terminal operations on the elements. for example, finding the sum or maximum element or finding the average etc...Stream operations can either be executed sequentially or parallel. when performed parallelly, it is called a parallel stream.   Stream map() Method   The Java 8 Stream map() is an intermediate operation. It converts Stream<obj1> to Str...
Read more

Registration and Login with Spring Boot + Spring Security + Thymeleaf

Image
In this section, we will learn how to create user registration and login using Spring boot, Spring security, Thymeleaf, JPA, and H2DB. The GitHub repository link is provided at the end of this tutorial. You can download the source code. Technologies used: Spring Boot 2.5.5  Spring Data JPA  Spring Security  Thymeleaf  Maven 3+  Java 17  H2 Database   User Interface User Registration Authentication Failed Authentication success Sign out Final Project Structure: Complete pom.xml: <? xml version ="1.0" encoding ="UTF-8" ?> < project xmlns ="http://maven.apache.org/POM/4.0.0" xmlns: xsi ="http://www.w3.org/2001/XMLSchema-instance" xsi :schemaLocation ="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd" > < modelVersion >4.0.0</ modelVersion > < parent > < groupId >org.springframework.boot</ groupId > < artifactId >spring-boot-starter-pare...
Read more

Java, Spring Boot Mini Project - Library Management System - Download

Image
Hello everyone, Hope you are doing well. Here, we provide a Library Management System mini-project for beginners, it's free for download. Maybe you can build more on top of this project and create your own product, that's all about you. The source code download link is provided at the end of this post. 1. Benefit of this project You could cover the following technologies and features by learning and debugging this mini project. Java 17 Java Stream Java Lamda Expressions Java Records Functional programming Object-oriented programming var Keyword in Java Generics in Java Java Switch Spring boot 2.6.4 Spring Security Thymeleaf template engine OpenCSV Spring Data JPA Server Side Pagination H2 Database  Bootstrap HTML CSS and more... 2. Objective Of Spring Boot On Library Management System The main objective of this project is to manage the details of the books, author, category, and publisher. Only Admin/Librarian will manage all these activities. Also, they can export the data in ...
Read more

ReactJS, Spring Boot JWT Authentication Example

Image
In this tutorial, we’ll create a user registration & login example using ReactJS, Spring Boot, Spring Security, and JWT authentication. You could download the source code from our Github repository, the download link is provided at the end of this tutorial. Technologies used Backend Technologies: Java 17 Spring Boot 2.7.0 Spring Security Spring Data JPA JWT H2 Database Frontend Technologies: React 17.0.1 Axios 0.27.2 Redux 4.0.5 Bootstrap 4.5.2 ReactJS - SpringBoot - JWT - Flow Backend Project Directory: Frontend Project Directory: Following is the screenshot of our application - User Registration: User Signin: Profile View: Access Resource: We will build two projects:  1. Backend:   spring-boot-security-jwt 2. Frontend: react-redux-jwt Project 1: spring-boot-security-jwt Pom.xml <?xml version = "1.0" encoding = "UTF-8" ?> <project xmlns = "http://maven.apache.org/POM/4.0.0" xmlns:xsi = "http://www.w3.org/2001/XMLSchema-instance"...
Read more

Top 5 Java ORM tools - 2024

Image
  The term ORM refers to object-relational mapping, which uses objects to adapt programming languages to database systems while allowing them to function with SQL and object-oriented programming ideas. Any kind of database management system that can accomplish object mapping to the table in the virtual system is suitable for the implementation of ORM. Java has a wide variety of ORM tools, some of which are mentioned below: 1. Hibernate Hibernate is a lightweight, open-source, non-invasive Java ORM (Object-Relational Mapping) framework that allows developers to create objects that function independently of database software and create persistence logic in any Java or JavaEE application. It offers an abstraction layer, so developers don't have to bother about implementations; Hibernate handles these tasks internally, such as connecting to databases and creating queries to carry out CRUD operations, among other things. Persistence logic development is done with it. Persistence logic r...
Read more

Java - Blowfish Encryption and decryption Example

Image
Blowfish is an encryption method developed by Bruce Schneier in 1993 as an alternative to the DES encryption method. It is significantly faster than DES and provides good encryption speed, although no effective cryptanalysis technique has been found to date. It is one of the first secure block ciphers that is not protected by any patents and is therefore freely available for anyone to use. This is a symmetric block cipher algorithm. Block size: 64 bits Key size: variable size from 32 to 448 bits number of subsections: 18 [P-array] number of rounds: 16 number of substitution blocks: 4 [each with 512 records of 32 bits each] develop  Example 1: Encryption and decryption using Blowfish import java.io.UnsupportedEncodingException; import java.nio.charset.Charset; import java.security.InvalidKeyException; import java.security.NoSuchAlgorithmException; import java.util.Base64; import javax.crypto.BadPaddingException; import javax.crypto.Cipher; import javax.crypto.IllegalBlockSiz...
Read more

Spring boot video streaming example-HTML5

Image
Hello everyone, Today we will learn how to stream MP4 video using Spring Boot and Spring Web Flux.  The Source code download link is provided at the end of this post. New, Quarkus Practice User Interface   Project Structure: G radle(build.gradle) plugins { id 'org.springframework.boot' version '2.5.4' id 'io.spring.dependency-management' version '1.0.11.RELEASE' id 'java' } group = 'com.knf.demo' version = '0.0.1-SNAPSHOT' sourceCompatibility = '11' repositories { mavenCentral() } dependencies { implementation 'org.springframework.boot:spring-boot-starter-webflux' testImplementation 'org.springframework.boot:spring-boot-starter-test' testImplementation 'io.projectreactor:reactor-test' } test { useJUnitPlatform() } Service (VideoStreamingService.java) package  com.knf.demo.service ; import org.springframework.beans.factory.annotation. Autowired ; import org.springframework...
Read more

Google Cloud Storage + Spring Boot - File Upload, Download, and Delete

Image
In this section, we will learn  how to  create a simple spring boot application to perform different file operations such as upload, download, list, and delete files from the Google Cloud Storage. 1.  A little bit of Background Google Cloud Storage Cloud Storage is a managed service for storing unstructured data. Store any amount of data and retrieve it as often as you like. More Info - click here! Spring Boot Spring Boot makes it easy to create stand-alone, production-grade Spring-based Applications that you can "just run".  More Info - click here! 2. Create a GCP Project First, Sign into the Google console at  https://console.cloud.google.com . You can create a new project by first selecting the project dropdown in the top left and selecting " New Project ". Next, specify your GCP  Project name  and  Project ID . Then  Click on the " CREATE " button. Copy " Project ID " and keep it for future purposes. 3. Create a Google Cloud Storage bucke...
Read more