Logs Monitoring in Microservices using EFK - Architecture
The EFK Stack is an amassment of three open-source products — Elasticsearch, Fluentd, and Kibana. EFK stack provides centralized logging in order to identify quandaries with servers or applications. It sanctions us to search all the logs in a single place. It withal avails to find issues in multiple servers by connecting logs during a concrete time frame.
The EFK stack provides a simple yet robust log analysis solution for our developers and DevOps engineers to gain valuable insights on failure diagnosis, application performance, and infrastructure monitoring.
Modern log management and analysis solutions include the following capabilities:
- Aggregation – the facility to accumulate and ship logs from multiple data sources.
- Processing – the faculty to transform log messages into consequential data for more facile analysis.
- Storage – the faculty to store data for elongated time periods to sanction for monitoring, trend analysis, and security use cases.
- Analysis – the faculty to dissect the data by querying it and engendering visualizations and dashboards on top of it.
EFK Stack Architecture
For a small-sized development environment, the classic architecture will look as follows:
Elasticsearch:
The open-source, distributed, RESTful, JSON-predicated search engine. Facile to utilize, scalable, and flexible, it earned hyper-popularity among users and a company composed around it, we know, for search.
The transformed data from Fluentd is Store, Search, and indexed.
Fluentd:
Fluentd is an open-source data collector, which lets you unify the data collection and consumption for better use and understanding of data.
Collect logs and events data. It even parses and transforms data.
Kibana:
Kibana is a free and open frontend application that sits on top of the Elastic Stack, providing search and data visualization capabilities for data indexed in Elasticsearch.
Kibana uses Elasticsearch DB to Explore, Visualize, and Share
However, one more component is needed for Data collection called Beats.
Beats:
Beats is a free and open platform for single-purport data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.
While dealing with prodigiously and sizably voluminous quantities of data, we may need Kafka or RabbitMQ for buffering and resilience.
Apache Kafka:
Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Substructure, inscribed in Scala and Java. The project aims to provide a cumulated, high-throughput, low-latency platform for handling real-time data aliments.
Redis:
Redis is an in-recollection data structure store, utilized as a distributed, in-memory key–value database, cache and message broker, with optional durability.
