Log Aggregation using ELK Stack

Mohpreet SinghDecember 25th, 2018Last Updated: December 22nd, 2018

0 393 6 minutes read

1. Introduction

With the use of microservices, it has become easy to create stable distributed applications and get rid of many legacy problems. But the use of microservices has introduced a few challenges also and Distributed log management is one of them. As microservices are isolated hence they don’t share database and log files, so it becomes challenging to search, analyse and view log data in real time. This is where ELK stack comes to rescue.

2. ELK

It is a collection of three open source products:

Elastic search is a JSON Based NoSQL database
Logstash a log pipeline tool that takes inputs from various sources, executes different conversions and exports the data to various targets (here to elastic search)
Kibana is a visualization layer that works on top of the elastic search

Refer to the architecture given below:

Log stash fetches the logs from the microservices. Fetched logs
are transformed into JSON and fed to the elastic search. Logs present in elastic search are viewed by the developer using the Kibana.

3. Installing ELK

ELK is java based. Before installing ELK it has to be made sure that JAVA_HOME and PATH are set up and
installation is done using JDK 1.8.

3.1 Elasticsearch

The latest version of Elasticsearch can be downloaded from the download page and it can be
extracted in any folder
It can be executed from the command prompt by using bin\elasticsearch.bat
By default, it would start at http://localhost:9200

3.2 Kibana

The latest version of Kibana can be downloaded from the download
page and
it can be extracted into any folder
It can be executed from the command prompt using bin\kibana.bat
Once started successfully, Kibana will start on default port 5601 and
Kibana UI will be available at http://localhost:5601

3.3 Logstash

The latest version of Logstash can be downloaded
from the download page and it
can be extracted into any folder
Create one file cst_logstash.conf as
per configuration
instructions
It can be executed from the command prompt using bin/logstash -f cst_logstash.conf to
start logstash

4. Creating a sample microservice component

It is required to create a microservice so that logstash can
point to the API log. Below listing shows the code for the sample microservice.

pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.xyz.app</groupId>
    <artifactId>ArtDemo1001_Rest_Controller_Full_Deployment_Logging</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <!-- Add Spring repositories -->
    <!-- (you don't need this if you are using a .RELEASE version) -->
    <repositories>
        <repository>
            <id>spring-snapshots</id>
            <url>http://repo.spring.io/snapshot</url>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </repository>
        <repository>
            <id>spring-milestones</id>
            <url>http://repo.spring.io/milestone</url>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>spring-snapshots</id>
            <url>http://repo.spring.io/snapshot</url>
        </pluginRepository>
        <pluginRepository>
            <id>spring-milestones</id>
            <url>http://repo.spring.io/milestone</url>
        </pluginRepository>
    </pluginRepositories>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.2.RELEASE</version>
    </parent>
    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <spring-cloud.version>Dalston.SR3</spring-cloud.version>
    </properties>
    <!-- Add typical dependencies for a web application -->
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
    </dependencies>
 
    <!-- Package as an executable jar -->
    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
 
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
 
</project>

Above code of the pom.xml has configured dependencies that are required by a spring boot based project.

EmployeeDAO.java

package com.xyz.app.dao;
 
import java.util.Collection;
import java.util.LinkedHashMap;
import java.util.Map;
 
import org.springframework.stereotype.Repository;
 
import com.xyz.app.model.Employee;
 
@Repository
public class EmployeeDAO {
    /**
     * Map is used to Replace the Database 
     * */
    static public Map<Integer,Employee> mapOfEmloyees = 
                 new LinkedHashMap<Integer,Employee>();
    static int count=10004;
    static
    {
        mapOfEmloyees.put(10001, new Employee("Jack",10001,12345.6,1001));
        mapOfEmloyees.put(10002, new Employee("Justin",10002,12355.6,1002));
        mapOfEmloyees.put(10003, new Employee("Eric",10003,12445.6,1003));
    }
     
    /**
     * Returns all the Existing Employees
     * */
    public Collection getAllEmployee(){
        return mapOfEmloyees.values();          
    }
     
 
    /**Get Employee details using EmployeeId .
     * Returns an Employee object response with Data if Employee is Found
     * Else returns a null
     * */
    public Employee getEmployeeDetailsById(int id){
        return mapOfEmloyees.get(id);
    }
    /**Create Employee details.
     * Returns auto-generated Id
     * */
    public Integer addEmployee(Employee employee){
        count++;
        employee.setEmployeeId(count);
        mapOfEmloyees.put(count, employee);
        return count;
    }
     
    /**Update the Employee details,
     * Receives the Employee Object and returns the updated Details  
     * */
    public Employee updateEmployee (Employee employee){
        mapOfEmloyees.put(employee.getEmployeeId(), employee);
        return employee;
    }
    /**Delete the Employee details,
     * Receives the EmployeeID and returns the deleted employee's Details  
     * */
    public Employee removeEmployee (int id){
        Employee emp= mapOfEmloyees.remove(id);
        return emp;
    }
     
}

Above code represents the DAO layer of the application. CRUD operations are executed on Map collection containing Employee objects, in order to avoid database dependency and keep the application light.

EmployeeController.java

package com.xyz.app.controller;
 
import java.util.Collection;
 
import org.apache.log4j.Logger;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.PathVariable;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RestController;
 
import com.xyz.app.dao.EmployeeDAO;
import com.xyz.app.model.Employee;
 
@RestController
public class EmployeeController {
     
    @Autowired
    private EmployeeDAO employeeDAO;
     
     
    public static Logger logger = Logger.getLogger(EmployeeController.class);
     
    /** Method is used to get all the employee details and return the same 
     */
    @RequestMapping(value="emp/controller/getDetails",method=RequestMethod.GET,produces=MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity<Collection> getEmployeeDetails(){
        logger.info("From Producer method[getEmployeeDetails] start");
            logger.debug("From Producer method[getEmployeeDetails] start");
            Collection  listEmployee =employeeDAO.getAllEmployee();
            logger.debug("From Producer method[getEmployeeDetails] start");
        logger.info("From Producer method[getEmployeeDetails] end");
        return new ResponseEntity<Collection>(listEmployee, HttpStatus.OK);
    }
    /** Method finds an employee using employeeId and returns the found Employee 
        If no employee is not existing corresponding to the employeeId, 
        then null is returned with HttpStatus.INTERNAL_SERVER_ERROR as status
     */
    @RequestMapping(value="emp/controller/getDetailsById/{id}",method=RequestMethod.GET,produces=MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity getEmployeeDetailByEmployeeId(@PathVariable("id") int myId){
        logger.info("From Producer method[getEmployeeDetailByEmployeeId] start");
        Employee employee = employeeDAO.getEmployeeDetailsById(myId);
        if(employee!=null)
        {
            logger.info("From Producer method[getEmployeeDetailByEmployeeId] end");
            return new ResponseEntity(employee,HttpStatus.OK);
        }
        else
        {
            logger.info("From Producer method[getEmployeeDetailByEmployeeId] end");
            return new ResponseEntity(HttpStatus.NOT_FOUND);
        }
         
    }
     
    /** Method creates an employee and returns the auto-generated employeeId */
    @RequestMapping(value="/emp/controller/addEmp",
            method=RequestMethod.POST,
            consumes=MediaType.APPLICATION_JSON_VALUE,
            produces=MediaType.TEXT_HTML_VALUE)
    public ResponseEntity addEmployee(@RequestBody Employee employee){
        logger.info("From Producer method[addEmployee] start");
            logger.debug("From Producer method[addEmployee] start");
            int empId= employeeDAO.addEmployee(employee);
            logger.debug("From Producer method[addEmployee] start");
        logger.info("From Producer method[addEmployee] end");
        return new ResponseEntity("Employee added successfully with id:"+empId,HttpStatus.CREATED);
    }
 
    /** Method updates an employee and returns the updated Employee 
        If Employee to be updated is not existing, then null is returned with 
        HttpStatus.INTERNAL_SERVER_ERROR as status
     */
    @RequestMapping(value="/emp/controller/updateEmp",
            method=RequestMethod.PUT,
            consumes=MediaType.APPLICATION_JSON_VALUE,
            produces=MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity updateEmployee(@RequestBody Employee employee){
        logger.info("From Producer method[updateEmployee] start");
        if(employeeDAO.getEmployeeDetailsById(employee.getEmployeeId())==null){
            Employee employee2=null;
            return new ResponseEntity(employee2,HttpStatus.INTERNAL_SERVER_ERROR);
        }
        System.out.println(employee);
        employeeDAO.updateEmployee(employee);
        logger.info("From Producer method[updateEmployee] end");
        return new ResponseEntity(employee,HttpStatus.OK);
    }
     
    /** Method deletes an employee using employeeId and returns the deleted Employee 
        If Employee to be deleted is not existing, then null is returned with 
        HttpStatus.INTERNAL_SERVER_ERROR as status
     */
    @RequestMapping(value="/emp/controller/deleteEmp/{id}",
            method=RequestMethod.DELETE,
            produces=MediaType.APPLICATION_JSON_VALUE)
    public ResponseEntity deleteEmployee(@PathVariable("id") int myId){
        logger.info("From Producer method[deleteEmployee] start");
        if(employeeDAO.getEmployeeDetailsById(myId)==null){
            Employee employee2=null;
            return new ResponseEntity(employee2,HttpStatus.INTERNAL_SERVER_ERROR);
        }
        Employee employee = employeeDAO.removeEmployee(myId);
        System.out.println("Removed: "+employee);
        logger.info("From Producer method[deleteEmployee] end");
        return new ResponseEntity(employee,HttpStatus.OK);
    }
}

Above code represents the controller layer of the application with the request handlers. Request handlers invoke the DAO layer functions and perform the CRUD operations.
application.properties

server.port = 8090
logging.level.com.xyz.app.controller.EmployeeController=DEBUG
#name of the log file to be created
#same file will be given as input to logstash
logging.file=app.log
spring.application.name = producer

Above code represents the properties configured for a spring boot based application.

5. Logstash configuration

As discussed in section 3.3 it is required to create a configuration file for logstash. This configuration file will be used by logstash to take input from the microservice logs. Logs are transformed into JSON and fed to elasticsearch.

cst_logstash.conf

input {
  file {
    # If more than one log files from different microservices have to be tracked then a comma-separated list of log files can 
    # be provided
    path => ["PATH-TO-UPDATE/app.log"]
    codec => multiline {
      pattern => "^%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}.*"
      negate => "true"
      what => "previous"
    }
  }
}
output {
  stdout {
    codec => rubydebug
  }
  # Sending properly parsed log events to elasticsearch
  elasticsearch {
    hosts => ["localhost:9200"]
  }
}

The above logstash configuration file listens to
the log file and pushes log messages to elastic search.
Note: Change the log path as per to your setup.

6. Execution and Output

6.1 Executing microservice for logs

Spring Boot application can be deployed using clean install spring-boot:run and following
URL can be hit from a browser or postman client: http://localhost:8090/emp/controller/getDetails. This will hit the microservice and produce logs on the microservice’s side. These logs will be read by logstash and pushed to elastic search and further, these logs can be viewed using Kibana using next steps.

6.2 Steps to view output on Kibana

Configure the index in
the management console. Use index value as logstash-* as a default
configuration. Open the link: http://localhost:5601/app/kibana#/management/kibana/index?_g=() and it will display
the screen as shown below:

Click on the Next Step and
the following screen is displayed

Select the option as highlighted above and click on “Create index pattern”

On selecting Discover option from the left side menu, the page is displayed as shown below:

Logs can be visualized and filtered based upon the properties highlighted above. Upon hovering on any of the properties, “add” button for the property is displayed. Here on selecting the message property view is displayed as shown below: