The Elastic Stack (ELK): how to set up centralized logging with Logstash, Elasticsearch & Kibana

An introduction to the Elastic Stack as part of a DevOps architecture and step-by-step guide to setting up the collection and visualization of system logs

BlogUPDATED ON January 12, 2022

Centralised logging with the ELK stack

In this post, we will introduce the Elastic Stack, also referred to as the ELK stack – Logstash, Elasticsearch and Kibana. And provide step-by-step set-up instructions on how to set up the three tools that comprise the stack to automate the collection and visualization of system logs in a DevOps, cloud-native app architecture.

Agile & DevOps teams and consultants

Supercharge your next cloud development project!

What is the Elastic Stack (formerly known as the ELK Stack)?

The Elastic Stack is the evolution of the ELK Stack, whose name is an abbreviation of the three core open-source tools the DevOps stack is built around: Elasticsearch, Logstash and Kibana.

Elasticsearch is a search and analytics engine.

Logstash is a server-side data processing pipeline. It collects multiple simultaneous data flows from different sources, parses each event, identifies named fields to build structure and transforms them into a common format for more powerful analysis and business value.

Kibana visualises data in Elasticsearch through charts and tables.

The Elastic Stack extends the ELK Stack with the addition of Beats, an open source platform for lightweight data shippers which allows users to tail files. All four of the open-source projects that make up the Elastic Stack, Elasticsearch, Logstash, Kibana and Beats, are products developed by Elastic.

Elastic describes the evolution of the ELK to Elastic Stack as

“…the same open source products that they were familiar with, only better integrated, more powerful, more user-friendly and with lots and lots of potential.”

This video by Coding Explained goes deeper into the technical details of the Elastic Stacks’ component tools and how they can be configured to work together as a powerful and flexible log management system.

Elastic Stack Use Cases

Elastic has published a number of use cases of how the Elastic Stack has been utilised by well-known brands to solve a variety of technical and, ultimately, business problems. Two examples include:

HappyFresh solves search latency problem with App Search on Elastic Cloud

HappyFresh, a market-leading grocery shopping and delivery platform in Indonesia, Malaysia, and Thailand, was losing customers before they had checked out because of latency issues with their apps’ product search function causing user frustration. A switch to Elastic’s App Search on Elastic Cloud halved search latency on its online and mobile e-commerce portals while being able to handle a 300% increase in search traffic during the Covid-19 pandemic.

The company is convinced its legacy solution would have failed if it had still been relied upon at the point traffic across its apps multiplied tenfold, search traffic threefold, as shoppers moved online during the pandemic.

Adobe image search capabilities built on the Elastic Stack

Several of Adobe’s products feature image search functionality powered by Elastic Stack, with Adobe Stock a prime example. Adobe’s DevOps engineers manage the following Elastic Stack deployment:

18 production clusters hosting over 10 billion documents, with a live ingestion rate of about 6,000 documents per second. To better support search in Adobe Lightroom, for example, they made the switch from Amazon’s Elasticsearch Service to their own self-managed clusters, moving over nearly 3.5B documents. Much of Adobe’s content is non-textual — images, videos, Photoshop files, and the like — but also includes standard enterprise document types, especially PDFs. The Elastic Stack, along with custom-built Elasticsearch plugins, helps drive the following content search experiences:

  • Search based on computer vision and metadata

  • Deep textual and hybrid content search

  • Video and richer format search

  • Enterprise search

  • Discovery and recommendations

diagram of Adobe inhouse Elastic Stack deployment

How to build a centralised logging system using the Elastic Stack

Sometimes we need to look through the logs, searching for the required lines on several servers; to do this, we log in and look for the logs repeating the same commands on the servers.

Suppose we are an email hosting company with 3 MX servers. A customer files a complaint about a missing message which he sent to his wife at a particular time.

If we had set up a centralised logging system using the Elastic Stack, we would be able to find that message in the logs in a couple of clicks. And there are countless other reasons why organisations need a centralised logging system to keep track of what is happening across their IT infrastructure. While exceptions are not uncommon, it is fair to say the Elastic Stack is our go-to at K&C when our teams are building log management systems.

Elastic Stack alternatives for log management

Of course, while effective and popular, the Elastic Stack is not the only choice a software development team has for log management. Datadog, Splunk, Graylog, and Papertrail are the most popular alternatives and competitors to the components of the Elastic stack. Each of these tools has its technological strengths and weaknesses and different charging models, so could be preferred to the traditional ELK tools in particular circumstances.

But in a majority of circumstances, the Elastic stack is our choice for a log management system. And this is exactly, step-by-step, how we would set up a centralized logging system:

Our centralized logging system stack

  • CentOS 7: The most recent version of the Linux distribution operating system
  • Logstash: Server-based part for processing incoming logs
  • Elasticsearch: For storing logs
  • Kibana: Web interface for searching through and visualizing the logs
  • Logstash Forwarder: It is installed on the servers as an agent for sending logs to a Logstash server.
  • Beats: Platform for lightweight data shippers

Step-by-step tutorial to setting up a centralised log management system with ELK

We will install the first three components on our collection server, and Logstash Forwarder on the servers we want to collect logs from.

Install Java 8

Java is needed for Logstash and Elasticsearch. We are going to install OpenJDK 8.

cd /opt
sudo wget --no-cookies --no-check-certificate --header "Cookie:; oraclelicense=accept-securebackup-cookie" 


sudo tar xvf jre-8*.tar.gz

Grant the necessary rights:

sudo chown -R root: jre1.8*

Create simlinks with the use of alternatives:

sudo alternatives --install /usr/bin/java java /opt/jre1.8*/bin/java 1

Delete the downloaded archive

sudo rm /opt/jre-8*.tar.gz

Install Elasticsearch

Import Elasticsearch public GPG key:

sudo rpm --import

Create and edit the repository file for Elasticsearch:

sudo vi /etc/yum.repos.d/elasticsearch.repo
name=Elasticsearch repository for 1.4.x packages

Install Elasticsearch

sudo yum -y install elasticsearch-1.4.4

Modify the configuration file:

sudo vi /etc/elasticsearch/elasticsearch.yml

Close access to elascticsearch from the outside: localhost

Run Elasticsearch:

sudo systemctl start elasticsearch.service

And add it to the autorun:

sudo systemctl enable elasticsearch.service

Install Kibana:

Download and unpack Kibana 4:

cd ~; wget; tar xvf kibana-*.tar.gz

Edit the configuration file:

vi ~/kibana-4*/config/kibana.yml

In the Kibana configuration file, find the line that determines the host and replace the IP ( by default) with the IP from the localhost:

host: "localhost"

This parameter indicates that Kibana will be accessible only locally. This is good, as we will use the reverse proxy server Nginx to grant access from the outside.

sudo mkdir -p /opt/kibana

And relocate the unpacked files there:

sudo cp -R ~/kibana-4*/* /opt/kibana/

Kibana is run as /opt/kibana/bin/kibana, but we will run it as a service. Create Kibana Systemd:

sudo vi /etc/systemd/system/kibana4.service

Now, run it and add it to the autorun

sudo systemctl start kibana4
sudo systemctl enable kibana4

Set epel repository

sudo yum -y install epel-release

Install Nginx

sudo yum -y install nginx httpd-tools

Using htpasswd, create a user and a password

sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin

Now, edit the main configuration

sudo vi /etc/nginx/nginx.conf

Find and delete the whole section server{}. Two lines should remain at the end

    include /etc/nginx/conf.d/*.conf;

Now, create the configuration file nginx for kibana4

server {
    listen 80;
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        

Run Nginx:

sudo systemctl start nginx
sudo systemctl enable nginx

Now, Kibana is accessible at https://FQDN/

Install Logstash:

Create the repository file for Logstash:

sudo vi /etc/yum.repos.d/logstash.repo
name=logstash repository for 1.5.x packages

Save and exit

Install Logstash:

sudo yum -y install logstash

Generate SSL certificates

Generate certificates for checking server authenticity

cd /etc/pki/tls
sudo openssl req -subj '/CN=logstash_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

The file logstash-forwarder.crt should be copied to all servers, which will send logs to the Logstash server

Configure Logstash:

The configuration files for Logstash are written in json format and are located at /etc/logstash/conf.d. Configuration includes 3 sections: inputs, filters, and outputs.

Create file 01-lumberjack-input.conf and set up “lumberjack” input (the protocol used by Logstash and Logstash Forwarder to communicate)

sudo vi /etc/logstash/conf.d/01-lumberjack-input.conf
input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"

Save and exit. It was noted here that lumberjack will listen to TCP port 5000 and will use the certificates we had generated before.

Now, create a file named 10-syslog.conf, and add it to the settings of syslog messages filtration:

sudo vi /etc/logstash/conf.d/10-syslog.conf
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

Save and exit

Create the last file 30-lumberjack-output.conf:

sudo vi /etc/logstash/conf.d/30-lumberjack-output.conf
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }

Restart Logstash:

sudo service logstash restart

Now that Logstash is set up, we go to Logstash Forwarder

Set up Logstash Forwarder

Copy the SSL certificate to the server where Logstash Forwarder will work

scp /etc/pki/tls/certs/logstash-forwarder.crt user@server_private_IP:/tmp

Download the key:

sudo rpm --import

Create the repository configuration file:

sudo vi /etc/yum.repos.d/logstash-forwarder.repo

Creating repo for Logstash Forwarder

name=logstash-forwarder repository

Install Logstash Forwarder

sudo yum -y install logstash-forwarder

Copy the certificates to the required location:

sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/

Let’s get to setting it up:

sudo vi /etc/logstash-forwarder.conf
"servers": [ "logstash_server_private_IP:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
Между квадратнымы скобками вставляем
      "paths": [
      "fields": { "type": "syslog" }

Add Logstash Forwarder to the autorun and run it:

sudo service logstash-forwarder restart

Now, Logstash Forwarder will send logs to your Logstash server.

Enter kibana, open Dashboard, and enjoy the view.


Search and data management is becoming an increasingly key component for software systems and technology-driven businesses. Making sure search functions perform optimally in customer-facing apps such as e-commerce apps is less of a competitive advantage than a must-have. The need for technology stacks like the Elastic Stack will only grow in the future, especially as new forms of search, like voice search, gain more traction.

If your next software development project, or future iterations of an evolving application, rely on Elastic Stack expertise and you could benefit from a custom-built dedicated team or augmentation of an existing team with the Elastic Stack, please do get in touch!

K&C - Creating Beautiful Technology Solutions For 20+ Years . Can We Be Your Competitive Edge?

Drop us a line to discuss your needs or next project