The Elastic Stack (ELK): how to set up centralized logging with Logstash, Elasticsearch & Kibana

An introduction to the Elastic Stack as part of a DevOps architecture and step-by-step guide to setting up the collection and visualization of system logs

Blog • UPDATED ON January 12, 2022

Roman Prydybailo

Author

In this post, we will introduce the Elastic Stack, also referred to as the ELK stack – Logstash, Elasticsearch and Kibana. And provide step-by-step set-up instructions on how to set up the three tools that comprise the stack to automate the collection and visualization of system logs in a DevOps, cloud-native app architecture.

What is the Elastic Stack (formerly known as the ELK Stack)?

The Elastic Stack is the evolution of the ELK Stack, whose name is an abbreviation of the three core open-source tools the DevOps stack is built around: Elasticsearch, Logstash and Kibana.

Elasticsearch is a search and analytics engine.

Logstash is a server-side data processing pipeline. It collects multiple simultaneous data flows from different sources, parses each event, identifies named fields to build structure and transforms them into a common format for more powerful analysis and business value.

Kibana visualises data in Elasticsearch through charts and tables.

The Elastic Stack extends the ELK Stack with the addition of Beats, an open source platform for lightweight data shippers which allows users to tail files. All four of the open-source projects that make up the Elastic Stack, Elasticsearch, Logstash, Kibana and Beats, are products developed by Elastic.

Elastic describes the evolution of the ELK to Elastic Stack as

“…the same open source products that they were familiar with, only better integrated, more powerful, more user-friendly and with lots and lots of potential.”

This video by Coding Explained goes deeper into the technical details of the Elastic Stacks’ component tools and how they can be configured to work together as a powerful and flexible log management system.

Elastic Stack Use Cases

Elastic has published a number of use cases of how the Elastic Stack has been utilised by well-known brands to solve a variety of technical and, ultimately, business problems. Two examples include:

HappyFresh solves search latency problem with App Search on Elastic Cloud

HappyFresh, a market-leading grocery shopping and delivery platform in Indonesia, Malaysia, and Thailand, was losing customers before they had checked out because of latency issues with their apps’ product search function causing user frustration. A switch to Elastic’s App Search on Elastic Cloud halved search latency on its online and mobile e-commerce portals while being able to handle a 300% increase in search traffic during the Covid-19 pandemic.

The company is convinced its legacy solution would have failed if it had still been relied upon at the point traffic across its apps multiplied tenfold, search traffic threefold, as shoppers moved online during the pandemic.

Adobe image search capabilities built on the Elastic Stack

Several of Adobe’s products feature image search functionality powered by Elastic Stack, with Adobe Stock a prime example. Adobe’s DevOps engineers manage the following Elastic Stack deployment:

18 production clusters hosting over 10 billion documents, with a live ingestion rate of about 6,000 documents per second. To better support search in Adobe Lightroom, for example, they made the switch from Amazon’s Elasticsearch Service to their own self-managed clusters, moving over nearly 3.5B documents. Much of Adobe’s content is non-textual — images, videos, Photoshop files, and the like — but also includes standard enterprise document types, especially PDFs. The Elastic Stack, along with custom-built Elasticsearch plugins, helps drive the following content search experiences:

• Search based on computer vision and metadata

• Deep textual and hybrid content search

• Video and richer format search

• Enterprise search

• Discovery and recommendations

How to build a centralised logging system using the Elastic Stack

Sometimes we need to look through the logs, searching for the required lines on several servers; to do this, we log in and look for the logs repeating the same commands on the servers.

Suppose we are an email hosting company with 3 MX servers. A customer files a complaint about a missing message which he sent to his wife at a particular time.

If we had set up a centralised logging system using the Elastic Stack, we would be able to find that message in the logs in a couple of clicks. And there are countless other reasons why organisations need a centralised logging system to keep track of what is happening across their IT infrastructure. While exceptions are not uncommon, it is fair to say the Elastic Stack is our go-to at K&C when our teams are building log management systems.

Elastic Stack alternatives for log management

Of course, while effective and popular, the Elastic Stack is not the only choice a software development team has for log management. Datadog, Splunk, Graylog, and Papertrail are the most popular alternatives and competitors to the components of the Elastic stack. Each of these tools has its technological strengths and weaknesses and different charging models, so could be preferred to the traditional ELK tools in particular circumstances.

But in a majority of circumstances, the Elastic stack is our choice for a log management system. And this is exactly, step-by-step, how we would set up a centralized logging system:

Our centralized logging system stack

Install Java 8

cd /opt
sudo wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" 
"https://download.oracle.com/otn-pub/java/jdk/8u40-b25/jre-8u40-linux-x64.tar.gz"

sudo tar xvf jre-8*.tar.gz

sudo chown -R root: jre1.8*

sudo alternatives --install /usr/bin/java java /opt/jre1.8*/bin/java 1

sudo rm /opt/jre-8*.tar.gz

Install Elasticsearch

sudo rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

sudo vi /etc/yum.repos.d/elasticsearch.repo

[elasticsearch-1.4]
name=Elasticsearch repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

sudo yum -y install elasticsearch-1.4.4

sudo vi /etc/elasticsearch/elasticsearch.yml

network.host: localhost

sudo systemctl start elasticsearch.service

sudo systemctl enable elasticsearch.service

Install Kibana:

cd ~; wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz; tar xvf kibana-*.tar.gz

vi ~/kibana-4*/config/kibana.yml

host: "localhost"

sudo mkdir -p /opt/kibana

sudo cp -R ~/kibana-4*/* /opt/kibana/

sudo vi /etc/systemd/system/kibana4.service

[Service]
ExecStart=/opt/kibana/bin/kibana
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=kibana4
User=root
Group=root
Environment=NODE_ENV=production
 
[Install]
WantedBy=multi-user.target

sudo systemctl start kibana4
sudo systemctl enable kibana4

sudo yum -y install epel-release

sudo yum -y install nginx httpd-tools

sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin

sudo vi /etc/nginx/nginx.conf

    include /etc/nginx/conf.d/*.conf;
}

server {
    listen 80;
 
    server_name example.com;
 
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;
 
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

sudo systemctl start nginx
sudo systemctl enable nginx

Install Logstash:

sudo vi /etc/yum.repos.d/logstash.repo
 
[logstash-1.5]
name=logstash repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.5/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

sudo yum -y install logstash

Generate SSL certificates

cd /etc/pki/tls
sudo openssl req -subj '/CN=logstash_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

Configure Logstash:

sudo vi /etc/logstash/conf.d/01-lumberjack-input.conf
 
input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

sudo vi /etc/logstash/conf.d/10-syslog.conf
 
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

sudo vi /etc/logstash/conf.d/30-lumberjack-output.conf
 
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

sudo service logstash restart

Set up Logstash Forwarder

scp /etc/pki/tls/certs/logstash-forwarder.crt user@server_private_IP:/tmp

sudo rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

sudo vi /etc/yum.repos.d/logstash-forwarder.repo

[logstash-forwarder]
name=logstash-forwarder repository
baseurl=http://packages.elasticsearch.org/logstashforwarder/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

sudo yum -y install logstash-forwarder

sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/

sudo vi /etc/logstash-forwarder.conf

"servers": [ "logstash_server_private_IP:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
 
Между квадратнымы скобками вставляем
 
  {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
       ],
      "fields": { "type": "syslog" }
    }

sudo service logstash-forwarder restart