ic_agile_128ic_agile_white_128ic_banknote_smile_128ic_business_128ic_business_128ic_checkmark_128ic_client_team_manager_128ic_code_file_128ic_code_files_128ic_corporate_cloud_platforms_128ic_crossplatform_apps_128ic_dedicated_team_128ic_developer_128ic_development_team_128ic_enterprise_128ic_faster_timeframe_128ic_fixed_price_128ic_graph_down_128ic_graph_down_128ic_hourly_128ic_hourly_white_128ic_information_finder_128ic_junior_developer_128ic_managed_team_128ic_message_128ic_mobile_app_startups_128ic_mobile_development_128ic_mobile_development_up_128ic_mobile_devices_128ic_multiplatform_128ic_multiplatform_white_128ic_pricetag_128ic_project_checklist_128ic_project_management_128ic_project_management_team_128ic_research_and_development_team_128ic_scalable_team_128ic_senior_developer_128ic_smaller_codebase_128ic_smaller_price_128ic_startup_128ic_team_manager_128ic_three_times_faster_128K&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxK&C_Icons_32pxArrow_Dropdownic_001_google+_16ic_002_xing_16Group 2ic_003_facebook_16ic_004_linkedIn_16Groupic_005_message_16ic_006_upload_16ic_007_remove_16ic_008_email_16ic_009_attachment_16ic_010_file_16ic_011_name_16ic_012_arrow_left_16ic_013_arrow_right_16ic_014_arrow_down_16ic_015_arrow_up_16ic_016_dropdown_arrow_down_16ic_016_dropdown_arrow_leftic_016_dropdown_arrow_rightic_017_K&C_dropdown_arrow_up_16ic_018_language_16ic_019_Quote_16ic_020_+_16ic_021_=_16ic_022_phone_16ic_023_twitter_16ic_024_position_16ic_025_company_16ic_026_search_16ic_027_mobile_16ic_028_fax_16ic_029_location_16ic_030_enlarge_16ic_031_downscale_16ic_032_contactic_download_normal_16pxic_033_skype_16ic_006_download_16 copyic_Interest_based_64ic_acrivate_card_64ic_api_client_64ic_application_architecture_64ic_application_architecture_ white_64ic_application_development_user_64ic_arrow_down_64ic_automated_backups_64ic_automated_infrastructure_provisioning_64ic_automated_infrastructure_provisioning_white_64ic_automated_storage_64ic_automated_storage_64ic_automation_64ic_avaliability_across_the_world_64ic_avaliability_across_the_world_white_64ic_brackets_64ic_brackets_64ic_build_64ic_build_64ic_business_64ic_business_partnership_64ic_business_partnership_white_64ic_business_64ic_calculator_64ic_calendar_64ic_car_rent_64ic_card_renewal_64ic_chat_64ic_chat_bubbles_64ic_chat_bubbles_64ic_chat_white_64ic_checklist_64ic_checkmark_64ic_checkmark_white_64ic_clock_64ic_clock_white_64ic_cloud_media_64ic_cluster_64ic_cluster_white_64ic_code_base_optimization_64ic_coding_64ic_coding_white_64ic_commenting_widget_64ic_commenting_widget_64ic_containers_64ic_containers_white_64ic_continious_64ic_continious_delivery_64ic_continious_delivery_white_64ic_continious_release_64ic_continious_release_white_64ic_continious_white_64ic_cost_saving_64ic_cost_saving_white_64ic_cpu_load_64ic_credit_card_64ic_crossplatform_app_development_64ic_crossplatform_app_development_white_64ic_custom_crm_64ic_custom_crm_64ic_independence_consulring_64ic_database_calls_64ic_database_calls_white_64ic_dedicated_teams_64ic_dedicated_teams_64ic_desktop_application_user_64ic_desktop_application_user_64ic_desktop_code_64ic_desktop_code_white_64ic_developer_64ic_developer_white_64ic_development_64ic_devops_64ic_documents_64ic_documents_graph_64ic_documents_graph_white_64ic_documents_white_64ic_download_presentation_64ic_education_64ic_email_open_64ic_email_open_white_64ic_environment_healthcheckic_euro_64ic_euro_white_64ic_failure_solved_64ic_globe_outlines_64ic_good_quality_64ic_high_load_websites_64ic_high_load_websites_white_64ic_hotel_booking_64ic_inability_64ic_inability_white_64ic_increase_64ic_increase_white_64ic_increasing_team_64ic_independence_64ic_integration_64ic_it_outsourcing_64ic_it_outsourcing_64ic_knowledge_sharing_64ic_mobile_devices_64ic_laptop_user_64ic_laptop_user_64ic_launch_64ic_launch_white_64ic_learning_64ic_learning_two_white_64ic_lighthouse_64ic_link_64ic_load_balancer_64ic_load_balancer_64ic_load_card_64ic_lock_64ic_lock_white_64ic_low_cost_64ic_low_load_websites_64ic_maintenance_tools_64ic_maintenance_tools_white_64ic_media_player_64ic_messaging_platforms_64ic_microservice_architecture_64ic_microservices_64ic_microservices_64ic_mobile_app_64ic_mobile_content_64ic_mobile_development_64ic_mobile_development_white_64ic_mobile_devices_64ic_mobile_devices_white_64ic_mobile_payments_64ic_mobile_social_media_applications_64ic_mobile_workflows_64ic_money_transfers_64ic_multimedia_sharing_64ic_multimedia_sharing_white_64ic_my_garage_64ic_no_access_64ic_no_access_white_64ic_no_oldschool_64ic_online_marketplaces_64ic_online_marketplaces_white_64ic_online_trading_64ic_online_trading_64ic_pair_device_64ic_parallels_64ic_parallels_white_64ic_passcode_64ic_performance_64ic_performance_issues_64ic_performance_issues_white_64ic_performance_white_64ic_plane_64ic_plane_white_64ic_plus_64ic_plus_64ic_pricetags_64ic_product_64ic_product_search_64ic_product_white_64ic_productivity_tools_64ic_productivity_tools_64ic_project_delivery_64ic_project_delivery_white_64ic_project_management_64ic_project_management_collaboration_64ic_project_management_team_64ic_project_management_team_white_64ic_project_risks_reduced_64ic_quality_mark_64ic_quality_mark_white_64ic_question_64ic_react_native_64ic_response_time_64ic_response_time_white_64ic_rest_api_64ic_retail_64ic_transparency_consulting_64ic_scale_up_64ic_scale_up_white_64ic_self_healing_64ic_self_healing_64 copyic_send_money_64ic_server_64ic_server_white_64ic_shopping_64ic_sleep_mode_64ic_small_is_beautiful_64ic_smaller_price_64ic_social_benefits_64ic_social_connections_64ic_socket_64ic_spare_parts_for_cars_64ic_spare_parts_for_cars_white_64ic_speedometer_64ic_performance_consulting_64ic_speedometer_white_64ic_startup_64ic_target_64ic_team_64ic_testing_64ic_testing_checklist_64ic_testing_checklist_64ic_testing_white_64ic_three_times_faster_64ic_touch_64ic_touch_id_64ic_touch_white_64ic_transparency_64ic_ui_design_desktop_64ic_ui_design_mobile_64ic_ui_design_mobile_white_64ic_umbrella_64ic_umbrella_white_64ic_up_and_down_scaling_64ic_up_and_down_scaling_64ic_users_64ic_users_white_64ic_ux_design_64ic_ux_design_desktop_64ic_ux_design_white_64ic_vehicle_64ic_web_based_search_64ic_web_based_search_white_64ic_web_browser_code_64ic_web_browser_developer_mode_64ic_web_browser_user_64ic_web_development_64ic_web_development_white_64ic_web_portals_64ic_web_portals_64ic_web_user_64ic_web_user_white64ic_workflow_64ic_workflow_steps_64ic_workflow_steps_white_64ic_workflow_white_64ic_working_environment_64

Centralized Logging with Logstash, Elasticsearch & Kibana

white envelope icon
contact us nowwhite envelope icon
Up to 3000 characters
Oops! Something went wrong, please try again later

In this post, we will set up the collection and visualization of system logs with the use of Logstash and Kibana.


Sometimes we need to look through the logs, searching for the required lines on several servers; to do this, we log in and look for the logs repeating the same commands on the servers.


Suppose we have 3 MX servers, and some customer files a complaint on a missing message; which he sent abroad to his wife at a particular time. 


Using the ELK stack (E-elasticsearch L-logstash K-kibana), we can find that message in the logs in a couple of clicks. 


What you have to replace is marked in italic.

What we use for building the centralized logging system

- Centos 7: The most recent version of the operation system

- Logstash: Server-based part for processing incoming logs

- Elasticsearch: For storing logs

- Kibana: Web interface for searching through and visualizing the logs

- Logstash Forwarder: It is installed on the servers as an agent for sending logs to a logstash server.


We will install the first three components on our collection server, and Logstash Forwarder on the servers we want to collect logs from.

Install Java 8

Java is needed for Logstash and Elasticsearch. We are going to install OpenJDK 8.

cd /opt
sudo wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" \
"http://download.oracle.com/otn-pub/java/jdk/8u40-b25/jre-8u40-linux-x64.tar.gz"

Unpack

sudo tar xvf jre-8*.tar.gz

Grant the necessary rights:

sudo chown -R root: jre1.8*

Create simlinks with the use of alternatives:

sudo alternatives --install /usr/bin/java java /opt/jre1.8*/bin/java 1

Delete the downloaded archive

sudo rm /opt/jre-8*.tar.gz

Install Elasticsearch

Import Elasticsearch public GPG key:

sudo rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

Create and edit the repository file for Elasticsearch:

sudo vi /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-1.4]
name=Elasticsearch repository for 1.4.x packages
baseurl=http://packages.elasticsearch.org/elasticsearch/1.4/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Install Elasticsearch

sudo yum -y install elasticsearch-1.4.4

Modify the configuration file:

sudo vi /etc/elasticsearch/elasticsearch.yml

Close access to elascticsearch from the outside:

network.host: localhost

Run Elasticsearch:

sudo systemctl start elasticsearch.service

And add it to the autorun:

sudo systemctl enable elasticsearch.service

Install Kibana:

Download and unpack Kibana 4:

cd ~; wget https://download.elasticsearch.org/kibana/kibana/kibana-4.0.1-linux-x64.tar.gz; tar xvf kibana-*.tar.gz

Edit the configuration file:

vi ~/kibana-4*/config/kibana.yml

In the Kibana configuration file, find the line that determines the host and replace the IP (0.0.0.0 by default) with the IP from the localhost:

host: "localhost"

This parameter indicates that Kibana will be accessible only locally. This is good, as we will use the reverse proxy server Nginx to grant access from the outside.

sudo mkdir -p /opt/kibana

And relocate the unpacked files there:

sudo cp -R ~/kibana-4*/* /opt/kibana/

Kibana is run as /opt/kibana/bin/kibana, but we will run it as a service. Create Kibana Systemd:

sudo vi /etc/systemd/system/kibana4.service
[Service]
ExecStart=/opt/kibana/bin/kibana
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=kibana4
User=root
Group=root
Environment=NODE_ENV=production
 
[Install]
WantedBy=multi-user.target

Now, run it and add it to the autorun

sudo systemctl start kibana4
sudo systemctl enable kibana4

Set epel repository

sudo yum -y install epel-release

Install Nginx

sudo yum -y install nginx httpd-tools

Using htpasswd, create a user and a password

sudo htpasswd -c /etc/nginx/htpasswd.users kibanaadmin

Now, edit the main configuration nginx.com:

sudo vi /etc/nginx/nginx.conf

Find and delete the whole section server{}. Two lines should remain at the end

    include /etc/nginx/conf.d/*.conf;
}

Now, create the configuration file nginx for kibana4

server {
    listen 80;
 
    server_name example.com;
 
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;
 
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

Run Nginx:

sudo systemctl start nginx
sudo systemctl enable nginx

Now, Kibana is accessible at http://FQDN/

Install Logstash:

Create the repository file for Logstash:

sudo vi /etc/yum.repos.d/logstash.repo
 
[logstash-1.5]
name=logstash repository for 1.5.x packages
baseurl=http://packages.elasticsearch.org/logstash/1.5/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Save and exit

Install Logstash:

sudo yum -y install logstash

Generate SSL certificates

Generate certificates for checking server authenticity

cd /etc/pki/tls
sudo openssl req -subj '/CN=logstash_server_fqdn/' -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

The file logstash-forwarder.crt should be copied to all servers, which will send logs to the Logstash server

Configure Logstash:

The configuration files for Logstash are written in json format and are located at /etc/logstash/conf.d. Configuration includes 3 sections: inputs, filters, and outputs.

Create file 01-lumberjack-input.conf and set up “lumberjack” input (the protocol used by Logstash and Logstash Forwarder to communicate)

sudo vi /etc/logstash/conf.d/01-lumberjack-input.conf
 
input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

Save and exit. It was noted here that lumberjack will listen to TCP port 5000 and will use the certificates we had generated before .


Now, create a file named 10-syslog.conf, and add it to the settings of syslog messages filtration:

sudo vi /etc/logstash/conf.d/10-syslog.conf
 
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

Save and exit


Create the last file 30-lumberjack-output.conf:

sudo vi /etc/logstash/conf.d/30-lumberjack-output.conf
 
output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

Restart Logstash:

sudo service logstash restart

Now that Logstash is set up, we go to Logstash Forwarder

Set up Logstash Forwarder

Copy the SSL certificate to the server where Logstash Forwarder will work

scp /etc/pki/tls/certs/logstash-forwarder.crt user@server_private_IP:/tmp

Download the key:

sudo rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch

Create the repository configuration file:

sudo vi /etc/yum.repos.d/logstash-forwarder.repo

Creating repo for Logstash Forwarder

[logstash-forwarder]
name=logstash-forwarder repository
baseurl=http://packages.elasticsearch.org/logstashforwarder/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Install Logstash Forwarder

sudo yum -y install logstash-forwarder

Copy the certificates to the required location:

sudo cp /tmp/logstash-forwarder.crt /etc/pki/tls/certs/

Let's get to setting it up:

sudo vi /etc/logstash-forwarder.conf
"servers": [ "logstash_server_private_IP:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
 
Между квадратнымы скобками вставляем
 
  {
      "paths": [
        "/var/log/messages",
        "/var/log/secure"
       ],
      "fields": { "type": "syslog" }
    }

Add Logstash Forwarder to the autorun and run it:

sudo service logstash-forwarder restart

Now, Logstash Forwarder will send logs to your Logstash server.


Enter kibana, open Dashboard, and enjoy the view.

SHARE WITH FRIENDS
You might find this interesting
Our cases
Bosch Classic Cars - Digital Engagement Platform for 19K Vintage Car Owners
Our cases
Liferay Portal Performance Tuning Services for a Major Online Gaming Software Supplier
Web
Three Authentication Approaches to Keep Your Clients Safe
Our cases
How to apply React Native while developing heavy cross-platform mobile apps
Mobile
Reasons to believe in Ionic hybrid app
Web
SEO Tips & Tricks for Single Page Web Applications
E-book
How to Secure Web Product Development — FREE eBook
E-book
Digital Transformation: the Philosopher’s Stone of Economic Growth
Web,Outsourcing,Other
Angular 5 VS React.js – Who’s Going to Set the Tone in the Upcoming Year?
Web
Agile and DevOps are Key Drivers of Digital Transformation
Web
K&C insights: how to make your workflow work for you
Web,Other
GoLang: Features, Pros and Cons
Our cases
Reference: Major producer of auto electronics and spare parts
Outsourcing,Other
Hybrid, SaaS+PaaS, IoT: Cloud Trends to Catch in 2018
Web,Outsourcing,Other
Angular vs. React vs. Vue – Let the Fight Start!
DevOps
DevOps with Puppet: Tips on Setting it up for Configuring Servers
Web,Outsourcing,Other
JS Frameworks: The Trendiest Frameworks You Should Know
Our cases
Micro-service Architecture for New AngularJS Application - Case Study
Mobile
Native or Hybrid Apps: A Quick Comparison
DevOps
Installation and setting up: Nextcloud as a local network storage on CentOS7
Web
A Guidance for Keeping Your Web Development Project Within the Budget: Three Key Pillars
Web
Plan to Succeed: 4 Tips for Building Scalable Software
Web
Microservices… when do we need them?
Web
Debunking imaginary shortcomings of cross-platform frameworks
Our cases
Portal Performance Tuning For Major German Travel Agency
Web
Technologies that Foster Digital Transformation
Web,Other
How to Make Your Web Solution Rock: 7 Areas to Check
Web
4 Time-Saving Ways to Test Your Cross Platform Mobile App
Web,Outsourcing,Other
Angular 5.0.0 – A Better Version of Itself
E-book
Top Tools for Cost-Effective Web Development — eBook
Web
A secret formula of an agile dream team
Web
Cloud Deployment: Overview of Options
Web
How to Motivate Your Dedicated Team to Work with Legacy Projects
Web,Our cases
White Label: A Customized Software Solution from a Business and Tech Perspective
Web
Cost efficient technologies
Web
Scaling software solutions - how it works
DevOps,Outsourcing,Other
How to setup Kubernetes cluster on AWS
DevOps
Docker: Virtualize Your Development Environment Right
Web
When Microservices Help Make Future-Ready Products
Outsourcing,Other
How to Ramp up Your Team Wisely
Outsourcing,Testing
How the QA Team Tests Your Project
DevOps
How We Manage Our Infrastructure with Chef
Our cases
Reformation of Deployment Cycle for Bosch Classic Cars Portal
Our cases
Fast and Lightweight Mobile Application based on PhoneGap/ Cordova
Our cases
Drivelog.de — Web Marketplace for Car Owners and Service Providers
DevOps
Use case: how to build and run Docker containers with NVIDIA GPUs
DevOps
How We Use Ansіble for Configuration of Our Environments
Web
Angular 2.0 vs Angular 1.4. What fits you best?
Other
Big Data: Why Your Business Needs it ASAP
Web,Outsourcing,Other
How a Company Can Benefit from White Label: K&C experience
DevOps
How to Build a Rancher & Docker Based Cloud
DevOps
Setting Up: Traefik Balancer In Rancher Cloud
E-book
Determining Approaches to Mobile App Development
Web,Other
Dedicated Teams for Web Development: Choice Criteria to be Checked
Web
Angular 4 vs React – what to choose in 2017
Outsourcing
The BPM in the Microservice Environment
DevOps,Outsourcing,Other
ROCKET.CHAT as an internal messaging system and helpdesk platform
Web
Monolith, Microservices, Serverless... Are We in the Middle of the Way?
Web
JQuery vs. Angular: Ad Astra per Aspera
DevOps
How to start services on Linux
Web
Advanced Technologies for Marketing Automation
Stay tuned!
We'll gladly share fresh blog updates and our best practices to your email.
We protect your data and will notify you on important updates only.