Prometheus Operator – Installing Prometheus Monitoring Within The Kubernetes Environment

Prometheus Operator – Installing Prometheus Monitoring Within The Kubernetes Environment

Kubernetes Monitoring With Prometheus Operator

In this post, part of our Kubernetes consulting series,  we will provide an overview of and step-by-step setup guide for the open source Prometheus Operator software. Prometheus Operator is used in the integration of the Prometheus monitoring system within a Kubernetes environment. 

Operators are a new class of software introduced in 2016 by CoreOS – recently acquired by Red Hat. CoreOS is the company behind Tectonic, the commercial Kubernetes distribution platform that brings the CoreOS stack together with Kubernetes to provide companies with a Google-esque infrastructure in any cloud or on-premise/bare-metal environment.

Operators

Operators are a new class of software introduced in 2016 by CoreOS  – recently acquired by Red Hat. CoreOS is the company behind Tectonic, the commercial Kubernetes distribution platform that brings the CoreOS stack together with Kubernetes to provide companies with a Google-esque infrastructure in any Cloud or on-premise/bare-metal environment.

CoreOS created Operators as a class of software that operates other software. They inject human-sourced operational knowledge into software. The Prometheus Operator optimised the running of Prometheus on top of Kubernetes, while retaining Kubernetes-native configuration options

Operator software incorporates application domain knowledge able to automate regular tasks, building upon the standard Kubernetes resource and controller concepts. At the heart of operator software is that it removes the burden of manual deployment and lifecycle management, leaving DevOps engineers to focus on optimising configuration.

Prometheus itself is closely related to Kubernetes, with both under the governance of the Cloud Native Computing Foundation (CNCF). Kubernetes evolved as an open source progression from Google’s Borg cluster system and is also originally a Google release. Prometheus shares much of the same design concept blueprint of Borgmon – the monitoring system Google developed to work within Borg. That shared ancestry is apparent from a look under the Kubernetes hood – which reveals the latter exports its internal metrics in the same format that is native to Prometheus.

Prometheus Operator for Kubernetes

The mission of the Prometheus Operator is to make running Prometheus on top of Kubernetes as easy as possible while preserving configurability as well as making the configuration Kubernetes native. (source)

The operator saves the user (administrator) from editing configuration files, automatically configures Prometheus based on YAML files.

How does it work

 

Prometheus Operator Architecture (source)

 

The Prometheus Operator: Optimising Prometheus and Kubernetes Integration

Installing Prometheus Operator is as clean as writing a single command line. The result of that simple command line is DevOps engineers being able to configure and manage Prometheus instances with stripped-back declarative configuration. This configuration will result in the creation, configuration and management of Prometheus monitoring instances.

Prometheus Operator then offers a range of key features:

Create/Destroy

A Prometheus instance can be simply launched in the Kubernetes namespace, a team using the Operator or a particular application.

Easy Configuration

The fundamentals of Prometheus are configured like versions, persistence, retention policies and replicas from a native Kubernetes resource.

Target Services via Labels

Monitoring target configurations are automatically generated, based on well-known Kubernetes label queries. Prometheus doesn’t involve developers learning a unique configuration language.

Custom Resource Definitions (CRD) in Prometheus Operator

Prometheus Operator uses CRD (Custom Resource Definitions) to generate configuration files and identify Prometheus resources.

  • alertmanagers – defines installation for Alertmanager
  • podmonitors – determines which pods should be monitored
  • prometheuses – defines installation for Prometheus
  • prometheusrules – defines rules for alertmanager
  • servicemonitors – determines which services should be monitored

The operator monitors Prometheus resources and generates StatefullSet (Prometheus and Alertmanager) and configuration files (prometheus.yaml, alertmanager.yaml)

The operator also monitors resources from ServiceMonitors, PodMonitors and ConfigMaps, generating prometheus.yaml based on them.

Prometheus Pod

3 containers are launched in the Prometheus hearth:

  • Prometheus
  • prometheus-config-reloader – an add-on to prometheus that monitors changes in prometheus.yaml and an HTTP request reloads the prometheus configuration
  • rules-configmap-reloader – monitors changes in the alerts file and also reloads prometheus

Service Monitors Processing Principle:

  • Prometheus Operator subscribes to Service Monitor resource events, monitoring their addition, removal or modification.
  • Based on ServiceMonitors, the prometheus generates part of the configuration file and stores the secret in kubernetes
  • From kubernetes secret config gets into under
  • Changes discovered by prometheus-config-reloader and reload prometheus
  • Prometheus reloads configuration after reboot and then collects new metrics according to this logic

Alertmanager Pod

As in the case with prometheus, 2 containers run in one pod:

  • alertmanager
  • config-reloader – add-on to alertmanager which monitors changes and reloads alert manager via HTTP request

Grafana Pod

  • Grafana
  • Grafana-sc-dashboard – an add-on to grafana which will subsribe to ConfigMaps resources and generate json dashboards for Grafana based on them

How to Install and Configure Prometheus Operator “ A Step-by-Step Guide

Prometheus is installed using helm.

We clone the repository and update the envy:

Now install Prometheus:

You should now see:

After all the pods have started, we can look at the web UI Prometheus:

Open in the browser http: // localhost: 9090. The services that were set by default should be listed in the ‘service discovery’:

 

To see what metrics are going to be needed to run the command:

Conclusion:

We saw the same thing in the web UI.

Now add our own metrics to Prometheus. As an example we will use traefik. Create traefik-deployment.yaml file and install it in kubernetes

Check if there are metrics:

Open in the browser http: // localhost: 8080 / metrics. Must see:

Now create the traefik-metrics-service.yaml service file for the metrics:

Deploy it to our Kubernetes:

We check our service:

At http: // localhost: 8080 / metrics you should see the same metrics as port-forward described above

Now deployServiceMonitors. Prometheus discovers ServiceMonitors by label. You need to know which ServiceMonitors label it is looking for. To do this:

We are looking for the serviceMonitorSelector block:

In our case, this is release: monitoring. Knowing the label, we create the traefik-servicemonitor.yaml file

A new target should appear in our Prometheus, check:

Open in the browser http: // localhost: 9090:

Metrics successfully Prometheus takes, you can move on to creating a dashboard for Grafana.

Download the dashboard here – . And change it on ConfigMap:

Indent the json of our dashboard with the JSON Dashboard starts ……. JSON Dashboard ends. In json itself, it is important to escape expressions like {{instance}} into {{{{instance}}}}.

We now put our file to prometheus-operator / templates / grafana / dashboards, apply:

Now Dashboard should appear in our Grafana.

Prometheus Operator Is Under Heavy Development

These are the basic principles of the Prometheus Operator. Note that the Prometheus Operator is under heavy development and this could mean that the step-by-step instructions provided go quickly out of date. We will update this guide periodically but please also refer to the CoreOS user guide and Github project for the latest information.

DevOps Consulting From K&C

Prometheus Operator monitoring of infrastructure to forecast outages and errors and detect security threats is just one ‘spoke’ of K&C’s Kubernetes and DevOps Consulting & Development wheel. Whatever your needs, from expertise on a specific tool or platform, to full dedicated teams of DevOps experts, we are here to advise and serve. Please do get in touch to explain your specific needs and situation and we’ll be delighted to help.

 

Add comment

E-mail is already registered on the site. Please use the Login form or enter another.

You entered an incorrect username or password

Sorry that something went wrong, repeat again!
Contact us