Ceph Ansible For A Quick And Error-Free Installation Of Ceph Clusters

Ceph Ansible For A Quick And Error-Free Installation Of Ceph Clusters

Ceph-Ansible – Use Cases And Technical Guide To Deploying Ceph In Kubernetes

In this post of our DevOps consulting series, we follow on from our earlier comparative analysis of CEPH or NFS as alternative Kubernetes data storage solutions with a guide to Ceph-Ansible. We’ll examine how Ceph-Ansible can be used for quick, error-free ceph cluster deployment within a DevOps architecture. We’ll also look at particular use cases, the strengths and drawbacks of these tools in comparison to alternatives and conclude with a detailed step-by-step guide to installing cephusing ceph-ansible for a Kubernetes environment.

What Is Ceph Data Storage And How And When Is It Used?

The exponential rate of growth in data storage needs in modern organizations has resulted in the need for effective big data storage solutions. The Ceph storage tool has stepped up to meet that challenge.

Ceph is an open source software project initiated by Red Hat. It is used to enable scalable object, block and file-based storage under a single system. Ceph clusters are paired with the CRUSH (Controlled Replication Under Scalable Hashing) algorithm to run commodity hardware. CRUSH manages the optimal distribution of data across a Ceph cluster, while also freely retrieving data.

Ceph storage clusters contain a large amount of data. The use of subclusters beneath a cluster breaks that data up into more manageable and relevant chunks. But the subclusters must be properly configured as part of a ‘parent’ cluster in order for the data to be organized according to the appropriate lineage. CRUSH’s role, as a scalable hashing algorithm, nicely divides a large data set into the appropriate clusters and subcluster, allowing for optimized retrieval. So Ceph’s role in big data storage is one which combines both optimized storage and simple data access and retrieval.

Ceph-Ansible Use Case

Modern bare metal and hybrid architectures that combine bare metal with cloud solutions (eg. AWS, Google Cloud), used in an architecture Docker and orchestrators like Docker Swarm, Kubernetes and Rancher – often encounter a problem:

How and where to store data applications in a way that they are accessible from anywhere in the infrastructure, regardless of the location of the Docker container with the application. Ceph helps to solve this problem through a distributed storage system with high availability and scalability. For example, in Kubernetes-based architectures, Ceph has a provisioner for k8s PersistentVolume- CephFS and k8s PersistentVolumeClaims – RBD (Ceph Block Device). Ceph is also often used in big data processing and storage solutions because of its particularly strong horizontal scalability qualities.

Ceph-ansible, an automation engine for provisioning and configuration management is also from Red Hat. It is widely considered to be the most flexible way to install and manage a significant Ceph cluster. Some engineers shy away from ceph-ansible as it isn’t necessarily the easiest solution to install and manage Ceph storage. But it also isn’t difficult with some experience and the production-grade clusters use of ceph-ansible produces means it is certainly worthwhile to put in the extra effort.

Weaknesses of Ceph Storage

The downside to architecture solutions based on Ceph is that it leads to relatively high redundancy rates for servers and/or virtual machines. So while an effective big data and kubernetes storage solution, it is not a cheap one.

Ceph storage should also not be used for any critical data as it is does not offer high levels of security.


Deploying Ceph in Kubernetes Using Ceph-Ansible

Let’s see how to deploy Ceph using ceph-ansible for future use in Kubernetes as block devices (PersistentVolumeClaims – RBD).

For our test bench we will use:

1x virtual server with ansible for external traffic for internal traffic
3x servers for ceph + on each server 3 free HDDs for OSD
for external traffic
for internal traffic

Grafana and Ceph Dashboard for visualization the Ceph cluster will be installed on one of the servers too.

For our 4 servers, the internal network is configured, which we will use for internal Ceph traffic


Step-by-Step Instructions To Preparing A Server With Ansible


It is necessary to generate ssh keys and deploy them to all servers.

Download repository:

Now switch to the version we need in accordance with the following structure. It is also worth noting that for different versions there are also different requirements for the ansible version.

The stable- * brunches have been checked by QE and rarely receive corrections during their life cycle:

Brunch Version
stable-3.0 support for Ceph versions of jewel and luminous. Ansible version 2.4 is required
stable-3.1 support for Ceph versions of luminous and mimic. Ansible version 2.4 is required
stable-3.2 support for Ceph versions of luminous and mimic. Ansible version 2.6 is required
stable-4.0 support for Ceph version of nautilus. Ansible version 2.8 is required

We will use the nautilus version in the example

Install all the necessary dependencies

Rename example config files:

Create an inventory file with a description of all our servers

We bring the main site.yml file to the following form:

We edit the group_vars/all.yml file in it we set such important parameters as the version of the future cluster, specify the internal subnet, interfaces, log size and much more. For our example, the configuration of variables will be as follows:

Next, edit the file of variables responsible for configuring OSD group_vars/osds.yml

You can perform a fully automatic search and installation of OSD on the server by specifying a variable:

However, we explicitly specify the drives of our servers:

Preparation is completed and you now you are ready to run ansible-playbook

The approximate deployment time is 10 minutes. The result of successful execution in the console will be the following output:

We have successfully installed the following components:

To access the dashboard, use the link:

Dashboard web UI

Default login and password – admin / admin

We have successfully completed the basic setup of a Ceph distributed storage system. If the cluster needs to be subsequently expanded, just add a new server to the inventory and you’re good to go!

Add comment

E-mail is already registered on the site. Please use the Login form or enter another.

You entered an incorrect username or password

Sorry that something went wrong, repeat again!
Contact us