In 2017, Weaveworks, a start-up focused on Kubernetes solutions, introduced the concept of ‘GitOps’ – a set of best practices for the deployment and management of applications with cloud native tools and cloud services.
Weaveworks open-sourced a GitOps operator called Flux that runs within a Kubernetes or OpenShift cluster. Its role is to provide automated monitoring that guarantees the state of a Kubernetes cluster matches the configuration supplied in Git.
But GitOps is more than just Flux, or alternative monitoring tools such as ArgoCD. Building on ideas drawn from DevOps and Site Reliability Engineering, GitOps has become the latest hot trend in Cloud Native development.
If it’s not simply a tool, what exactly is GitOps? What advantages does the adoption of the best practises it advocates bring to DevOps teams and the applications they build? And, ultimately, what is the business case for the organisations funding their development to introduce GitOps practises?
Those are the questions we’ll address here and which answer why GitOps has been wholeheartedly adopted and integrated into K&C’s cloud native DevOps services.
Weaveworks summarises GitOps as an operating model for Kubernetes and other technologies used in cloud native development that:
An expanded definition of GitOps is that it is a standard for implementing Continuous Deployment for cloud native applications that focuses on a developer-centric experience for infrastructure operation. It does so by approaching Kubernetes cluster management and application delivery in a way that uses tools developers are already familiar with – Git and Continuous Deployment tools.
Git takes on the mantle of ‘single source of truth’ for declarative infrastructure and applications. Any discrepancy between Git and what is actually running in a K8s cluster (or alternative cluster orchestrator with GitOps intended as technology agnostic) is isolated by software agents. Kubernetes reconcilers can then either rollback the cluster to match the Git, or automatically update it – again as an exact replica of the Git repository.
Having Git at the centre of delivery pipelines allows developers to make pull requests using tools they are familiar with. An automated process matches up the described state in the repository with the production environment. GitOps has been described as “cruise control for managing your applications in production”.
GitOps workflows managing Kubernetes clusters are based on the following 4 principles:
GitOps works with the declarative systems cloud native tools, including Kubernetes, often represent. A declarative system can be treated as code because configuration is guaranteed by facts rather than instructions. When an app’s declarations are versioned in Git, this becomes the ‘single source of truth’ – the crux of GitOps.
With their single source of truth written in stone, or Git, apps can be easily deployed and rolled back and forth from Kubernetes. In a worst-case scenario, a cluster’s infrastructure can be accurately and quickly reproduced from the Git repository.
The declaration of the system stored in Git is always the canonically desired system. If the production declaration ever diverges from the Git version from which everything is derived and driven (as the single source of truth), an alert is raised. If a rollback is necessary, a simple Git revert reinstates the correct system declaration.
Git’s robust security means SSH keys can be used to sign commits so the authorship and provenance of code is always tracked.
With Git as the declared state the production cluster must exactly reflect, changes to that state can be automatically deployed. Cluster credentials are not needed for system changes because the state definition exists outside of the production environment in the Git. So, GitOps allows for the separation of what is being done and how it is to be done.
If the reality of the production cluster does not match your declared system software, agents can send an alert. This allows for self-healing across the system, including in the case of human error and not simply the failure of nodes or pods. Software agents provide the continuous feedback and control loop at the heart of GitOps.
We’ve described the features of GitOps but what are the concrete business case benefits to the system? You would be hard pressed to find a Continuous Deployment technology or methodology that doesn’t promise faster, more frequent deployment. GitOps is no exception but has plenty to back the promise up.
Automated delivery pipelines push changes made to Git into your infrastructure. GitOps goes a step further. It uses tools, like Weaveworks’ Flux, to automate the comparison between the state of a cluster and the Git configuration. An operator in the cluster triggers deployments inside Kubernetes, making the need for any other CD tool redundant.
That means CI doesn’t need access to the cluster. Each update is atomic, transactional and in the Git audit log. Transactions either succeed in a clean state or fail. Thanks to an entirely code-based approach, no new infrastructure is needed.
The Git as the ‘source of truth’ for both infrastructure and application code speeds up deployment and reliability because any misstep is quickly identified. It’s then extremely easy to either roll-back the production state or update it from the Git.
The automation of continuous deployment with an integrated feedback control loop significantly reduces the Mean Time to Deployment ratio. Weaveworks estimates that allows for the shipment of 30 to 100 times more changes per day. The knock-on effect of that is development output should rise by a multiple of 2-3.
A 200% to 300% boost to development output is a massive shot in the arm for productivity and perfectly illustrates just why GitOps is quickly attracting disciples.
Because GitOps involves the pushing of code, not containers, developers use familiar tools like Git to manage Kubernetes updates and add new features. Negating the need for developers to become familiar with the inside of Kubernetes means the onboarding process for new developers to get up to speed with a project is reduced from the typical duration of months to just days.
A consequence of using Git workflows to manage a K8s is that it provides a handy audit log of all cluster changes that take place outside of Kubernetes. That trail of who did what when both ensures stability and can be used to meet SOC 2 compliance requirements.
Because Git can revert and fork, stable, replicable rollbacks are simple. The fact the entire system is described in Git provides a ‘single source of truth’, that can be recovered if anything goes badly wrong. That reduces recovery times to minutes rather than hours. A lot of money can be lost over hours of application downtime!
A single model used to build infrastructure, apps and Kubernetes changes allows for consistency in workflows across a whole organisation. CI/CD pipelines are based on pull requests and operations tasks fully reproducible in Git.
Security has always been important but it’s becoming more so by the day. Git’s security is great thanks to the cryptography used to track and manage changes and being able to sign changes proves author and origin, which allows for a secure definition of desired cluster state.
In the still dominant push approach incumbent to DevOps, an external system (usually a CD pipline tool) initiates deployment of changes to the cluster after a Git repository has been committed to or earlier CI pipeline successfully executed.
The pipeline system needs to have access to the cluster.
In GitOps, CI/CD pipelines enact changes to a system when changes are made by pull request to Git. This is automated by the observer agent within the cluster comparing its live state to that of the repository. When it diverges, an alert is issued and if confirmed the repository state change is pulled into the cluster.
Why is that important? A push pipeline means code starts in the CI system and a flow of encoded scripts pushes changes to the K8s cluster. It’s possible to secure these CI scripts but that still means working outside of the cluster’s trust domain.
A pull pipeline involves a Kubernetes Operator, such as Flux, deploying new images from inside the cluster. It spots any new images in the Git repository and pulls them into the cluster state. This avoids cluster credentials being exposed outside of the production environment.
GitOps doesn’t mean not using a CI tool or automating testing or even deploying straight from development to production. You are simply informed when the observable state of the cluster differs from that of the ‘source of truth’ Git repository. How pull deployments are then actioned is completely down to your policy. They can be deployed to development, staging or QA.
Declarative infrastructure as code is central to the implementation of GitOps but GitOps is much more than only infrastructure as code. The GitOps operating model applies the entirety of the ecosystem and tooling of Git to infrastructure.
The desired state of the infrastructure being deployed in the production environment is guaranteed by CD systems. And there are additional advantages of code review, pull requests and an immutable log of timestamps, source and comments on changes to infrastructure.
The focus of DevOps is aligning the goals and responsibilities of development and operations teams so everyone is pulling in the same direction and not passing responsibility down the line. GitOps is a set of best practises around implementing Continuous Delivery and means development teams are also directly involved in operations.
Both DevOps and GitOps share automation and self-serviced infrastructure principles but otherwise comparisons are not hugely helpful. These shared principles do, however, make it easier for GitOps workflows to be applied on top of existing DevOps techniques.
GitOps does make it possible to transfer much of what traditional operations do to development but in most cases there is still an important role for operations. GitOps can automate cloud resources but certain elements of the infrastructure such as network configuration or Kubernetes clusters still need to be managed centrally by operations specialists.
Focusing equally on developers and CI/CD, GitOps places itself at the centre of the evolution of the Dev and Ops relationship.
DevOps supports a wider range of application models than GitOps does but if the modern environment is moving irrevocably towards containers and Kubernetes as the default orchestrator technology, as it appears to be, DevOps versatility becomes less important.
DevOps can be adapted to a containerised environment but orchestration standards are not always compatible with the quickly expanding K8s ecosystem.
DevOps and GitOps are not mutually incompatible and have many common goals. But with their application and infrastructure frameworks diverging, GitOps and Kubernetes seem set to dominate a cloud native and Kubernetes world. DevOps may well begin to fade into the background in coming years with GitOps in the ascendancy as the dominant approach to development pipelines.