Kubernetes Backup: We Use Velero/Heptio Ark (Here’s How)

Kubernetes Velero (Ark) Backup Tutorial: Step-by-Step Instructions To Installing Velero

In this post in our Kubernetes consulting series we’re going to cover Kubernetes backup with Velero, previously known as Heptio Ark, within an AWS environment. Velero is also supported by the Microsoft Azure and GCP cloud platforms.

This step-by-step tutorial on how to install Velero as your backup facility is specific to an AWS environment that we’re running Kubernetes clusters on. But the installation process is almost exactly the same in other non-AWS cloud environments. Simply switch out the AWS-specific commands for those of whatever cloud platform you are running you K8s on.

Before we get into the step-by-step tutorial, let’s first quickly look at Kubernetes backups more generally, and Velero as a disaster recovery management tool. Feel free to skip down the next two short sections to move straight to the tutorial.

Why Do We Need A Backup And Recovery System For Kubernetes Clusters?

The headline is something of a rhetorical question, with it fairly obvious why any application needs to be backed up. It means the application and especially data it holds can be recovered if anything goes wrong.

We’ve probably all had a lost, stolen or broken laptop or smartphone before cloud storage became a relative standard. If data hadn’t been recently backed up externally – it was gone. And even if it was, setting up a repaired machine or replacement with all the same programmes etc. is a monumental pain in rear.

And that was just in the case of a hardware device holding one person’s data. When a software application, especially an enterprise-level application, fails the problem is far bigger. Not only can large volumes of data be lost (even if most of it is presumably backed up on the cloud or physical servers in multiple copies not exposed to any single point of failure) but it can be difficult and time consuming to fix the software itself.

That means if an application were to fail, especially a large complex application, it would be down for some time. That can cost companies a huge amount of money in lost sales and reputational damage. But if it an application has backup and restore tools built into it, it can simply be restored to a point before it failed, and the cause of the failure isolated and resolved. No major harm done.

Which is why any software should have backup and restore tools built into it. However, traditional backup and restore tools weren’t designed for the specifics of containerised applications that employ microservices and have been built to run on Kubernetes clusters.

Which means applications that employ architectures based on Kubernetes need to integrate a backup and restore utility that is specifically designed for a container-based architecture orchestrated by Kubernetes.

That’s exactly what Velero (Ark) is.

What is Velero (Heptio Ark)? A Brief Introduction To The Kubernetes Clusters Disaster Management Tool

Velero is an open source disaster recovery management, data migration and protection tool designed to work in tandem with Kubernetes. It allows for relatively easy backup and restore through a series of checkpoints, and supports AWS, GCP, and Azure as well supported. As of version 0.6.0, Velero also offers plugins for compatibility with other backup and volume storage platforms. Velero is also used in the migration of Kubernetes resources from one cluster to another and when replicating the production environment for testing and troubleshooting.

Despite being around for a few years now, first as Heptio Ark and then the renamed Velero, there are still a lot of questions around how to set the tool up for backups.

As we use Velero with Kubernetes on AWS, we have plenty of experience with its set-up and hope that our step-by-step how-to will make it a bit easier for you, saving some time as you get started.

So, without further ado…

Step-by-Step Tutorial To Setting Up Velero (Heptio Ark) For Kubernetes on AWS

Velero requires an S3 bucket to store the pod states archive from etcd.

First, let’s create a bucket:

aws s3api create-bucket \

    --bucket <YOUR_BUCKET> \

    --region <YOUR_REGION> \

    --create-bucket-configuration LocationConstrain <YOUR_REGION>

Creating IAM user:

aws iam create-user --user-name heptio-ark

Defining user policy:

BUCKET=<YOUR_BUCKET>

 cat > heptio-ark-policy.json <<EOF

 {

     "Version": "2012-10-17",

     "Statement": [

         {

             "Effect": "Allow",

             "Action": [

                 "ec2:DescribeVolumes",

                 "ec2:DescribeSnapshots",

                 "ec2:CreateTags",

                 "ec2:CreateVolume",

                 "ec2:CreateSnapshot",

                 "ec2:DeleteSnapshot"

             ],

             "Resource": "*"

         },

         {

             "Effect": "Allow",

             "Action": [

                 "s3:GetObject",

                 "s3:DeleteObject",

                 "s3:PutObject",

                 "s3:AbortMultipartUpload",

                 "s3:ListMultipartUploadParts"

             ],

             "Resource": [

                 "arn:aws:s3:::${BUCKET}/*"

             ]

         },

         {

             "Effect": "Allow",

             "Action": [

                 "s3:ListBucket"

             ],

             "Resource": [

                 "arn:aws:s3:::${BUCKET}"

             ]

         }

     ]

 }

 EOF




 aws iam put-user-policy \

   --user-name heptio-ark \

   --policy-name heptio-ark \

   --policy-document file://heptio-ark-policy.json

Creating access key and secret key:

aws iam create-access-key --user-name heptio-ark

And we should get this:

{

     "AccessKey": {

           "UserName": "heptio-ark",

           "Status": "Active",

           "CreateDate": "2017-07-31T22:24:41.576Z",

           "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>,

           "AccessKeyId": <AWS_ACCESS_KEY_ID>

       }

  }

Creating file credentials-ark:

[default]

 aws_access_key_id=<AWS_ACCESS_KEY_ID>

 aws_secret_access_key=<AWS_SECRET_ACCESS_KEY>

Let’s run the server in Kubernetes: 

Creating file prereqs.yaml:

# Copyright 2017 the Heptio Ark contributors.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: backups.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: backups

    kind: Backup




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: schedules.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: schedules

    kind: Schedule




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: restores.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: restores

    kind: Restore




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: configs.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: configs

    kind: Config




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: downloadrequests.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: downloadrequests

    kind: DownloadRequest




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: deletebackuprequests.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: deletebackuprequests

    kind: DeleteBackupRequest




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: podvolumebackups.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: podvolumebackups

    kind: PodVolumeBackup




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: podvolumerestores.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: podvolumerestores

    kind: PodVolumeRestore




---

apiVersion: apiextensions.k8s.io/v1beta1

kind: CustomResourceDefinition

metadata:

  name: resticrepositories.ark.heptio.com

  labels:

    component: ark

spec:

  group: ark.heptio.com

  version: v1

  scope: Namespaced

  names:

    plural: resticrepositories

    kind: ResticRepository




---

apiVersion: v1

kind: Namespace

metadata:

  name: heptio-ark




---

apiVersion: v1

kind: ServiceAccount

metadata:

  name: ark

  namespace: heptio-ark

  labels:

    component: ark




---

apiVersion: rbac.authorization.k8s.io/v1beta1

kind: ClusterRoleBinding

metadata:

  name: ark

  labels:

    component: ark

subjects:

  - kind: ServiceAccount

    namespace: heptio-ark

    name: ark

roleRef:

  kind: ClusterRole

  name: cluster-admin

  apiGroup: rbac.authorization.k8s.io

Deploying to Kubernetes:

kubectl apply -f prereqs.yaml

Creating Kubernetes secret from credentials-ark file:

kubectl create secret generic cloud-credentials \

    --namespace <ARK_NAMESPACE> \

    --from-file cloud=credentials-ark

Creating files config.yaml and deployment.yaml:

# Copyright 2017 the Heptio Ark contributors.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.




---

apiVersion: ark.heptio.com/v1

kind: Config

metadata:

  namespace: heptio-ark

  name: default

persistentVolumeProvider:

  name: aws

  config:

    region: <YOUR_REGION>

backupStorageProvider:

  name: aws

  bucket: <YOUR_BUCKET>

  # Uncomment the below line to enable restic integration.

  # The format for resticLocation is <bucket>[/<prefix>],

  # e.g. "my-restic-bucket" or "my-restic-bucket/repos".

  # This MUST be a different bucket than the main Ark bucket

  # specified just above.

  # resticLocation: <YOUR_RESTIC_LOCATION>

  config:

    region: <YOUR_REGION>

backupSyncPeriod: 30m

gcSyncPeriod: 30m

scheduleSyncPeriod: 1m

restoreOnlyMode: false

1 # Copyright 2017 the Heptio Ark contributors.

2 #

3 # Licensed under the Apache License, Version 2.0 (the "License");

4 # you may not use this file except in compliance with the License.

5 # You may obtain a copy of the License at

6 #

7 #     http://www.apache.org/licenses/LICENSE-2.0

8 #

9 # Unless required by applicable law or agreed to in writing, software

10 # distributed under the License is distributed on an "AS IS" BASIS,

11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

12 # See the License for the specific language governing permissions and

13 # limitations under the License.

14

15 ---

16 apiVersion: apps/v1beta1

17 kind: Deployment

18 metadata:

19 namespace: heptio-ark

20  name: ark

21 spec:

22  replicas: 1

23  template:

24    metadata:

25     labels:

26       component: ark

27     annotations:

28        prometheus.io/scrape: "true"

29        prometheus.io/port: "8085"

30        prometheus.io/path: "/metrics"

31    spec:

32      restartPolicy: Always

33      serviceAccountName: ark

34      containers:

35        - name: ark

36          image: gcr.io/heptio-images/ark:latest

37          command:

38            - /ark

39         args:

40            - server

41          volumeMounts:

42            - name: cloud-credentials

43              mountPath: /credentials

44            - name: plugins

45              mountPath: /plugins

46            - name: scratch

47              mountPath: /scratch

48          env:

49            - name: AWS_SHARED_CREDENTIALS_FILE

50              value: /credentials/cloud

51            - name: ARK_SCRATCH_DIR

52              value: /scratch

53      volumes:

54        - name: cloud-credentials

55          secret:

56            secretName: cloud-credentials

57        - name: plugins

58          emptyDir: {}

59        - name: scratch

60          emptyDir: {}

Deploying to Kubernetes:

kubectl apply -f config.yaml

kubectl apply -f deployment.yaml

Configuring Your K8s Backups

Creating daily executable file daily.yaml:

apiVersion: ark.heptio.com/v1

kind: Schedule

metadata:

  name: daily

  namespace: heptio-ark

spec:

  schedule: 5 0 * * *

  template:

    excludedNamespaces: null

    excludedResources: null

    hooks:

      resources: null

    includeClusterResources: null

    includedNamespaces:

    - '*'

    includedResources: null

    labelSelector: null

    snapshotVolumes: true

    ttl: 168h0m0s

Deploying:

kubectl apply -f daily.yaml

Checking:

kubectl -n heptio-ark get schedules.ark.heptio.com




NAME      AGE

daily     13h

1 kubectl -n heptio-ark get backups.ark.heptio.com

2

3 NAME                   AGE

4 daily-20180814220346   13h

5 daily-20180815000534   11h

Checking S3 bucket in AWS:

aws s3 ls <BUCKET_NAME>




    PRE daily-20180814220346/

    PRE daily-20180815000534/

Checking snapshots:

aws ec2 describe-snapshots




{

            "Description": "",

            "Tags": [

                {

                    "Value": "owned",

                    "Key": "kubernetes.io/cluster/aws.cluster.com"

                },

                {

                    "Value": "internal-services",

                    "Key": "kubernetes.io/created-for/pvc/namespace"

                },

                {

                    "Value": "aws.cluster.com",

                    "Key": "KubernetesCluster"

                },

                {

                    "Value": "daily-20180814220346",

                    "Key": "ark.heptio.com/backup"

                },

                {

                    "Value": "pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",

                    "Key": "ark.heptio.com/pv"

                },

                {

                    "Value": "pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",

                    "Key": "kubernetes.io/created-for/pv/name"

                },

                {

                    "Value": "prometheus-claim0",

                    "Key": "kubernetes.io/created-for/pvc/name"

                },

                {

                    "Value": "aws.cluster.com-dynamic-pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",

                    "Key": "Name"

                }

            ],

            "Encrypted": false,

            "VolumeId": "vol-0733b80fe4057339a",

            "State": "completed",

            "VolumeSize": 489,

            "StartTime": "2018-08-14T22:03:49.000Z",

            "Progress": "100%",

            "OwnerId": "412224592913",

            "SnapshotId": "snap-04173b3945360a237"

        },

Download the Velero client for convenience: https://github.com/vmware-tanzu/velero/releases

Great, now we know our way around Velero backups, from the creation of S3 bucket and server in Kubernetes to the configuration process. Read more on secret management and Consul with Hashicorp and on Kubernetes security in our K8s consulting series blog posts.

And we hope you’ve found the above Velero installation tutorial useful and you Kubernetes clusters are now safely backed up and bullet-proof!