In this post in our Kubernetes consulting series we’re going to cover Kubernetes backup with Velero, previously known as Heptio Ark, within an AWS environment. Velero is also supported by the Microsoft Azure and GCP cloud platforms.
This step-by-step tutorial on how to install Velero as your backup facility is specific to an AWS environment that we’re running Kubernetes clusters on. But the installation process is almost exactly the same in other non-AWS cloud environments. Simply switch out the AWS-specific commands for those of whatever cloud platform you are running you K8s on.
Before we get into the step-by-step tutorial, let’s first quickly look at Kubernetes backups more generally, and Velero as a disaster recovery management tool. Feel free to skip down the next two short sections to move straight to the tutorial.
The headline is something of a rhetorical question, with it fairly obvious why any application needs to be backed up. It means the application and especially data it holds can be recovered if anything goes wrong.
We’ve probably all had a lost, stolen or broken laptop or smartphone before cloud storage became a relative standard. If data hadn’t been recently backed up externally – it was gone. And even if it was, setting up a repaired machine or replacement with all the same programmes etc. is a monumental pain in rear.
And that was just in the case of a hardware device holding one person’s data. When a software application, especially an enterprise-level application, fails the problem is far bigger. Not only can large volumes of data be lost (even if most of it is presumably backed up on the cloud or physical servers in multiple copies not exposed to any single point of failure) but it can be difficult and time consuming to fix the software itself.
That means if an application were to fail, especially a large complex application, it would be down for some time. That can cost companies a huge amount of money in lost sales and reputational damage. But if it an application has backup and restore tools built into it, it can simply be restored to a point before it failed, and the cause of the failure isolated and resolved. No major harm done.
Which is why any software should have backup and restore tools built into it. However, traditional backup and restore tools weren’t designed for the specifics of containerised applications that employ microservices and have been built to run on Kubernetes clusters.
Which means applications that employ architectures based on Kubernetes need to integrate a backup and restore utility that is specifically designed for a container-based architecture orchestrated by Kubernetes.
That’s exactly what Velero (Ark) is.
Velero is an open source disaster recovery management, data migration and protection tool designed to work in tandem with Kubernetes. It allows for relatively easy backup and restore through a series of checkpoints, and supports AWS, GCP, and Azure as well supported. As of version 0.6.0, Velero also offers plugins for compatibility with other backup and volume storage platforms. Velero is also used in the migration of Kubernetes resources from one cluster to another and when replicating the production environment for testing and troubleshooting.
Despite being around for a few years now, first as Heptio Ark and then the renamed Velero, there are still a lot of questions around how to set the tool up for backups.
As we use Velero with Kubernetes on AWS, we have plenty of experience with its set-up and hope that our step-by-step how-to will make it a bit easier for you, saving some time as you get started.
So, without further ado…
Velero requires an S3 bucket to store the pod states archive from etcd.
|
1 2 3 4 5 6 7 |
aws s3api create-bucket \ Â Â Â Â --bucket <YOUR_BUCKET> \ Â Â Â Â --region <YOUR_REGION> \ Â Â Â Â --create-bucket-configuration LocationConstrain <YOUR_REGION> |
|
1 |
aws iam create-user --user-name heptio-ark |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 |
BUCKET=<YOUR_BUCKET> Â cat > heptio-ark-policy.json <<EOF Â { Â Â Â Â "Version": "2012-10-17", Â Â Â Â "Statement": [ Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â "Effect": "Allow", Â Â Â Â Â Â Â Â Â Â Â Â "Action": [ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "ec2:DescribeVolumes", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "ec2:DescribeSnapshots", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "ec2:CreateTags", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "ec2:CreateVolume", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "ec2:CreateSnapshot", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "ec2:DeleteSnapshot" Â Â Â Â Â Â Â Â Â Â Â Â ], Â Â Â Â Â Â Â Â Â Â Â Â "Resource": "*" Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â "Effect": "Allow", Â Â Â Â Â Â Â Â Â Â Â Â "Action": [ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "s3:GetObject", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "s3:DeleteObject", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "s3:PutObject", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "s3:AbortMultipartUpload", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "s3:ListMultipartUploadParts" Â Â Â Â Â Â Â Â Â Â Â Â ], Â Â Â Â Â Â Â Â Â Â Â Â "Resource": [ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "arn:aws:s3:::${BUCKET}/*" Â Â Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â Â "Effect": "Allow", Â Â Â Â Â Â Â Â Â Â Â Â "Action": [ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "s3:ListBucket" Â Â Â Â Â Â Â Â Â Â Â Â ], Â Â Â Â Â Â Â Â Â Â Â Â "Resource": [ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "arn:aws:s3:::${BUCKET}" Â Â Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â Â } Â Â Â Â ] Â } Â EOF Â aws iam put-user-policy \ Â Â --user-name heptio-ark \ Â Â --policy-name heptio-ark \ Â Â --policy-document file://heptio-ark-policy.json |
|
1 |
aws iam create-access-key --user-name heptio-ark |
And we should get this:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
{ Â Â Â Â "AccessKey": { Â Â Â Â Â Â Â Â Â Â "UserName": "heptio-ark", Â Â Â Â Â Â Â Â Â Â "Status": "Active", Â Â Â Â Â Â Â Â Â Â "CreateDate": "2017-07-31T22:24:41.576Z", Â Â Â Â Â Â Â Â Â Â "SecretAccessKey": <AWS_SECRET_ACCESS_KEY>, Â Â Â Â Â Â Â Â Â Â "AccessKeyId": <AWS_ACCESS_KEY_ID> Â Â Â Â Â Â } Â Â } |
|
1 2 3 4 5 |
[default] Â aws_access_key_id=<AWS_ACCESS_KEY_ID> Â aws_secret_access_key=<AWS_SECRET_ACCESS_KEY> |
Creating file prereqs.yaml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 |
# Copyright 2017 the Heptio Ark contributors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #    http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: backups.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: backups     kind: Backup --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: schedules.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: schedules     kind: Schedule --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: restores.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: restores     kind: Restore --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: configs.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: configs     kind: Config --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: downloadrequests.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: downloadrequests     kind: DownloadRequest --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: deletebackuprequests.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: deletebackuprequests     kind: DeleteBackupRequest --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: podvolumebackups.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: podvolumebackups     kind: PodVolumeBackup --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: podvolumerestores.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: podvolumerestores     kind: PodVolumeRestore --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata:   name: resticrepositories.ark.heptio.com   labels:     component: ark spec:   group: ark.heptio.com   version: v1   scope: Namespaced   names:     plural: resticrepositories     kind: ResticRepository --- apiVersion: v1 kind: Namespace metadata:   name: heptio-ark --- apiVersion: v1 kind: ServiceAccount metadata:   name: ark   namespace: heptio-ark   labels:     component: ark --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata:   name: ark   labels:     component: ark subjects:   - kind: ServiceAccount     namespace: heptio-ark     name: ark roleRef:   kind: ClusterRole   name: cluster-admin   apiGroup: rbac.authorization.k8s.io Deploying to Kubernetes: kubectl apply -f prereqs.yaml |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 |
kubectl create secret generic cloud-credentials \     --namespace <ARK_NAMESPACE> \     --from-file cloud=credentials-ark Creating files config.yaml and deployment.yaml: # Copyright 2017 the Heptio Ark contributors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #    http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. --- apiVersion: ark.heptio.com/v1 kind: Config metadata:   namespace: heptio-ark   name: default persistentVolumeProvider:   name: aws   config:     region: <YOUR_REGION> backupStorageProvider:   name: aws   bucket: <YOUR_BUCKET>   # Uncomment the below line to enable restic integration.   # The format for resticLocation is <bucket>[/<prefix>],   # e.g. "my-restic-bucket" or "my-restic-bucket/repos".   # This MUST be a different bucket than the main Ark bucket   # specified just above.   # resticLocation: <YOUR_RESTIC_LOCATION>   config:     region: <YOUR_REGION> backupSyncPeriod: 30m gcSyncPeriod: 30m scheduleSyncPeriod: 1m restoreOnlyMode: false 1 # Copyright 2017 the Heptio Ark contributors. 2 # 3 # Licensed under the Apache License, Version 2.0 (the "License"); 4 # you may not use this file except in compliance with the License. 5 # You may obtain a copy of the License at 6 # 7 #    http://www.apache.org/licenses/LICENSE-2.0 8 # 9 # Unless required by applicable law or agreed to in writing, software 10 # distributed under the License is distributed on an "AS IS" BASIS, 11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 # See the License for the specific language governing permissions and 13 # limitations under the License. 14 15 --- 16 apiVersion: apps/v1beta1 17 kind: Deployment 18 metadata: 19 namespace: heptio-ark 20  name: ark 21 spec: 22  replicas: 1 23  template: 24    metadata: 25     labels: 26       component: ark 27     annotations: 28        prometheus.io/scrape: "true" 29        prometheus.io/port: "8085" 30       prometheus.io/path: "/metrics" 31    spec: 32      restartPolicy: Always 33      serviceAccountName: ark 34      containers: 35        - name: ark 36          image: gcr.io/heptio-images/ark:latest 37          command: 38            - /ark 39         args: 40            - server 41          volumeMounts: 42            - name: cloud-credentials 43              mountPath: /credentials 44            - name: plugins 45              mountPath: /plugins 46            - name: scratch 47              mountPath: /scratch 48          env: 49            - name: AWS_SHARED_CREDENTIALS_FILE 50              value: /credentials/cloud 51            - name: ARK_SCRATCH_DIR 52              value: /scratch 53      volumes: 54        - name: cloud-credentials 55          secret: 56            secretName: cloud-credentials 57        - name: plugins 58          emptyDir: {} 59        - name: scratch 60          emptyDir: {} Deploying to Kubernetes: kubectl apply -f config.yaml kubectl apply -f deployment.yaml |
Configuring Your K8s Backups
Creating daily executable file daily.yaml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 |
apiVersion: ark.heptio.com/v1 kind: Schedule metadata:   name: daily   namespace: heptio-ark spec:   schedule: 5 0 * * *   template:     excludedNamespaces: null     excludedResources: null     hooks:       resources: null     includeClusterResources: null     includedNamespaces:     - '*'     includedResources: null     labelSelector: null     snapshotVolumes: true     ttl: 168h0m0s Deploying: kubectl apply -f daily.yaml |
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 |
kubectl -n heptio-ark get schedules.ark.heptio.com NAME      AGE daily    13h 1 kubectl -n heptio-ark get backups.ark.heptio.com 2 3 NAME                  AGE 4 daily-20180814220346  13h 5 daily-20180815000534  11h Checking S3 bucket in AWS: aws s3 ls <BUCKET_NAME>     PRE daily-20180814220346/     PRE daily-20180815000534/ Checking snapshots: aws ec2 describe-snapshots {             "Description": "",             "Tags": [                 {                     "Value": "owned",                     "Key": "kubernetes.io/cluster/aws.cluster.com"                 },                 {                     "Value": "internal-services",                     "Key": "kubernetes.io/created-for/pvc/namespace"                 },                 {                     "Value": "aws.cluster.com",                     "Key": "KubernetesCluster"                 },                 {                     "Value": "daily-20180814220346",                     "Key": "ark.heptio.com/backup"                 },                 {                     "Value": "pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",                     "Key": "ark.heptio.com/pv"                 },                 {                     "Value": "pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",                     "Key": "kubernetes.io/created-for/pv/name"                 },                 {                     "Value": "prometheus-claim0",                     "Key": "kubernetes.io/created-for/pvc/name"                 },                 {                     "Value": "aws.cluster.com-dynamic-pvc-37895bcc-67ec-11e8-b075-0689fdf74ef8",                     "Key": "Name"                 }             ],             "Encrypted": false,             "VolumeId": "vol-0733b80fe4057339a",             "State": "completed",             "VolumeSize": 489,             "StartTime": "2018-08-14T22:03:49.000Z",             "Progress": "100%",             "OwnerId": "412224592913",             "SnapshotId": "snap-04173b3945360a237"         }, |
Download the Velero client for convenience:Â https://github.com/vmware-tanzu/velero/releases
Great, now we know our way around Velero backups, from the creation of S3 bucket and server in Kubernetes to the configuration process. Read more on secret management and Consul with Hashicorp and on Kubernetes security in our K8s consulting series blog posts.
And we hope you’ve found the above Velero installation tutorial useful and you Kubernetes clusters are now safely backed up and bullet-proof!
Add comment