Kubernetes cluster backup and restore using “etcdctl” tool

Take backup of your Kubernetes cluster and restore same using etcdctl tool.

https://etcd.io/

What is etcd and etcdctl?

etcd is A distributed, reliable key-value store for the most critical data of a distributed system. etcd is a consistent and highly-available key value store used as Kubernetes’ backing store for all cluster data.

etcdctl, a command line tool for interacting with etcd server.

Install etcdctl tool:

Cluster Backup:

By the way if you not sure about values used in above command then you can check values from your /etc/kubernetes/manifests/etcd.yaml file.

Let’s confirm status of our backup using,

Now by mistake we deleted our nginx deployment or something went wrong and want to restore cluster using etcdctl tool.

Restore Cluster:

Now, update values in /etc/kubernetes/manifests/etcd.yaml to use our restored data-dir.

Update — data-dir to use new target location

--data-dir=/var/lib/etcd-from-backup

Update new initial-cluster-token to specify new cluster

--initial-cluster-token=etcd-cluster-new

Update volumes and volume mounts to point to new path

volumeMounts:
    - mountPath: /var/lib/etcd-from-backup
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /var/lib/etcd-from-backup
      type: DirectoryOrCreate
    name: etcd-data
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs

After the manifest files are modified ETCD cluster should automatically restart.

Let’s check our pods,

You can also use tools like Velero for same purpose.

Velero is an open source tool to safely backup and restore, perform disaster recovery, and migrate Kubernetes cluster resources and persistent volumes. Backup and migrate Kubernetes resources and persistent volumes.

Thank you for reading 😃

Monitoring Docker containers using cAdvisor and Prometheus

Getting Started with Kubernetes (K8s)

Microservices with gRPC