Kubernetes Backup & Restore

Velero etcd Backup Application Backup Disaster Recovery

A comprehensive guide to backing up and restoring Kubernetes clusters covering Velero, etcd backup, application backup, disaster recovery strategies, and practical implementation for production environments.

Velero etcd Backup Disaster Recovery Application Backup

Why Backup and Restore Matter

Kubernetes clusters are critical infrastructure for modern applications. A disaster (cluster failure, etcd corruption, accidental deletion, security breach) can cause significant downtime and data loss. A robust backup and restore strategy is essential for:

Disaster Recovery: Recover from cluster failures, hardware issues, or cloud provider outages
Data Protection: Prevent data loss from accidental deletions or application bugs
Compliance: Meet regulatory requirements for data retention and audit
Migration: Move applications between clusters or cloud providers
Testing: Create copies of production environments for testing

Key Backup Principles:

RPO (Recovery Point Objective): Maximum acceptable data loss (e.g., 1 hour)
RTO (Recovery Time Objective): Maximum acceptable downtime (e.g., 4 hours)
3-2-1 Rule: 3 copies, 2 different media, 1 offsite
Regular Testing: Restore backups regularly to verify integrity
Automation: Automate backup creation and retention

Backup Options Overview

Velero

Application-level backup

Velero is the most popular Kubernetes backup tool. It backs up Kubernetes objects (Deployments, Services, ConfigMaps, etc.) and optionally persistent volumes. Supports scheduled backups and restores.

Full application backup & restore

etcd Backup

Cluster state backup

etcd stores all cluster state including configuration, secrets, and resource status. Backing up etcd is essential for cluster recovery in case of control plane failure.

Cluster recovery

PV/CSI Snapshots

Volume-level backup

CSI drivers support volume snapshots, creating point-in-time copies of persistent volumes. These can be used to restore data or create clones.

Data recovery

Database Backups

Application-specific backups

Databases like PostgreSQL, MySQL, and MongoDB have their own backup tools (pg_dump, mysqldump, mongodump) for logical backups.

Database-specific recovery

Velero: Kubernetes Backup & Restore

Velero is the industry-standard tool for backing up and restoring Kubernetes clusters. It provides a comprehensive solution for disaster recovery, cluster migration, and application backup.

                # Install Velero (AWS example)
velero install \
    --provider aws \
    --bucket my-backup-bucket \
    --backup-location-config region=us-east-1 \
    --snapshot-location-config region=us-east-1 \
    --plugins velero/velero-plugin-for-aws:v1.7.0 \
    --secret-file ./credentials-velero

# Install Velero (Azure example)
velero install \
    --provider azure \
    --bucket my-backup-container \
    --secret-file ./credentials-azure \
    --backup-location-config resourceGroup=velero-rg,storageAccount=velerosa \
    --snapshot-location-config apiTimeout=5m

# Install Velero (GCP example)
velero install \
    --provider gcp \
    --bucket my-backup-bucket \
    --secret-file ./credentials-gcp \
    --backup-location-config region=us-central1

# Create a backup
velero backup create full-backup --include-namespaces default,production

# Create a backup with specific resources
velero backup create app-backup \
    --include-namespaces default \
    --include-resources deployments,services,configmaps,secrets

# Exclude resources from backup
velero backup create partial-backup \
    --include-namespaces default \
    --exclude-resources events,events.events.k8s.io

# Schedule daily backups
velero schedule create daily-backup \
    --schedule="0 2 * * *" \
    --include-namespaces default,production \
    --ttl=168h  # Keep for 7 days

# View backups
velero backup get

# Describe backup details
velero backup describe full-backup

# Restore from backup
velero restore create --from-backup full-backup

# Restore specific namespace
velero restore create --from-backup full-backup --include-namespaces production

# Restore with mapping (for cluster migration)
velero restore create \
    --from-backup full-backup \
    --namespace-mappings default:new-default,production:new-production
            

Velero Best Practices:

Store backups in a different region from your cluster
Use scheduled backups for regular backups (daily, weekly)
Set TTL to automatically expire old backups
Test restores regularly in a staging environment
Enable volume snapshots for persistent data backup
Monitor backup status and alert on failures

etcd Backup: Cluster State Protection

etcd stores all cluster state and is the single source of truth for Kubernetes. Backing up etcd is essential for recovering from control plane failures or cluster corruption.

                # etcd backup (using etcdctl)
ETCDCTL_API=3 etcdctl snapshot save /backups/etcd-$(date +%Y%m%d-%H%M%S).db \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key

# Verify snapshot integrity
ETCDCTL_API=3 etcdctl snapshot status /backups/etcd-20250115-020000.db

# etcd backup script (automated)
#!/bin/bash
BACKUP_DIR="/backups/etcd"
DATE=$(date +%Y%m%d-%H%M%S)
ETCDCTL_API=3 etcdctl snapshot save $BACKUP_DIR/etcd-$DATE.db \
    --endpoints=https://127.0.0.1:2379 \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key

# Keep only last 30 days of backups
find $BACKUP_DIR -name "etcd-*.db" -mtime +30 -delete

# Restore etcd from snapshot
ETCDCTL_API=3 etcdctl snapshot restore /backups/etcd-20250115-020000.db \
    --data-dir /var/lib/etcd-restore

# For HA etcd cluster, restore on all nodes
# Stop etcd, replace data directory, restart etcd

# Check etcd health
etcdctl endpoint health --cluster

# Monitor etcd metrics
curl http://localhost:2382/metrics
            

Critical: etcd backup is the most critical backup for your cluster. Without it, you cannot recover the cluster state. Store etcd backups in a secure, off-cluster location. Test restore procedures regularly.

Volume Snapshots with CSI

CSI drivers support volume snapshots, enabling point-in-time backups of persistent volumes. This is essential for stateful applications and databases.

                # VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-snapshot-class
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  force-create: "true"

# Create VolumeSnapshot
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: pvc-snapshot
  namespace: default
spec:
  volumeSnapshotClassName: ebs-snapshot-class
  source:
    persistentVolumeClaimName: data-postgres-0

# Check snapshot status
kubectl get volumesnapshot
kubectl describe volumesnapshot pvc-snapshot

# Restore from snapshot (create new PVC)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-restored
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  dataSource:
    name: pvc-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

# Using Velero with volume snapshots
velero backup create app-with-snapshots \
    --include-namespaces default \
    --snapshot-volumes

# Restore with volume snapshots
velero restore create --from-backup app-with-snapshots
            

Snapshot Best Practices:

Use CSI drivers that support snapshots (EBS CSI, GCE PD CSI, etc.)
Snapshots are more efficient than copying data
Combine with Velero for application-level backup
Test snapshot restore procedures regularly
Implement snapshot retention policies

Application-Specific Backup Strategies

                # PostgreSQL backup (pg_dump)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-backup
spec:
  schedule: "0 2 * * *"  # Daily at 2 AM
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: postgres:15
            command:
            - /bin/sh
            - -c
            - |
              export PGPASSWORD=$POSTGRES_PASSWORD
              pg_dump -h postgres-0.postgres -U postgres mydb > /backups/mydb-$(date +%Y%m%d).sql
            env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: postgres-secret
                  key: password
            volumeMounts:
            - name: backup
              mountPath: /backups
          restartPolicy: Never
          volumes:
          - name: backup
            persistentVolumeClaim:
              claimName: backup-pvc

# MySQL backup (mysqldump)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: mysql-backup
spec:
  schedule: "0 3 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: mysql:8.0
            command:
            - /bin/sh
            - -c
            - |
              mysqldump -h mysql-0.mysql -u root -p$MYSQL_ROOT_PASSWORD --all-databases > /backups/all-dbs-$(date +%Y%m%d).sql
            env:
            - name: MYSQL_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mysql-secret
                  key: root-password
            volumeMounts:
            - name: backup
              mountPath: /backups
          restartPolicy: Never

# MongoDB backup (mongodump)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: mongodb-backup
spec:
  schedule: "0 4 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: mongo:6.0
            command:
            - /bin/sh
            - -c
            - |
              mongodump --host mongodb-0.mongodb --out /backups/mongo-$(date +%Y%m%d)
            volumeMounts:
            - name: backup
              mountPath: /backups
          restartPolicy: Never
            

Application Backup Best Practices:

Use database-specific tools for logical backups
Schedule backups during low-traffic periods
Compress backups to save storage space
Store backups in a separate location (object storage)
Implement backup verification and alerting
Test database restore procedures regularly

Disaster Recovery Strategies

A comprehensive disaster recovery strategy combines multiple backup approaches to ensure data protection and cluster recovery.

Full Cluster Backup

Velero + etcd backup

Combine Velero (application backup) with etcd backup (cluster state). Store backups in multiple locations (object storage, cloud regions). Test full cluster recovery in a test environment.

Complete cluster recovery

Multi-Region DR

Geographically distributed backups

Store backups in different cloud regions or providers. Use Velero with multiple backup locations. Implement active-passive or active-active cluster setups for high availability.

Regional failure recovery

Point-in-Time Recovery

Granular data recovery

Use database-specific PITR (Point-in-Time Recovery) with WAL archives. Combine volume snapshots for frequent recovery points. Document and test granular recovery procedures.

Data corruption recovery

Cross-Cluster Migration

Velero for cluster migration

Use Velero to migrate applications between clusters. Backup from source cluster, restore to target cluster with namespace mapping. Useful for version upgrades and cloud migrations.

Cluster upgrades, cloud migration

                # Disaster Recovery Procedure

# 1. etcd restore (for control plane failure)
# Stop etcd on all nodes
systemctl stop etcd

# Restore from snapshot on each node
ETCDCTL_API=3 etcdctl snapshot restore /backups/etcd-20250115-020000.db \
    --data-dir /var/lib/etcd-restore

# Start etcd with new data directory
systemctl start etcd

# 2. Velero restore (for application failure)
# Check available backups
velero backup get

# Restore full cluster
velero restore create --from-backup full-backup

# 3. Database restore (for data corruption)
# Restore from database backup
pg_restore -h postgres-0.postgres -U postgres -d mydb /backups/mydb-20250115.sql

# 4. Verify cluster health
kubectl get nodes
kubectl get pods --all-namespaces
kubectl get pv,pvc --all-namespaces

# 5. Post-recovery validation
# Check application functionality
# Verify data integrity
# Monitor for any issues
            

RTO and RPO Planning

Define and document your Recovery Time Objective (RTO) and Recovery Point Objective (RPO) to guide your backup strategy.

Objective	Definition	Example Target	Strategy
RTO	Maximum acceptable downtime	4 hours	Automated restore, documented procedures
RPO	Maximum acceptable data loss	1 hour	Hourly backups, WAL archiving
Backup Frequency	How often backups are taken	Daily (full), Hourly (incremental)	Velero schedules, database backups
Retention Period	How long backups are kept	30 days	TTL settings, lifecycle policies
Restore Testing	How often restores are tested	Monthly	Automated restore tests

Critical: A backup is only useful if you can successfully restore from it. Test your restore procedures regularly. Document the restore process and ensure the team is trained. Consider using tools like Velero's backup validation hooks.

Backup Monitoring and Alerting

                # Velero backup monitoring
# Check backup status
velero backup get
velero backup describe full-backup

# Monitor backup metrics (Prometheus)
# Velero exposes metrics on port 8085
curl http://localhost:8085/metrics | grep velero

# Key metrics to monitor:
# - velero_backups_total (total backups)
# - velero_backup_success_total (successful backups)
# - velero_backup_failure_total (failed backups)
# - velero_backup_duration_seconds (backup duration)
# - velero_restore_success_total (successful restores)

# Alerting rules (Prometheus)
- alert: VeleroBackupFailed
  expr: increase(velero_backup_failure_total[5m]) > 0
  annotations:
    summary: "Velero backup failed"
    description: "A Velero backup has failed. Check the logs for details."

# Check backup logs
kubectl logs -n velero deployment/velero

# Velero backup validation hook
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: validated-backup
spec:
  hooks:
    resources:
    - name: post-backup-validation
      includedNamespaces:
      - default
      post:
      - exec:
          container: app
          command:
          - /bin/sh
          - -c
          - /scripts/validate-backup.sh
          onError: Fail
            

Monitoring Best Practices:

Monitor backup success/failure rates
Alert on backup failures immediately
Track backup storage usage and costs
Monitor backup completion time against SLOs
Implement automated backup health checks

Frequently Asked Questions

What is the difference between Velero and etcd backup?

Velero backs up Kubernetes objects (Deployments, Services, ConfigMaps, etc.) and can optionally backup persistent volumes. etcd backup backs up the entire cluster state including all resources. etcd backup is essential for control plane recovery, while Velero is better for application-level backup and migration.

How often should I back up my Kubernetes cluster?

The frequency depends on your RPO requirements. For most production clusters, daily full backups with Velero plus hourly etcd backups are recommended. For critical databases, consider WAL archiving for point-in-time recovery. Schedule backups during low-traffic periods.

Can I restore a backup to a different cluster?

Yes! Velero supports cluster migration. Use `--namespace-mappings` to map namespaces between clusters. You may need to adjust StorageClasses and other cloud-specific configurations. Velero also supports restoring to clusters with different Kubernetes versions.

What happens to PVCs during backup and restore?

With Velero, PVCs are backed up as part of the backup. When restoring with `--snapshot-volumes`, Velero can restore volume data from snapshots. Without snapshots, PVCs are restored as empty volumes. Volume snapshots require CSI drivers.

How do I test my backup restore process?

Create a separate test cluster (or namespace) and perform a restore. Validate application functionality and data integrity. Automate restore tests using Velero hooks. Document and practice the restore procedure regularly.

What storage locations are supported by Velero?

Velero supports AWS S3, Azure Blob Storage, Google Cloud Storage, and any S3-compatible storage (MinIO, Ceph, etc.). Use `--backup-location-config` to configure the storage location. Support for multiple backup locations is available.

How do I handle secrets in backups?

Secrets are backed up by default with Velero. To encrypt secrets, enable Velero's encryption feature or use sealed secrets. Ensure backup storage is secure. Use Kubernetes encryption at rest for etcd.

What is the cost impact of backups?

Backups incur storage costs (object storage) and data transfer costs. Use lifecycle policies to automatically delete old backups. Compress backups to reduce storage costs. Monitor backup storage usage and adjust retention policies as needed.

Previous: Stateful Applications Next: Kubernetes Security

A robust backup and restore strategy is essential for production Kubernetes clusters. Implement regular backups, test your restores, and document your disaster recovery procedures to ensure business continuity.