Backup and disaster recovery, honestly

“We back up the cluster” usually means one of three quite different things, and conflating them is how people discover — mid-incident — that they can’t actually restore. There are three layers, and each needs its own answer.

Watch on YouTube ↗

Layer 1 — etcd (the cluster’s brain)

etcd holds all cluster state: every object, config, and secret. If the control plane is lost, an etcd snapshot is what rebuilds it. OpenShift ships a script on the control-plane nodes:

# run on a control-plane node
sudo /usr/local/bin/cluster-backup.sh /home/core/backup

That produces a snapshot plus the static pod resources. Get it off the node — to object storage (S3/MinIO) on a schedule. An etcd backup sitting only on the master it came from protects you against nothing that also takes out the master.

Caution

etcd backup restores the cluster, not your application data. Restoring etcd rolls cluster state back to snapshot time; it does nothing for the contents of your PostgreSQL volumes. Never treat it as a data backup.

Layer 2 — cluster resources and PVs (OADP / Velero)

For “restore my namespaces, their objects, and their volumes” — including to a different cluster for DR — the tool is Velero, shipped by Red Hat as the OADP operator (OpenShift API for Data Protection). It backs up Kubernetes objects to object storage and can move volume data via CSI snapshots or a file-level copy.

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: payments-daily
  namespace: openshift-adp
spec:
  includedNamespaces: ["payments-prod"]
  storageLocation: default
  ttl: 720h0m0s

OADP is the backbone of cluster-to-cluster DR and namespace-level restore. It’s what you schedule for the “our cluster is gone, stand it up elsewhere” scenario.

Layer 3 — application-consistent data

This is the layer people skip, and the one that hurts. A CSI snapshot or a volume-level copy captures the disk at a moment in time — but a database mid-write can be snapshotted in an inconsistent state, giving you a backup that restores into a corrupt or recovery-needing database.

For stateful data, prefer application-aware backups:

• Databases: use the engine’s own mechanism (continuous WAL archiving + base backups for PostgreSQL, for instance — a Postgres operator like CloudNativePG does this declaratively to object storage, giving point-in-time recovery).
• Where you must snapshot at the volume level, quiesce or use a pre/post-snapshot hook so the app flushes to a consistent state first.

CSI snapshots are excellent for fast rollback of a volume; they are not, on their own, a database backup.

What a real strategy looks like

Layer	Tool	Protects against	Cadence
etcd snapshot → object store	cluster-backup.sh + cron	Control-plane loss	Hourly/daily
Namespaces + PVs	OADP / Velero	Namespace loss, whole-cluster DR	Daily
App data (PITR)	DB operator / native tooling	Data corruption, “restore to 14:03”	Continuous

Enterprise backup suites (NetBackup, Commvault and similar) plug in through CSI and OADP integrations — useful when policy requires backups to land in the same system as everything else, but the layering above is what actually matters.

The rule that saves you

A backup you have never restored is a hypothesis, not a backup. Schedule restore drills — into a scratch namespace or a DR cluster — on a calendar, and treat a successful restore, not a successful backup job, as the thing that’s green.