etcd Operation and Backup

etcd is the database that holds all of a cluster's state — every object, every status. The cluster is, in a real sense, whatever etcd says it is. Operating it well and backing it up reliably is the single most important durability task for a self-managed cluster.

Lose etcd without a backup and the cluster's desired state is gone. Restore from a tested backup and you can rebuild from disaster. The gap between those two outcomes is preparation you do before you need it.

What etcd Stores and Why It Matters

Every Kubernetes object lives in etcd: Deployments, Services, ConfigMaps, Secrets, and their status. The API server is the only component that talks to it. Because it is the sole source of truth, etcd's health is the cluster's health — and its loss is the cluster's loss. Treat it as the critical database it is, not as an implementation detail.

Quorum and Performance

etcd cluster · 5 members (Raft)

Quorum: 3 of 5 members must agree to commit a write

m1member 1

m2member 2

m3member 3

m4member 4

m5member 5

Lose two members and three remain — quorum holds, writes continue. Lose three and the cluster goes read-only until restored. Always run an odd number.

etcd uses the Raft consensus protocol and needs a quorum — a majority of members — to accept writes. That is why you run an odd number: three members tolerate one failure, five tolerate two. An even count adds cost without improving fault tolerance and complicates quorum. etcd is also acutely sensitive to disk latency; on slow storage, write latency rises and the whole control plane becomes sluggish and flaky. Give it fast, dedicated disks.

Backup with Snapshots

The backup mechanism is a point-in-time snapshot: etcdctl snapshot save writes a consistent copy of the entire keyspace to a file. Snapshots should be automated on a schedule and shipped off the cluster — a backup on the same failed disk is no backup. Crucially, take a snapshot before any risky operation, especially a control-plane upgrade. On managed clusters the provider handles etcd backup, which is one of the larger operational burdens that managed Kubernetes removes.

# back up
ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=ca.crt --cert=server.crt --key=server.key

# restore (rebuilds the data dir from the snapshot)
etcdctl snapshot restore snapshot.db \
  --data-dir=/var/lib/etcd-restored

Restore, Defrag, and Encryption

A backup you have never restored is a hope, not a plan — rehearse the restore, because the procedure (stop the API server, restore the data dir, restart) is unforgiving under real pressure. Over time etcd needs compaction and defragmentation to reclaim space from old revisions. And remember that Secrets sit in etcd: without encryption at rest (Topic 35), an etcd snapshot is a plaintext copy of every secret — secure the backups accordingly.