Runbook: Disaster Recovery for the Management Plane

Status: Draft — implementation pending

Purpose

Recover the Inari management plane in the event of Kubernetes cluster failure, data corruption, or total loss of the management-plane cluster. This runbook covers Kubernetes-level recovery only; underlying bare-metal infrastructure is handled separately.

Prerequisites

• Current backups of etcd, persistent volumes, and Git repository state
• Access to the replacement Kubernetes cluster or rebuilt control plane
• Access to 7KGroup secret store

Steps

1. Verify or restore the underlying Kubernetes cluster on bare metal.
2. Restore etcd and persistent volume backups.
3. Reinstall core management-plane components (ArgoCD, Mimir, Loki, Grafana, Keycloak, Vault).
4. Restore management-plane ArgoCD applications from Git.
5. Restore observability stack state and verify Alloy clients can reconnect.
6. Restore Keycloak realm and Vault root key from backup.
7. Validate observability ingestion and fleet dashboards.
8. Notify affected clients if any observability gaps occurred.

TODO

• Define backup tooling and schedule.
• Add restore commands for Velero / etcd / Longhorn / Ceph.
• Define RPO/RTO targets.