5. Secret Management¶
Secrets in Inari are client-scoped. Each client control plane fetches its own secrets from its own secret store. 7KGroup does not centralize client secrets in the management plane.
There is one intentional exception: cluster provisioning credentials. The management plane holds a limited credential to provision and manage the lifecycle of the client control plane itself. Once the cluster exists, ongoing workload and platform credentials are client-scoped.
5.1 Secret Categories¶
| Secret | Owner | Storage | Delivery Mechanism |
|---|---|---|---|
| Client cloud credentials | Client | Client's chosen secret store (e.g., AWS Secrets Manager, Vault) | External Secrets Operator in client cluster |
| Dex IdP client secrets | Client | Client's secret store or sealed secret | External Secrets Operator or Sealed Secrets |
| Alloy mTLS certificates | Client | Vault PKI in management plane | cert-manager + Vault issuer in client cluster |
| Client ArgoCD Git credential | 7KGroup | 7KGroup secret store / GitHub per-client deploy key | Mounted into client ArgoCD via External Secrets Operator |
| ESO bootstrap secret | 7KGroup | Generated during onboarding | Kubernetes secret created in client cluster by 7KGroup |
| Cluster provisioning credentials | 7KGroup | Management-plane secret store | Used by management-plane Crossplane ProviderConfig |
| Keycloak credentials | 7KGroup | Management-plane secret store | Injected into Dex via External Secrets Operator |
5.2 Core Rule¶
No secrets are stored in Git. All sensitive material is delivered via External Secrets Operator, Sealed Secrets, or cert-manager.
5.3 Client Cloud Credentials¶
Each client cluster uses External Secrets Operator to pull cloud provider credentials from the client's own secret store. Example stores:
- AWS Secrets Manager
- HashiCorp Vault
- GCP Secret Manager
- Azure Key Vault
The ProviderConfig resources in the client overlay reference Kubernetes secrets that ESO populates.
5.4 Dex IdP Secrets¶
Dex connectors require client secrets (e.g., Google OAuth client secret, Microsoft Entra application secret). These are stored in the client's secret store and mounted into the Dex deployment via External Secrets Operator.
5.5 Alloy mTLS¶
Alloy connects to the management-plane Mimir/Loki using mTLS. Each client cluster runs cert-manager with a Vault issuer. Vault runs in the management plane with a PKI engine and per-client Kubernetes auth roles. See ADR-011: mTLS CA Strategy.
5.6 Management-Plane Secret Store¶
The management plane uses HashiCorp Vault as its central secret store for:
- Keycloak admin credentials
- ArgoCD admin credentials
- Observability stack admin credentials
- Management-plane Crossplane provisioning credentials
- Vault root key backup (offline)
These secrets are injected into management-plane pods via External Secrets Operator or Vault Agent Injector.
5.7 External Secrets Operator Bootstrap¶
External Secrets Operator (ESO) in the client cluster fetches secrets from the client's secret store. To do this, ESO needs an initial authentication secret.
Preferred Bootstrap Mechanisms¶
| Client Secret Store | Bootstrap Mechanism | Notes |
|---|---|---|
| AWS Secrets Manager | IRSA (IAM Roles for Service Accounts) | No long-lived secret stored in the cluster. ESO assumes an IAM role via OIDC. |
| HashiCorp Vault | Short-lived Vault token or Kubernetes auth | Tokens are rotated automatically; no static credentials in Git. |
| GCP Secret Manager | Workload Identity | Equivalent to IRSA on GCP. |
| Azure Key Vault | AAD Workload Identity | Equivalent to IRSA on Azure. |
Bootstrap Flow¶
- During client onboarding, 7KGroup creates or receives credentials for the client's secret store (e.g., an AWS IAM role, a Vault token, or a service account key).
- After the client cluster is provisioned, 7KGroup creates an initial Kubernetes secret in the client cluster containing the ESO authentication material.
- ESO uses this bootstrap secret to configure a
ClusterSecretStoreorSecretStore. - ESO then fetches the real platform secrets (Dex IdP secrets, cloud credentials, ArgoCD Git credential) from the client's secret store.
- Once ESO is operational, the bootstrap secret is rotated or removed if the secret store supports dynamic credentials (e.g., IRSA, Workload Identity, Kubernetes auth).
The bootstrap secret is the only client-scoped secret intentionally created by 7KGroup inside the client cluster. It is not stored in Git.
Management-Plane Secrets via ESO¶
Management-plane workloads that need client-scoped or shared secrets (e.g., ArgoCD admin, Keycloak admin, Vault config tokens) fetch them from Vault via ESO. ESO in the management plane authenticates to Vault using the management-plane Kubernetes auth backend.
One-Off Secrets¶
All other secrets (Dex client secrets, cloud credentials, TLS certs, API keys) are fetched on-demand by ESO from the client's secret store. They are never committed to Git and never stored long-term in Kubernetes unless required by the consuming workload.
5.7 Client ArgoCD Git Credential¶
Each client-local ArgoCD instance authenticates to the central 7k-inari repository using a dedicated credential. Options:
- GitHub per-client deploy key: read-only SSH key scoped to
7k-inari - GitHub App credential: read-only installation token
- GitHub personal access token: least preferred; rotate frequently
The credential is generated during onboarding, stored in the client's secret store, and mounted into the client ArgoCD namespace via External Secrets Operator.
5.8 Cluster Provisioning Credentials¶
The management plane holds a limited cloud credential used by Crossplane to provision and manage the lifecycle of the client control plane (EKS cluster, IAM roles, VPC, node groups). This credential:
- Lives in the management-plane secret store
- Is scoped to infrastructure provisioning only
- Does not grant access to client workloads or application data
- Is distinct from the client-scoped credentials used by Crossplane inside the client cluster for application infrastructure (RDS, S3, etc.)
- Is rotated quarterly or on compromise
This is an intentional exception to the client-scoped secret rule, justified by the need to automate cluster lifecycle from the management plane.
5.9 Management Plane Secrets¶
Secrets required by the management plane itself (Keycloak admin credentials, ArgoCD admin password, observability stack admin credentials, Vault root key) are stored in 7KGroup's own secret store and injected into the self-hosted management cluster.
5.10 Secret Rotation¶
| Secret | Rotation Frequency | Owner |
|---|---|---|
| Cloud credentials | Per client policy / quarterly recommended | Client |
| Dex IdP secrets | Per IdP policy / on employee departure | Client |
| Alloy mTLS certs | Short-lived (e.g., 30–90 days) | 7KGroup |
| Client ArgoCD Git credential | Quarterly or on compromise | 7KGroup |
| ESO bootstrap secret | Rotated to IRSA/Workload Identity after onboarding; otherwise quarterly | 7KGroup |
| Cluster provisioning credentials | Quarterly or on compromise | 7KGroup |
| ArgoCD admin | On personnel change | 7KGroup |
See ADR-006: Grafana Alloy, ADR-010: Client-Local ArgoCD, ADR-011: mTLS CA Strategy, and ADR-012: Management-Plane Crossplane for Client Cluster Provisioning.