Skip to content

ADR-012: Management-Plane Crossplane for Client Control Plane Provisioning

Status: Approved

Date: 2026-06-30

Decision

The 7KGroup management plane uses Crossplane to provision and manage the lifecycle of each client's control-plane infrastructure, including the AWS account boundary, VPC, EKS cluster, IAM, node groups, and the initial ArgoCD installation. The client cluster then runs its own ArgoCD and Crossplane instances for workload infrastructure.

Rationale

  • Solves the chicken-and-egg problem: the client cluster cannot provision itself
  • Centralizes control-plane lifecycle management in the management plane
  • Allows 7KGroup to use Crossplane for the entire provisioning pipeline while still giving clients self-service infrastructure via Hiroba claims
  • Avoids the need for Terraform as a separate provisioning tool in v1
  • Enables ongoing reconciliation and upgrades of cluster infrastructure

Architecture

Management Plane
├── Crossplane
│   ├── provider-aws (or provider-gcp, provider-azure)
│   ├── ProviderConfig (limited infra-provisioning credential)
│   └── Composition / Claim
│       └── AWS account / org resources (target state / scripted for v1)
│       └── VPC, subnets, route tables, NAT gateways, security groups
│       └── EKS control plane
│       └── Node groups
│       └── Cluster-level IAM roles and OIDC provider
│       └── ArgoCD Helm release
│       └── Initial ArgoCD Application pointing to client overlay
│
└── Vault, Mimir, Loki, Grafana, Keycloak

Client AWS Account
└── EKS Cluster (provisioned and reconciled by mgmt-plane Crossplane)
    ├── ArgoCD (installed and seeded by mgmt-plane Crossplane)
    ├── Crossplane (for workload infrastructure)
    ├── Backstage, Dex, Alloy, Hiroba Catalog
    └── Client workloads

Management-Plane Crossplane Scope

Managed and reconciled by management-plane Crossplane:

  • AWS account / organization resources (manual or scripted for v1)
  • VPC, subnets, routing, security groups, NAT gateways
  • EKS control plane
  • Node groups (including upgrades)
  • Cluster-level IAM roles, instance profiles, OIDC provider
  • ArgoCD Helm installation
  • Initial ArgoCD Application pointing to clients/<client-id>/overlays/<env>/

Not managed by management-plane Crossplane:

  • ArgoCD configuration changes after seeding
  • ArgoCD version upgrades (controlled by the client overlay)
  • Backstage, Dex, Alloy, workload Crossplane, Hiroba XRDs, OPA policies
  • Workload-level IAM roles (e.g., IRSA for Crossplane providers)
  • Client workloads

Credential Boundary

The management plane holds a dedicated, limited-scope cloud credential for provisioning and managing the client control plane. This credential:

  • Lives in the management-plane secret store
  • Is used by a management-plane Crossplane ProviderConfig
  • Has permissions only for cluster infrastructure (EKS, IAM, VPC, EC2, etc.)
  • Does not have access to client workload data or application secrets
  • Is distinct from client-scoped credentials used inside the client cluster

Once the client cluster exists, application infrastructure (databases, caches, buckets) is provisioned by the client-local Crossplane using client-scoped credentials pulled by External Secrets Operator from the client's secret store.

Bootstrap Handoff

  1. Management-plane Crossplane provisions the client control-plane infrastructure and installs ArgoCD.
  2. Management-plane Crossplane creates the initial ArgoCD Application pointing to the client overlay.
  3. 7KGroup operator or automation authenticates to the new cluster with admin credentials.
  4. External Secrets Operator is installed.
  5. An initial bootstrap secret is created in the client cluster, allowing ESO to connect to the client's secret store.
  6. ESO fetches client-scoped credentials.
  7. Client-local ArgoCD syncs the overlay, deploying Backstage, Crossplane (workload), Dex, Alloy, Hiroba Catalog, etc.
  8. Management-plane Crossplane retains ongoing reconciliation of the infrastructure layer.

Reconciliation and Upgrades

  • If cluster infrastructure drifts from the desired state, management-plane Crossplane reconciles it.
  • EKS version and node group upgrades are driven by changes to the management-plane Crossplane claim.
  • If ArgoCD is deleted, management-plane Crossplane recreates the installation and re-seeds the initial Application.
  • Platform component upgrades (Backstage, Dex, Alloy, etc.) are driven by changes to the 7k-inari overlay and applied by client-local ArgoCD.

Trade-offs

Pros Cons
Pure Crossplane-based provisioning Management plane holds a powerful cloud credential
Central cluster lifecycle control Management plane becomes a critical dependency for cluster provisioning
No separate Terraform pipeline in v1 Requires careful credential scoping and rotation
Ongoing reconciliation prevents drift Blurred boundary if infra scope is not strictly enforced

Security Controls

  • Management-plane Crossplane ProviderConfig uses a dedicated IAM role/user per client cloud account
  • Credential permissions are minimal (only infrastructure provisioning)
  • Credentials are rotated quarterly or on compromise
  • All management-plane Crossplane actions are auditable via cloud provider logs
  • Workload-level IAM is owned by the client overlay, not the management plane

Relationship to Client-Local Crossplane

Layer Runs In Purpose Owned By
Cluster infrastructure Management-plane Crossplane VPC, EKS, IAM, node groups, ArgoCD install Management plane
ArgoCD Application contents Client-local ArgoCD Deploy platform baseline and overlay Client overlay / Git
Workload infrastructure Client-local Crossplane RDS, S3, Redis, etc. from Hiroba claims Client overlay / Git