ADR-006: Grafana Alloy for Observability Pipeline

Status: Approved

Date: 2026-06-30

Decision

Deploy Grafana Alloy as a DaemonSet in each client cluster to collect metrics, logs, and (optionally) traces, pushing to 7KGroup's management plane.

Rationale

• Single agent replaces Prometheus + Promtail + OTel Collector
• Native River configuration language unifies metrics, logs, and traces
• Built-in remote_write and log push capabilities
• Lower resource footprint per node

Pipeline

Client Cluster Alloy
├── prometheus.scrape (K8s targets)
├── loki.source.kubernetes (pod logs)
├── loki.source.kubernetes_events (K8s events)
├── otelcol.receiver.otlp (traces — optional)
└── Remote Write → Management Plane
    ├── Metrics → Mimir (header: X-Scope-OrgID=<client>)
    ├── Logs → Loki (header: X-Scope-OrgID=<client>)
    └── Traces → Tempo (header: X-Scope-OrgID=<client>)

Authentication

• Client Alloy agents authenticate to the management plane using mTLS
• Per-client certificates are distributed and rotated via cert-manager or Vault PKI
• X-Scope-OrgID header ensures tenant isolation in Mimir/Loki
• Bearer tokens are acceptable only for pre-GA pilots with explicit exception

Related

• Observability
• Secret Management
• Pilot Scope