6. Observability¶
Inari uses Grafana Alloy as a unified observability agent in each client cluster, pushing metrics and logs to the management-plane Mimir and Loki stacks.
6.1 Pipeline¶
Client Cluster Alloy
├── prometheus.scrape (K8s targets)
├── loki.source.kubernetes (pod logs)
├── loki.source.kubernetes_events (K8s events)
├── otelcol.receiver.otlp (traces — optional)
└── Remote Write → Management Plane
├── Metrics → Mimir (header: X-Scope-OrgID=<client-id>)
├── Logs → Loki (header: X-Scope-OrgID=<client-id>)
└── Traces → Tempo (header: X-Scope-OrgID=<client-id>)
6.2 Authentication¶
Client Alloy agents authenticate to the management plane using mTLS.
X-Scope-OrgIDheader ensures tenant isolation in Mimir and Loki.- mTLS certificates are client-scoped and rotated on a short-lived basis.
- Bearer tokens are acceptable only for pre-GA pilots with explicit exception.
6.3 Data Collected from Client Clusters¶
Metrics¶
- Kubernetes node metrics
- Container metrics
- Custom application metrics (via ServiceMonitor/PodMonitor)
- Crossplane resource status metrics
Logs¶
- Pod logs
- Kubernetes events
- Dex authentication logs
- Crossplane provider logs
Traces (Optional)¶
Alloy can receive OTLP traces from client workloads and push them to Tempo. This is a future enhancement.
6.4 Fleet Dashboards¶
Grafana in the management plane provides:
- Cluster health (per client)
- Resource utilization trends
- Crossplane claim reconciliation rates
- Cloud spend estimates
6.5 SLOs¶
| SLO | Target | Measurement |
|---|---|---|
| Management plane availability | 99.9% monthly | Uptime of Mimir, Loki, Grafana, Vault, and Keycloak endpoints |
| Client control-plane availability | 99.5% monthly | Backstage, Dex, workload Crossplane, and client-local ArgoCD uptime |
| ArgoCD sync latency (control-plane) | < 5 min p95 | Time from Git commit to sync completion for client overlays |
| Observability latency | < 60 sec p95 | Time from event/ metric emission to visibility in Grafana |
| ESO secret sync latency | < 2 min p95 | Time for External Secrets Operator to reflect a changed external secret |
These SLOs cover platform operations, not client workloads. Client workload availability is owned by the client.
6.6 Data Retention (Proposed)¶
| Data Type | Retention | Notes |
|---|---|---|
| Metrics | 30 days default, 1 year downsampled | Long-term retention per client contract |
| Logs | 30 days default | Security/audit logs may be retained longer |
| Traces | 7 days | Optional; retention depends on adoption |