Full graph →

Multi-Cluster Metrics

Several small clusters, one place to look at all of them, retention beyond any single VM's disk, and every cluster still pages when its link to the center is cut. Those four requirements pull in different directions, and the design that satisfies all of them is a full Prometheus at every edge with short retention, remote_write into a central Prometheus that carries a Thanos sidecar to object storage, and alert evaluation on both ends with rule sets that do not overlap.

Encountered at SFI: sfi/deployments Phase 8 (umbrella #68; #53 edge Prometheus, #98 luxor hub, #99 per-cluster substitution, #100 retiring the compose stack). Designed 2026-09-02 to replace the soundship compose Prometheus that scraped every cluster over the WireGuard mesh. Fleet: three lab k3s VMs plus a production VPS, all single-node, all mesh peers.

The problem shape

The predecessor was a single central Prometheus with hand-written static_configs: node_exporter on 9100, the kubelet's read-only cadvisor port, and kube-state-metrics on a NodePort, for every host, over the mesh. It worked, and it is the right first step for a handful of hosts. It stops being right once clusters run workloads that expose their own metrics.

Every new target meant a firewall opening on the host and a hand-maintained entry in the center. Anything living inside a cluster (the postgres operator, the ingress controller, the GitOps controllers, the applications) needed a NodePort to be reachable at all, or simply went unscraped. And when the mesh partitioned, the center saw nothing while the edge had no alerting of its own. The production cluster in particular sits on the far side of a WireGuard tunnel from the lab; "the tunnel is down" is exactly the condition it must be able to report.

The requirements, stated as constraints:

  • Applications declare their own scrape targets and alert rules next to their deployment, per cluster, in git.
  • One query surface for dashboards across clusters.
  • Retention measured in months, without growing every VM's disk.
  • Each cluster pages independently of the center and of the mesh.

Pull inside the cluster, push between clusters

Prometheus is a pull system, and inside a cluster that is still the right model: the operator turns ServiceMonitor and PodMonitor resources into scrape configs, so a chart that ships one gets scraped the moment it is deployed. Between clusters the direction inverts. Each edge Prometheus pushes its samples to the center with remote_write. The edge initiates the connection over the mesh, the center exposes one endpoint instead of the fleet exposing dozens, and the cluster external label travels with every series so nothing collides at the hub.

The general trade-off between the two collection models is in metrics. Here the choice is dictated by reachability: the center can reach lab VMs, but making it reach into every cluster's pod network is the NodePort sprawl the design is escaping.

A full Prometheus at the edge, not Agent mode

Prometheus Agent mode is the obvious candidate for an edge that only forwards, and it is the wrong choice whenever the edge must alert. Agent mode drops the local TSDB, and with it querying and rule evaluation. It cannot page.

A full Prometheus with short retention costs nearly the same. Memory is dominated by the write-ahead log and the head block in both modes; the difference is a few hours of compacted blocks on disk. Retention of a day, capped by size, keeps the edge small while leaving enough history to debug a page locally when the center is unreachable.

The observability budget on the smallest production shape (2 vCPU, 4 GB) is roughly a gigabyte: operator around 100 MiB, Prometheus 300 to 600 MiB for a small cluster, Alertmanager around 30 MiB, the log agent around 50 MiB. That is why the edge stack lands on the rehearsal cluster before the sizing decision is recorded (sfi/deployments#92).

Alerting on both ends, with disjoint rules

Prometheus sends every firing alert to every Alertmanager it is configured with. There is no failover semantics: two Alertmanagers means two notifications. The rule that follows is that each Prometheus points at exactly one Alertmanager, and the edge and hub evaluate different rules.

The edge owns "this cluster is unhealthy": node pressure, disk filling, the postgres cluster down, certificates expiring, pods crash-looping, backups failing. Its Alertmanager posts to the chat webhook over the public internet, a different network path from the mesh. That is the point: a mesh outage takes the center offline and the edge still pages.

The hub owns what only the hub can see: cross-cluster rules and the dead man's switch. A cluster that stops writing disappears from the hub after the staleness window, so absent(up{cluster="production"}) fires on the center when production goes dark. The edge cannot raise that alert about itself.

# Hub-side PrometheusRule (evaluated by Thanos Ruler against Thanos Query).
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: fleet
  namespace: monitoring
spec:
  groups:
    - name: fleet
      rules:
        - alert: ClusterStoppedReporting
          expr: absent(up{cluster="production"}) == 1
          for: 10m # remote_write backlog after a short mesh blip is normal
          labels:
            severity: critical
          annotations:
            summary: "production has not remote-written for 10m"

The GitOps notification controller is a third, independent path: a HelmRelease that fails to reconcile notifies the same channel with no dependency on Prometheus at all. See flux.

Thanos: sidecar on the center, not Receive

Thanos has two ways to get samples into object storage. A sidecar runs next to a Prometheus, serves its data over the StoreAPI, and uploads each finished 2-hour block to the bucket. Receive is an ingestion service that accepts remote_write directly, writes its own TSDB, and uploads from there; it is the textbook match for a push topology.

For this fleet the sidecar wins, for one reason: the Prometheus Operator manages it and does not manage Receive. Setting spec.thanos on a Prometheus resource injects the sidecar, pins block durations to 2 hours, and disables local compaction so the uploader sees stable blocks. ThanosRuler is a CRD too. Receive needs hand-rolled hashring configuration and a StatefulSet nobody reconciles. Prometheus itself accepts remote_write natively (enableRemoteWriteReceiver), so the central Prometheus ingests from the edges and the sidecar uploads what it ingested.

edge Prometheus ──remote_write──▶ central Prometheus ──sidecar──▶ S3 bucket
                                        ▲                          │
                                  Thanos Query ◀── Store Gateway ◀─┘
                                        ▲          Compactor (downsample, retain)
                                  Grafana, Thanos Ruler

Around the sidecar: Query fans out over the sidecar (recent data) and the Store Gateway (blocks in the bucket); the Compactor merges blocks, produces 5-minute and 1-hour downsampled copies, and enforces retention per resolution; the Ruler evaluates the fleet-wide rules against Query. All of it ships in one community chart, oci://ghcr.io/thanos-community/helm-charts/thanos, with each component toggled in values. The bitnami chart is not an option since its images moved to a legacy registry in 2025.

The trade-off is a single ingestion point with no high availability. On a fleet of single-node clusters that is consistent with everything else; Receive with a hashring is the upgrade path if it ever stops being acceptable.

Two Thanos rules that bite: every Prometheus feeding the bucket needs a unique external label set, and the Compactor is the only component allowed to write compacted blocks, which is why the sidecar-bearing Prometheus runs with local compaction off. As built, the uniqueness requirement is satisfied for free: the operator injects prometheus: <ns>/<name> and prometheus_replica: <pod> as external labels, and those names are reserved — a hand-set value under them is silently ignored (a custom hub= label was the design here; the operator's pair replaced it). Details in thanos and prometheus.

The manifests

Edge, in the shared infrastructure layer applied to every cluster:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: edge
  namespace: monitoring
spec:
  replicas: 1
  retention: 24h
  retentionSize: 4GB
  externalLabels:
    cluster: ${CLUSTER_NAME} # substituted per cluster by flux postBuild
  remoteWrite:
    - url: http://10.1.0.5:30990/api/v1/write # hub, mesh address only
      queueConfig:
        maxShards: 5 # single-node edges; keep the sender small
  alerting:
    alertmanagers:
      - name: alertmanager-edge # exactly one; see "disjoint rules"
        namespace: monitoring
        port: web
  serviceMonitorSelector: {} # every ServiceMonitor in the cluster
  podMonitorSelector: {}
  ruleSelector: # hub rules live in ThanosRuler, never here — exclusion, so
    matchExpressions: # apps can ship rules without opting in to a label
      - key: monitoring.sfi/scope
        operator: NotIn
        values: [central]
  resources:
    requests: { memory: 512Mi }
    limits: { memory: 1Gi }

Hub, in the platform cluster's app directory:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: hub
  namespace: monitoring
spec:
  replicas: 1
  enableRemoteWriteReceiver: true # edges write here
  retention: 7d # local; long-term lives in the bucket
  externalLabels:
    hub: luxor # unique per uploading Prometheus; Thanos requires it
  thanos:
    objectStorageConfig:
      name: thanos-objstore # sops-encrypted; type S3, endpoint = rustfs
      key: objstore.yml
  scrapeConfigSelector: # the lab hosts that are not clusters, as ScrapeConfig CRs
    matchLabels:
      scope: hub

The design had the static lab hosts in additionalScrapeConfigs (an opaque Secret) and the hub selecting its own ServiceMonitors. As built, the hub selects nothing but ScrapeConfig CRs: in-cluster scraping belongs to the edge instance even on the hub's own cluster, and a Prometheus whose selectors match zero resources gets an empty generated config — the operator drops additionalScrapeConfigs and externalLabels with it and the sidecar refuses to start. Typed ScrapeConfig CRs are both the fix and the better home (schema-validated, one CR per job). The full incident shape is in prometheus.

The cluster value comes from a per-cluster ConfigMap substituted at reconcile time; the shared layer is otherwise identical on every cluster. That mechanism is flux postBuild.substituteFrom, added to the fleet for exactly this need.

Object storage caveats

The bucket is only as durable as the disk under it. A rustfs or MinIO instance on the same VM as the central Prometheus gives you Thanos's query and downsampling features, not disaster recovery, until the bucket replicates to an offsite provider.

The Compactor is the component most sensitive to S3 semantics: list-after-write consistency, multipart uploads, and object overwrite behaviour. A beta S3 implementation needs thanos tools bucket verify and a full compaction cycle observed clean before long-term retention is trusted. A halted Compactor logs "overlapping blocks" or upload failures and stops compacting silently from the dashboard's point of view; alert on thanos_compact_halted.

Retention is set in two places and they mean different things. The hub Prometheus's retention is how long it keeps blocks locally after upload. The Compactor's --retention.resolution-raw, -5m, and -1h flags decide how long each resolution survives in the bucket. Downsampling is what makes the short raw retention safe: once the 5-minute and 1-hour series exist, a year-over-year dashboard no longer needs the raw samples, which is the general compaction pattern in rollups-as-query-output. Choose the raw horizon by asking which queries must still be answerable from the coarse data, not by disk size alone.

Things that go wrong

Duplicate pages. An edge Prometheus configured with both its local Alertmanager and the hub's. Fan-out, not failover. One Alertmanager per Prometheus.

Silent edge. Agent mode deployed "to save memory": rules load without error and never evaluate. Check prometheus_rule_group_last_evaluation_timestamp_seconds exists at all.

Colliding series. No cluster external label, so up{job="kube-state-metrics",instance="..."} from two clusters interleave at the hub. Every edge needs a distinct external label before it writes a single sample.

Lost samples after a long outage. remote_write retries from the write-ahead log, which Prometheus keeps for about two hours of segments. A mesh outage longer than that loses the gap at the hub. The edge still has the data locally for its retention window, which is one more reason the edge is a full Prometheus.

Leftover pull-era openings. The NodePorts and read-only kubelet port that fed the old central scraper stay open after the migration unless they are closed on purpose. Treat their removal as part of the cutover.

External label conflict. Two Prometheus instances uploading blocks with identical external labels make the Store Gateway refuse both. The hub's own label is not optional.

Sizing

Measured figures belong here once the edge stack has run on the rehearsal cluster (sfi/deployments#92). TODO: memory high-water for operator, Prometheus, Alertmanager, and the log agent on the 4 GB shape; hub totals on the platform cluster after Thanos, Grafana, and Loki land.

References