Full graph →

Labels

Modern observability systems share one data model: a record whose identity and context live in a set of key/value labels, not in a schema, a table, or a hierarchy. Prometheus stores labeled numeric samples. Loki stores labeled log streams. Traces are labeled intervals. Kubernetes selects pods by label match. The payloads differ; the model — and the query primitive, label matching — is the same. Recognizing this makes each new labeled system familiar on contact, and it opens up uses that don't look like "monitoring" at first glance.

Encountered at SFI while building the TraggoMenuApp menu-bar client: Traggo stores time tracking as tagged timespans, and the resemblance to Prometheus's data model turned out to be more than cosmetic.

One Model, Four Payloads

System Record shape Query primitive
Prometheus (metrics) {labels} → (timestamp, value) label matchers: rate(http_requests_total{status=~"5.."}[5m])
Loki (logging) {labels} → (timestamp, log line) stream selectors: {app="checkout"} |= "error"
Tracing {attributes} → (start, end, parent span) attribute filters over spans
Traggo {tags} → (start, end) tag filters over timespans

Read down the third column: each row is a labeled point, a labeled event, a labeled interval with causality, and a labeled interval without causality. A Traggo timespan is essentially a trace span minus the parent pointer — context comes entirely from its tags, exactly as a Prometheus series' meaning comes entirely from its label set.

The same primitive underpins kubernetes: a Service routes to pods whose labels satisfy its selector, a Deployment owns pods via matchLabels. There is no registry mapping services to pods — membership is the label match, evaluated continuously. Loose coupling through label matching is what lets these systems compose without central coordination.

Why Labels Won

The alternative to labels is hierarchy: prod.us-east.checkout.http.errors as a dotted metric name (the Graphite model), or logs routed by file path. Hierarchies force one fixed drill-down order and make cross-cutting questions ("all errors in us-east, any service") awkward. Labels are dimensions — every key is a first-class axis you can filter or aggregate on, in any order. The query languages differ in syntax, but sum by (service) and "group timespans by tag" are the same operation.

The universal cost is cardinality. Every distinct label combination is a distinct series, stream, or group; a user_id label turns one metric into a series per user. The discipline from metrics applies to every labeled system: labels carry dimensions you'll aggregate over, not identifiers. (Traggo is the lenient case — timespans are stored individually, so a unique tag value costs one row, not a new series. High-cardinality tags there hurt only readability.)

A Deliberate Scheme Across One Estate

The model rewards a decision most stacks never make explicitly: choose the identity label set once, and enforce it across every signal. In the SFI fleet (Phase 8, sfi/deployments#53/#54) that set is cluster, host, namespace, pod, container — stamped onto metrics by the edge Prometheus's external labels and onto log streams by the shipper's label line, with the same key names and the same values in both systems. cluster="production-test" means one thing everywhere, which is what makes pivoting between a metric spike and its logs a matter of copying a selector rather than translating one.

Three rules that came out of doing this for real:

  • Join on identity, never on scrape artifacts. instance means "whatever address this exporter was scraped at" — kube-state-metrics series carry the KSM pod's address, cAdvisor series carry the node's, so on(namespace, pod, instance) joins between them can never match. The predecessor system relabeled instance to the cluster name on both sides, so the dashboards worked by accident until the convention changed under them. Pod identity fleet-wide is on(cluster, namespace, pod); a label that encodes where data came from is not a label that identifies what it describes.
  • Know which labels the platform owns. The Prometheus operator reserves prometheus and prometheus_replica (your values are ignored); Kubernetes stamps kubernetes.io/metadata.name on every namespace (the standard way to name-match with a label selector). Fighting injected labels loses silently; leaning on them is free.
  • Uniformity is configuration, not hope. Every producer needs its defaults bent to the shared contract — the log shipper's label set is a hand-written line, the polyglot loggers need their level/message-key conventions aligned (see logging), and a host that predates the scheme gets the missing label stamped at the scrape config so queries never special-case it.

The estate-level topology these labels serve is multi-cluster-metrics; the cardinality economics are the same as everywhere else in this page.

Labeled Intervals as an Observability Signal

The interesting gap in the standard toolkit is the discrete episode: a batch job run, a deployment rollout stage, a data migration, an agentic workflow step. These are things with a start, an end, and context — and neither of the two workhorse signals represents them naturally:

  • Metrics flatten episodes into aggregates. You can emit a duration histogram or a last_completed gauge, but the individual run — this Tuesday's backup, with these attributes — is gone, and Prometheus's model actively resists long-lived per-run series.
  • Traces represent episodes perfectly, but the machinery (instrumentation, propagation, sampling, a trace backend) is sized for high-volume request paths, not for tens of events a day you want to keep forever.

A tagged-timespan store like Traggo sits precisely in that gap: durable labeled intervals, queryable and aggregatable by tag, cheap to write. Its GraphQL API (service-interactions) makes it scriptable — wrap a job in createTimeSpan / stopTimeSpan with tags like job=db-backup host=zen, and duration trends, overlap analysis, and "what ran during the incident window?" come free from the same queries the UI uses. Candidate uses beyond human time tracking:

  • batch and cron job runs (duration drift is a leading indicator of trouble)
  • deployment rollouts, with a timespan per stage
  • agentic workflow steps — which phase of a multi-step LLM pipeline is slow, across runs
  • manual toil, making "time spent on X" reportable alongside system metrics

Stated honestly: this is an analytical niche, not a monitoring platform. Traggo has no alerting, no retention policy, and a single-user permission model; it complements metrics rather than replacing them (you still want an alert when the backup doesn't run). The insight worth keeping is smaller and sharper — when data is labeled, anything that stores labeled records can serve as an observability backend for the payload shape it holds, and labeled intervals are the shape the mainstream tools underserve.

References