Full graph →

The SFI Platform

Every system in this section runs on infrastructure Streetfortress operates itself, and that platform is the longest-running project of the lot. It has moved through three distinct eras — Ansible-managed Ubuntu hosts, Compose stacks behind Caddy, and now NixOS hosts running Flux-driven GitOps on k3s — and each transition was forced by a specific failure of the previous era, not by fashion. This document is the case study of that evolution; the technique documents it links to are the reusable pieces.

SFI project: ongoing. The current migration lives in the deployments monorepo; snippets are lifted from fleet/clusters/luxor/ and .sops.yaml at current HEAD.

Three eras, three lessons

Era one: Ansible over Ubuntu. Playbooks provisioned VMs, a WireGuard mesh connected them (network-isolation), and configuration lived in inventory files. It worked until it drifted: playbooks describe how to change a host, not what a host is, so every manually-patched package and hand-edited config accumulated as unrecorded state. Rebuilding a host meant archaeology.

Era two: Compose + Caddy. Application deployment collapsed to a pattern still documented in simple-container-deployment: a justfile, a compose.yaml, sops-encrypted .env files, Caddy for TLS. This fixed the application story — deployments became declarative and portable — but the hosts underneath were still mutable snowflakes, and every service addition meant SSH.

Era three: NixOS + k3s + Flux. The current migration makes the host itself declarative. A machine is a flake output; its secrets identity is seeded at provisioning; everything above the OS reconciles from git. The goal is stated as a test: a host can be rebuilt from its flake and its workloads restored from git and backups, hands-off. Host-level reasoning is in host-config; this document covers how the pieces compose.

Git as the only path to production

Flux itself is installed declaratively — the flux-operator reads a FluxInstance resource rather than accumulating CLI-generated bootstrap commits:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
  name: flux
  namespace: flux-system
spec:
  distribution:
    version: 2.x
    registry: ghcr.io/fluxcd
  components:
    - source-controller
    - kustomize-controller
    - helm-controller
    - notification-controller
  sync:
    kind: GitRepository
    url: ssh://git@gitea.zen.lofi:30022/sfi/deployments.git
    ref: refs/heads/main
    path: fleet/clusters/luxor   # each cluster reconciles only its own path

The path field is the multi-cluster strategy. One monorepo holds every cluster's desired state; each cluster's Flux reconciles only fleet/clusters/<name>, which composes shared infrastructure with cluster-specific apps:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  dependsOn:
    - name: infrastructure    # operators reconcile before workloads
  path: ./fleet/apps/luxor
  prune: true                 # deleting from git deletes from the cluster
  decryption:
    provider: sops
    secretRef:
      name: sops-age

The monorepo trade is worth naming: one repo means atomic cross-layer changes, one .sops.yaml, one Renovate configuration — and it costs nothing at runtime because path-scoping keeps each cluster's blast radius to its own directory. Flux mechanics are covered in flux; the supply-chain reasoning for the self-hosted git and OCI registry is in self-hosted-artifacts and supply-chain.

Secrets: scoped by construction

Secrets are sops-encrypted in git, and the encryption rules enforce isolation structurally — a cluster's key can only decrypt its own secrets:

# .sops.yaml — path-scoped per cluster
creation_rules:
  - path_regex: fleet/(clusters|apps)/luxor/.*
    encrypted_regex: ^(data|stringData)$
    age: >-
      age1rgr7...,   # admin key
      age142ql...    # luxor's cluster key — private half lives only on luxor
  # fallback: admin-only, whole file
  - path_regex: .*
    age: age1rgr7...

Each host's age identity is seeded at provisioning time (the pattern in pre-seeded-identity), which is what makes bootstrap hands-off: the freshly-built host can already decrypt its own secrets, so Flux's first reconcile succeeds with no one pasting keys. A compromised staging cluster cannot read production's secrets because it never possessed a key that could. Application-level handling follows sops-application-secrets.

Restore drills as phase gates

The migration is sequenced in phases, and the discipline worth copying is the gate between them: nothing is adopted into GitOps until its backups have been restored, actually, from the latest backup. The greenfield cluster (luxor) rebuilds the existing stack in parallel while the original host stays frozen; only after a successful restore drill of the git service itself does the original get adopted. Postgres runs under CloudNativePG with layered physical and logical backups — the strategy and its reasoning are in backup-strategies.

This ordering exists because the failure mode it prevents is the expensive kind: migrating the source of truth onto a platform whose recovery path has never been exercised. A backup that has not been restored is a hypothesis.

Honest state

This is a migration in progress, not a finished platform. The git server is a single point of failure until its adoption phase completes (mitigated by an external mirror and offsite backups); observability of the fleet is a later phase; and the original host still runs the era-two Compose stack while parity is proven. The register lists this unit as Active Migration because that is what it is. The direction, though, is the point: every phase narrows the gap between "the infrastructure" and "what git says the infrastructure is."

References