Full graph →

Release Engineering

A release is a claim: this set of artifacts works together. Every release scheme is an answer to two questions — what does a version name (which exact code, across every component), and what does a version promise (what a consumer may assume when the number changes). Get those two right and cadence mostly takes care of itself; get them wrong and you end up maintaining compatibility matrices by hand or reading release notes to find out whether an upgrade is safe.

Provenance: symbology's first chart-driven release (v0.1.0, symbology PR #170) — one repo tag stamping a UI image, a worker image, and a Helm chart in lockstep, consumed by the fleet repo via a pinned OCI tag (deployments PR #79). The install failure that release shipped (PR #174) is the build-metadata example below. The manifest-of-SHAs model is the contrast case from larger multi-repo organizations.

What a version names: manifests vs tags

In a multi-repo system, "release 1.0.0" cannot point at code directly — the code lives in several repositories with independent histories. So the release becomes a document: a manifest asserting that repo A at abcd123 and repo B at efgh567 were built and tested together. Someone (a release team, a pipeline, a bot) assembles it, and the manifest itself needs versioning, review, and tooling. This is a real pattern at scale — OpenStack coordinates dozens of projects through a dedicated releases repository, and Android assembles hundreds of git trees from repo manifests — but the manifest exists to solve a coordination problem: proving that things developed apart can ship together.

A monorepo ships the manifest for free: the commit is the manifest. One SHA pins every component to a state that provably coexisted — same tree, same CI run. Tagging that commit v0.1.0 and stamping the tag into every artifact gives you the multi-repo manifest's guarantee with none of its machinery. The mechanics — tag as the single source of truth, stamping at package time, republishing unchanged components under the new number — are covered in lockstep-artifact-versioning; the paradigm-level point is that the two models answer the same question ("what exactly is in 1.0.0?") and the monorepo answer is structurally simpler. Choose the manifest model when components genuinely have independent consumers and lifecycles; choose lockstep when one product ships as several artifacts.

The corollary of lockstep is that the version bump is the maximum across components: a worker bugfix alone produces a patch release that also republishes an untouched UI image. That feels wasteful the first few times and isn't — registries dedupe layers, and the payoff is that one version string answers "what is running" with no cross-referencing.

What a version promises: SemVer for applications

Semantic Versioning is specified for APIs with external consumers: major for incompatible changes, minor for added functionality, patch for fixes. An application nobody imports has no API in that sense, so "breaking change" is undefined until you define it. The useful move is to pick the consumer the contract serves: for a deployed application, that consumer is the operator performing the rollout — concretely, the GitOps repo holding the version pin, and the person bumping it.

Against that consumer, the tiers acquire operational meanings:

Bump Promise to the operator
Patch Roll forward blindly, roll back freely. No config changes, no destructive migrations.
Minor New functionality; the rollout is still just "bump the pin". New settings are optional with working defaults.
Major Bumping the pin is not enough — required new configuration, renamed or removed settings, a migration you cannot roll back through, a manual step. Read the release notes first.

Under this contract the application's de facto public API is its deployment interface — for a Helm-delivered service, the chart's values schema. Renaming a value breaks every environment consuming the chart exactly the way renaming a function breaks a library's callers, and it is the clearest major-bump trigger in a lockstep setup. (Projects that version charts independently, like kube-prometheus-stack, bump the chart's major on values changes for the same reason — lockstep just merges that signal into the product version.)

Two boundary notes. Pre-1.0, SemVer officially promises nothing; the working convention is that 0.MINOR carries the breaking changes and 0.y.PATCH carries everything else, and 1.0.0 is a deliberate declaration (real users, production traffic) rather than a milestone you drift into. And build metadata is not free decoration: tooling appends it behind your back. Flux's helm-controller installs charts pulled from an OCI source with the artifact digest appended as SemVer build metadata (0.1.0+4b5e34c61bf7, to detect re-pushed tags) — legal SemVer, illegal as a Kubernetes label value. Symbology's first install failed exactly there, because the chart stamped the raw version into its helm.sh/chart label; the fix (PR #174) is the sanitization helm create scaffolds by default: replace "+" "_". Any code that turns a version into an identifier in another system — labels, filenames, OCI tags (which also forbid +) — needs to assume metadata will appear.

The compatibility surface the tag doesn't cover: the database

Lockstep guarantees that the artifacts in a release coexisted. It says nothing about the one stateful component that persists across releases: the database schema, versioned by the migration tool rather than the tag. Every upgrade opens a mixed-version window — with a pre-upgrade migration hook, old-version pods run against new-version schema until the rollout completes, and after a rollback, old code runs against new schema indefinitely.

The discipline that keeps that window safe is expand/contract (Fowler's parallel change): release N ships only additive migrations — new columns nullable or defaulted, new tables, writes to both old and new locations if needed — and the destructive contraction (drops, renames, constraint tightening) ships in a later release, after no running code depends on the old shape. Structured this way, every patch and minor release is rollback-safe by construction, which is precisely what the SemVer contract above promised. A schema change that cannot be split into expand and contract phases is the definition of a major release — and often of a maintenance window. The same mixed-version reasoning applied to wire formats — tolerant and strict readers, what an identifier pins forever — is schema-evolution.

Cadence and promotion

Tag small and often. The argument is failure bisection: a release's suspect surface is everything it contains. Symbology's v0.1.0 bundled the entire chart scaffold, so the install failure had a whole release of candidate causes; had the chart landed across three tags, the label bug would have been isolated by version arithmetic. Frequent small releases also keep each bump decision easy — one change, one obvious tier.

A tag is a promotion gate. In a release-based flow, cutting vX.Y.Z is the decision "this is deployable," and environments consume only blessed artifacts by pinning exact versions. The alternative is continuous deployment of the main branch: CI publishes an artifact per commit (versioned as a SemVer prerelease — 0.1.0-b7c9ab8, since chart versions must be SemVer and bare SHAs are not), and an environment tracks the stream instead of a pin. The common hybrid gives each environment the flavor it wants: staging tracks, production pins — staging follows a prerelease-inclusive range for fast feedback, production moves only when a human (or a reviewed bot PR) bumps an exact version. Switching a pinned environment to tracking is not a tag-format change; it is a different contract — you are deleting the promotion gate, and dependency automation like renovate loses its footing, because SHA-suffixed prereleases don't order.

Deciding the bump tier can be automated — conventional commits plus a release-please-style tool compute the next version from history — but that machinery earns its keep with many contributors. With one committer and a tag script, judgment against the operator contract above is cheaper and just as accurate.

References