Full graph →

Schema Evolution

A serialized contract outlives every build that speaks it. The moment two versions of your software can read the same stored or transmitted record — an old phone and a new laptop, a lagging service and an upgraded one — you have a schema evolution problem, and the decisions that make it survivable are made before version one ships, not when version two needs a change. This note records those decisions as they apply across wire formats, using Protocol Buffers (the format that solved this most deliberately) and CloudKit records (where we made these calls for real) as the two anchors.

Provenance: the strict-reader and identifier decisions were made in Moment Tally's CloudKit record codec (moment-tally PR #234, issues #121/#159). The protobuf side is grounded in the learning-grpc exercise — Go, Rust, and TypeScript programs sharing one .proto definition — plus upstream rules we have not yet had to survive in production.

What actually travels: the identifier decides what can change

Every format pins its fields to something, and that something determines what is renameable forever after.

Protobuf pins fields to numbers. A .proto field is string note = 4; — the 4 is what travels in the binary encoding; the name note is codegen convenience. You can rename note to comment freely across every language binding, because no serialized byte contains the name. The corresponding law is absolute: never reuse or renumber a field. A number that once meant "note" and later means "duration" corrupts old data silently — which is why deleted fields get reserved 4; tombstones in the schema.

Keyed formats pin fields to names. A CKRecord (like JSON, like most document stores) is a keyed bag: the string "modifiedAt" is the wire identifier. Renaming a field is a breaking change, permanently. The practical consequence is that key literals are as load-bearing as proto field numbers and deserve the same treatment: centralized in one codec file, never scattered as inline strings, never "cleaned up" in a refactor.

Registry identifiers are the extreme case. Some identifiers escape your codebase entirely and live in someone else's system: CloudKit container IDs can never be deleted from a developer account; promoted production record types and fields are additive-only forever; a published protobuf package name is baked into every consumer's codegen. Treat naming these as a one-way door and spell-check accordingly.

The tolerant reader

Protobuf's default philosophy: decode what you recognize, skip what you don't, keep going. Unknown fields aren't errors — they're preserved through a read-modify-write cycle so an old service doesn't strip data written by a new one. Evolution stays compatible as long as changes are additive: new fields are optional with sensible defaults, old fields keep their numbers, and readers never assume presence.

This model carries nearly all schema evolution everywhere, not just in protobuf. The CloudKit codec gets the same behavior from two habits: the decoder reads only the keys it knows (unknown fields are invisible, not fatal), and encoding goes through an apply(_:to:) that only sets known keys on the fetched record — so a newer build's extra fields survive an older build's write-back untouched. That second habit is protobuf's unknown-field preservation, reimplemented as a coding convention.

The tolerant reader's blind spot: it tells you nothing about whether skipping was safe. That's a semantic question the format can't answer.

The strict reader

Sometimes tolerance destroys data. The failure shape to check for: a read-modify-write cycle under last-writer-wins. An old device tolerantly half-reads a record written by a newer build, the user edits it, and the push wins the LWW race — the newer build's fields survive structurally (unknown-field preservation) but may now be semantically stale or contradictory, and there is no error anywhere in the chain. When fields are interdependent, "decoded what I recognized" plus "my write wins" equals silent corruption.

The CloudKit codec's answer is strictness at a coarser granularity: every record carries a format-version stamp from build one; additive changes don't bump it; breaking changes do, and decode refuses any record stamped newer than the running build. A refused record pauses — queued edits survive, nothing is overwritten — where a tolerated one would have clobbered. Two rules complete the pattern:

  • Never write to a record you couldn't decode. Blind re-encoding would stamp the version back down while half-updating fields — the one way to actually corrupt the contract.
  • Refusal is a rejection, not a transport failure. It must not abort the sync run or trigger retries; it must surface in UI ("N items need the app update to sync on this device"), because silent partial sync is indistinguishable from a bug.

The resulting degradation in a mixed-version fleet is per-record and one-directional: old builds keep reading and writing everything they understand (new builds read old records by construction), and only records the new build has rewritten pause on the old device until it updates.

Choosing, and the hybrid that usually wins

The axis is what happens after a partial read. Read-only consumers — metrics pipelines, log processors, display-only clients — should be tolerant; a skipped field costs a blank column. Read-modify-write participants in a conflict-resolving sync need a strict backstop, because their partial reads become authoritative writes.

In practice you want both, at different granularities: tolerant at field granularity, strict at version granularity. Additive evolution flows freely; the version stamp is the fire door, reserved for changes where partial reading is provably unsafe. If the fire door is triggering often, the schema design is wrong — new fields should be independent of old ones precisely so that an old build editing labels can't invalidate a new build's adjacent field.

Rules that transfer

  • Stamp a format version from the first shipped build. Retrofitting one after heterogeneous unstamped data exists is miserable.
  • Centralize wire identifiers (field numbers, key strings, record type names) in one place and treat them as append-only.
  • Additive-first: prefer new optional fields with defaults over changed semantics of existing ones. Never repurpose an identifier.
  • Ship readers before writers. A breaking format change deploys as two releases: first the build that can read v2, later the build that writes it — the fleet-skew window closes before the new format exists.
  • Preserve what you don't understand on write-back; refuse to write what you couldn't read.
  • Decide the degradation UX at design time. "Pause and tell the user" is a feature; discovering the behavior in production is an incident.

References