Full graph →

The Shell

The shell is the glue layer of modern systems. It is not a programming language competing with Python or Go — it is the automation substrate that is already present at every seam where systems meet: container builds, CI runners, boot sequences, git hooks, cron jobs, systemd units. Fluency with the shell is what makes those seams legible, and what lets you join two systems together with five lines instead of a service.

The examples in this article are drawn from SFI work: cluster diagnostics tooling from a bare-metal Kubernetes engagement, certificate-trust bootstrapping from the homelab fleet, and this repository's own automation.

Why Shell Survives

Fifty years on, three properties keep the shell load-bearing, and none of them is nostalgia.

It is already there. An alpine container image, a cloud-init boot script, a CI runner, a systemd unit's ExecStartPre — every one of these hands you a shell with zero installation. Compare a deployment health check written in shell against the same check in Python: the Python version needs requests or urllib, a try/except, and a virtual environment or container layer to carry the dependency. The shell version is four lines, depends only on curl, and runs on anything:

steps:
  - name: Verify deployment
    run: |
      STATUS=$(curl -s -o /dev/null -w '%{http_code}' https://app.example.com/healthz)
      if [ "$STATUS" -ne 200 ]; then
        echo "Health check failed with status $STATUS"
        exit 1
      fi

Text streams and exit codes are a universal interface. Every program reads stdin, writes stdout and stderr, and returns an exit code — zero for success, anything else for failure. The pipe (|) composes outputs into inputs; &&, ||, and if branch on exit codes. No other interoperability standard has this reach: a tool written in Rust last year composes with one written in C in 1979, with no shared library, schema, or protocol negotiation.

The ecosystem is the language. bash itself is thin connective tissue. The actual vocabulary is the toolset: curl for HTTP, jq for JSON, kubectl for cluster state, grep/sed/awk for text, xargs for fan-out. Learning the shell is mostly learning to compose these — the composition operators fit on an index card.

The Composition Pattern

Nearly every useful shell automation has the same shape: enumerate, transform, act. Some API or tool lists things; jq or a template filters and reshapes the list; a loop acts on each item.

The HTTP flavor — page through an API, select targets, mutate them:

#!/bin/bash
set -euo pipefail

# Archive every org repo with no push in the last two years.
page=1
while :; do
  repos=$(curl -sf -H "Authorization: token $GITEA_TOKEN" \
    "https://gitea.zen.lofi/api/v1/orgs/sfi/repos?page=$page&limit=50")
  [ "$(jq length <<<"$repos")" -eq 0 ] && break   # ran off the last page

  jq -r '.[] | select(.pushed_at < (now - 63072000 | todate)) | .full_name' <<<"$repos" |
  while read -r repo; do
    curl -sf -X PATCH -H "Authorization: token $GITEA_TOKEN" \
      -H 'Content-Type: application/json' -d '{"archived": true}' \
      "https://gitea.zen.lofi/api/v1/repos/$repo"
    echo "archived $repo"
  done
  page=$((page + 1))
done

curl -f turns HTTP errors into non-zero exit codes so set -e can see them; read -r keeps backslashes literal. Those two habits prevent most of this pattern's silent failures.

The cluster flavor — the same shape, plus concurrency. This is the core of a diagnostics gatherer built during a bare-metal Kubernetes engagement (the full script also dumps Cluster API and vendor CRDs; it lives at .files/shell/k8s/k8s-logs.sh):

gather_namespace() {
  local ns=$1
  for pod in $(kubectl get pods -n "$ns" -o jsonpath='{.items[*].metadata.name}'); do
    mkdir -p "$log_dir/$ns/$pod"
    for container in $(kubectl get pod -n "$ns" "$pod" \
        -o jsonpath='{.spec.containers[*].name}'); do
      kubectl logs -n "$ns" "$pod" -c "$container" \
        > "$log_dir/$ns/$pod/$container.log"
    done
  done
}

# One background job per namespace; the PID array turns
# "wait for everything" into two lines.
pids=()
for namespace in $(kubectl get namespaces -o jsonpath='{.items[*].metadata.name}'); do
  gather_namespace "$namespace" &
  pids+=($!)
done
for pid in "${pids[@]}"; do wait "$pid"; done

Sequential, this takes minutes against a loaded API server; backgrounded, it takes the duration of the slowest namespace. Parallel fan-out with join semantics costs six lines of shell — the same structure in most languages costs a thread pool or an async runtime. See kubernetes for the cluster side of this tooling.

Automation Seams

The shell's strongest claim is at the seams — the moments between systems where no application runtime exists yet, and the alternative to a shell script is not a nicer program but nothing at all.

First boot. A new VM on the private network must trust the internal certificate authority before anything TLS-dependent can start. At this point in the machine's life there is no Python, no container runtime, possibly no configuration management agent — there is a shell and the OS trust store:

# Post-boot provisioning step, from the homelab fleet setup.
step ca bootstrap --ca-url "https://ca.lofi:4443" --fingerprint "$CA_FINGERPRINT"
step certificate install "$(step path)/certs/root_ca.crt"  # lands in the OS trust store

Two lines, and every subsequent component — package mirrors, internal APIs, git over HTTPS — works without per-application certificate configuration. See networking for the CA architecture this bootstraps into.

Repository hooks. This knowledge base regenerates its tree listing on every commit with a three-line hook — the kind of invariant ("the map always matches the tree") that would be absurd to enforce with anything heavier:

#!/bin/sh
set -e
cd "$(git rev-parse --show-toplevel)"
just build-kb-map
git add kb-map.txt

Container builds and CI. RUN lines in a Dockerfile, verification steps in a pipeline, ENTRYPOINT wrappers that template a config file before exec-ing the real process — all shell, all chosen for the same reason: the seam already speaks it.

The Complexity Budget

Everything above works because the examples stay simple. The shell's costs arrive quietly as scripts grow: there is no type system, error handling is primitive, word-splitting corrupts data that contains spaces, and string manipulation beyond jq's reach turns into sed archaeology. The discipline that keeps unattended scripts safe:

  • set -euo pipefail, always. Exit on error, on undefined variables, and on failures inside pipelines. Without it, errors are silently ignored and the script keeps executing — a recipe for destructive bugs.
  • Quote every expansion ("$var", "${array[@]}"). Run ShellCheck in CI; it catches nearly all of this class mechanically.
  • Expect a minimal environment. Cron jobs and containers run with a stripped $PATH and no login profile. A script that works interactively but fails unattended almost always has an environment assumption in it.

Know the tripwires that mean you have exceeded the budget — the signals that the next change should be a rewrite in a real language, not another function:

  • You need to modify structured data or aggregate across records, not just filter it through jq.
  • You need retries with backoff, partial-failure recovery, or any state that survives between runs.
  • Functions are growing past a couple of positional parameters.
  • You want unit tests.

The diagnostics gatherer above is an honest borderline case: the full script is ~150 lines, and it is still on the right side of the budget — it is linear enumerate-and-dump with no business logic, and its failure mode is a missing file you fix by re-running. The moment it needs per-resource error recovery or structured output for a downstream consumer, it should become a Go program. Google's shell style guide draws the same line at roughly a hundred lines or any non-straightforward control flow; the exact number matters less than auditing your own scripts against it.

Use shell for glue, orchestration, and the seams. Use a real language for business logic, data processing, and anything that needs tests.

Interactive Ergonomics vs. Unattended Scripts

A wave of modern tools improves the interactive shell substantially: zoxide (frecency-ranked cd), eza (ls with git awareness), fd and ripgrep (faster, .gitignore-respecting find/grep), fzf (fuzzy-find anything), just (a make without the footguns). Install them everywhere you type — see zsh for the interactive setup.

Do not let them into production scripts, Dockerfiles, or CI. Stock distributions and minimal images do not ship them, so a script that calls fd instead of find breaks silently on exactly the machines you cannot log into. Code that runs unattended sticks to the POSIX and GNU baseline; ergonomics are for humans at keyboards.

References