Full graph →

Message Queues

A message queue separates deciding that work should happen from doing it. The moment those are separate acts — because the requester can't wait, because the worker might be busy or down, because the work needs to survive a crash — you need a place for pending work to live. That place is the queue, and the design questions it answers are always the same three: durability (does accepted work survive a process dying), progress (does work move through transient failure without human help), and coordination (do N workers each process an item exactly once, or close enough).

This article grounds the concepts in symbology, SFI's SEC-filing analysis platform. Its queue is not a broker: it is a single Postgres table (jobs), claimed by a pool of Python workers, fed by both a CLI and a SvelteKit UI that inserts rows directly. That choice — the queue as a table in the database you already run — turns out to answer all three questions with tools Postgres already has.

The Queue Is Where State Lives

The deepest reason to queue is not throughput; it is that a queue lets workers be stateless. A symbology worker is a loop: claim a job, run its handler, record the outcome, repeat. Everything about the work — what to do (job_type, params), how it's going (status, retry_count, error), when to try again (scheduled_at), what it produced (result) — lives in the job row, never in worker memory. Kill a worker mid-deploy and nothing is lost: the row is still there, and a heartbeat-reaping sweep returns any claimed-but-orphaned jobs to the pool without spending their retry budget.

Statelessness is what makes the mundane operations cheap. Scaling is starting more workers. Deploying is stopping them. Debugging starts with a SELECT, not with attaching to a process. And because the queue's contract is just "a row of this shape," it doubles as a cross-language API: symbology's UI has no HTTP path to the Python backend at all — when a supporter requests coverage of a new ticker, the SvelteKit server inserts a synthesis_request row, and the worker fleet takes it from there. The table is the interface.

A Table Is a Queue

The textbook objection to database-backed queues is lock contention: two workers grab the same row, or serialize behind each other's locks. Postgres dissolved this objection with FOR UPDATE SKIP LOCKED — take the first eligible row that nobody else has locked, without waiting. Symbology's claim is a single statement:

UPDATE jobs
SET status = 'in_progress', worker_id = :wid, started_at = now()
WHERE id = (
    SELECT id FROM jobs
    WHERE status IN ('pending', 'backoff')
      AND (scheduled_at IS NULL OR scheduled_at <= :now)
    ORDER BY priority, created_at
    FOR UPDATE SKIP LOCKED
    LIMIT 1
)
RETURNING id

The single-statement shape is load-bearing. A SELECT FOR UPDATE followed by a separate UPDATE only holds the lock across both statements on a direct connection; behind a transaction- or statement-pooling proxy like PgBouncer, the two statements can land on different backends and two workers can claim the same row. One statement is atomic under every pooling mode.

What this buys is at-least-once delivery: a claimed job whose worker dies is re-run by another. At-least-once is the honest guarantee — exactly-once is a marketing term for at-least-once plus idempotent consumers — so handlers must tolerate re-execution. Symbology leans on content hashing (generated content dedups by SHA256) and enqueue-time dedup (don't create a job that already exists for the same target) rather than pretending redelivery won't happen.

Not All Non-Success Is Failure

The naive queue has two outcomes: done or failed-retry-later. Running one for a while teaches you there are at least four, and conflating them either burns retries on certainties or retries the impossible forever. Symbology's worker distinguishes them in one except ladder:

try:
    result = handler(job.params or {})
    complete_job(job.id, result=result)
except DependencyNotReady as exc:
    # Not a failure: a dependency isn't ready yet. Defer without
    # consuming a retry; the job polls until the dependency lands.
    backoff_job(job.id, reason=str(exc))
except NonRetryableError as exc:
    # A genuine failure a retry can never fix (e.g. a ticker that
    # doesn't exist on EDGAR). Dead-letter immediately.
    fail_job(job.id, error=str(exc), terminal=True)
except Exception:
    # Transient until proven otherwise: retry with capped exponential
    # backoff, dead-letter (status FAILED) once max_retries is spent.
    fail_job(job.id, error=traceback.format_exc())

Waiting is a state, not an error. A company-page job that needs filing pages which haven't generated yet raises DependencyNotReady; the job goes to backoff with a scheduled_at in the near future and is re-claimed when its time arrives. No retry budget is consumed, because nothing went wrong. The first re-check is quick (dependencies usually land within minutes) and later ones stretch out to a cheap poll.

Some failures are certainties. When a user requests synthesis for a ticker EDGAR has never heard of, retrying three times with backoff is theater — the answer will not change. NonRetryableError dead-letters the job immediately, preserving the error message (prefixed with a stable marker like invalid_ticker:) as a contract the UI can branch on to show the user something actionable.

Everything else gets the classic treatment: capped exponential backoff, then the dead letter state (FAILED) where a human — or an alert — can inspect the preserved error. The dead letter state is what keeps one poisoned job from stalling the pipeline while still refusing to silently discard work someone asked for.

Priority Is Scheduling Policy

With the queue in a table, priority is just a column and an ORDER BY — which means scheduling policy is something you can design rather than configure around a broker's primitives. Symbology's bands put fast, required feedstock (ingestion, embedding) ahead of steady content backfill, and a dedicated front-of-queue band (REQUEST) for supporter-originated synthesis requests, because a human is actively watching that one.

Strict priority starves the low bands under sustained load, so the claim's real ordering is effective priority: priority - floor(age / interval), floored at zero. A long-waiting backfill job gradually climbs the bands and eventually ties fresh high-priority work, where its age wins the created_at tiebreak. Aging keeps the queue honest — everything eventually runs — without giving up the responsiveness that priority exists to provide.

Orchestration Without an Orchestrator

The temptation, once you have a queue, is to add a workflow engine on top. Symbology deliberately doesn't: the job is the only primitive, and handlers do their work in-process. The two places that need multi-job coordination use the queue's own states instead.

A synthesis_request job is a tracking job: it validates the ticker, enqueues one page-generation child per form (deduping against any already in flight), then raises DependencyNotReady to defer itself until the children settle. The UI polls one row for the whole request's progress; the "orchestrator" is an ordinary job using the same waiting state as everything else.

Re-entrant orchestration has one sharp edge worth recording: a handler that adopts existing jobs must distinguish its own children from the debris of prior attempts. Symbology's first cut adopted the newest page job per form regardless of status — so one failed page job poisoned its ticker forever: every new request instantly adopted the corpse and re-failed. The fix is a dividing line the queue already provides: the request's own created_at. A failed child older than the request is history — skip it and enqueue fresh, making a user's re-request a genuine retry. A failed child newer than the request is this request's own fan-out dying — fail terminally and notify, so a genuinely broken pipeline can't be hammered in a loop.

Things That Go Wrong

Orphaned claims. A worker that dies after claiming leaves a job in_progress forever. Symbology's workers heartbeat a registry table; a periodic reap returns jobs claimed by silent workers to the pool — without incrementing retry_count, because the job didn't fail, its worker did.

Enqueue races. Check-then-insert idempotency ("is there already an active request for this ticker?") has a window where two concurrent requests both pass the check. Close it with a partial unique index when correctness demands it; symbology tolerates it because the handler dedups children anyway, so the duplicate tracking row is cosmetic.

The invisible queue. A queue with no instrumentation fails silently: work piles up and nobody notices until a user does. The signals that matter are queue depth per type, age of the oldest eligible job (the latency a user actually experiences), processing rate against enqueue rate, and dead-letter count — which should be zero in steady state, so any non-zero value is an alert. See metrics for instrumenting these and logging for the event patterns that make a claimed-ran-completed lifecycle debuggable.

Enum migrations. With statuses and job types as Postgres enums, adding a value is ALTER TYPE ... ADD VALUE IF NOT EXISTS — but removing one is a full type-recreate dance. Plan for the type to only grow; see postgresql.

When a Table Isn't Enough

The table-as-queue pattern holds remarkably far — tens of jobs per second with SKIP LOCKED, which is orders of magnitude beyond symbology's needs — but it is a work queue, and not everything shaped like messaging is one. Publish/subscribe (one event, many independent consumers) fights the model: a row can be claimed once, so fan-out means duplicating rows per consumer. Event streaming platforms like Kafka keep an ordered, replayable log rather than a queue — the right substrate for audit trails and pipelines where consumers advance at their own pace through history. Dedicated brokers like RabbitMQ earn their operational cost when you need routing topology (exchanges, bindings) or cross-language delivery at volumes where Postgres would become the bottleneck; managed services like SQS trade features and portability for zero operations. The design questions stay the same — durability, progress, coordination — only the substrate answering them changes.

References