Full graph →

Symbology

Symbology ingests SEC EDGAR filings and synthesizes structured equity research with tiered LLM inference. It is the most instructive system in this section because of what coordinates it: there is no Redis, no RabbitMQ, no managed queue service. A scheduler and a fleet of worker processes cooperate through a job table in the same PostgreSQL database that holds the filings — the database the system already trusts is also the queue.

SFI project: in production at symbology.online since 2025. Snippets below are lifted from server/symbology/database/jobs.py, server/prompts/model_configs.yaml, and server/symbology/llm/ at current HEAD.

The shape of the system

Three kinds of process share one database. A scheduler polls EDGAR on an interval and enqueues work. Workers claim jobs and execute pipelines — fetch a filing, extract XBRL financials, chunk and embed documents, generate summaries. A SvelteKit app renders the generated content, reading the database directly from its server routes (Kysely over pg); the FastAPI service that once sat between them was retired when the UI became its only consumer — one fewer deployment, one fewer schema boundary to keep in sync. Nothing talks to anything else directly; every hand-off is a row in the jobs table.

That indirection is the design. Workers are stateless and horizontally scalable — add a node, point it at the database, and it starts claiming work. A worker that dies mid-job leaves a row in in_progress that a reaper can return to pending. There is no broker to operate, monitor, back up, or explain to the next engineer.

The queue is the database

The core of the queue is one statement. A worker claims exactly one job, atomically, without ever blocking on another worker:

claimed_id = session.execute(
    text(
        f"""
        UPDATE jobs
        SET status = 'in_progress',
            worker_id = :wid,
            started_at = now()
        WHERE id = (
            SELECT id FROM jobs
            WHERE status IN ('pending', 'backoff')
              AND (scheduled_at IS NULL OR scheduled_at <= :now)
            ORDER BY {order_by}
            FOR UPDATE SKIP LOCKED
            LIMIT 1
        )
        RETURNING id
        """
    ),
    {"wid": worker_id, "now": now, "aging": aging},
).scalar()

FOR UPDATE SKIP LOCKED is the load-bearing clause. FOR UPDATE locks the selected row so no other transaction can claim it; SKIP LOCKED tells competing workers to skip past locked rows instead of waiting on them. Ten workers hitting this query concurrently each get a different job or an empty result — never a duplicate, never a lock queue. The ORDER BY includes an aging term so low-priority jobs can't starve indefinitely behind a stream of high-priority ones.

message-queues covers when this pattern suffices and when a real broker earns its keep. The short version: a Postgres queue is the right call when your throughput is modest (Symbology processes filings, not clickstreams), your workers already need the database, and exactly-once claiming matters more than fan-out. You give up pub/sub semantics and horizontal queue scaling; you gain transactional enqueue — a pipeline step can insert its follow-up jobs in the same transaction that commits its results, so the queue can never disagree with the data.

Job taxonomy and lifecycle

Every unit of work is typed, and the lifecycle is explicit:

class JobStatus(str, Enum):
    PENDING = "pending"
    IN_PROGRESS = "in_progress"
    BACKOFF = "backoff"       # transient failure; retry after delay
    COMPLETED = "completed"
    FAILED = "failed"          # retries exhausted
    CANCELLED = "cancelled"

class JobType(str, Enum):
    COMPANY_INGESTION = "company_ingestion"
    FILING_INGESTION = "filing_ingestion"
    CONTENT_GENERATION = "content_generation"
    EMBED_FILING = "embed_filing"
    FILING_DIFF = "filing_diff"
    DIFF_SUMMARY = "diff_summary"
    # ... backfill and pipeline orchestration types elided

BACKOFF is distinct from FAILED on purpose. EDGAR rate-limits; LLM APIs have transient errors. A backoff job re-enters the claimable pool once its scheduled_at passes, with backoff_count and retry_count tracked separately so the system can distinguish "the API was busy" from "this filing is malformed." Each job row also records worker_id, started_at, duration, and a JSON result — which makes the jobs table double as an execution log. Most operational questions ("why is this company's page stale?") are answered with a SELECT, not a log dive. logging covers the structured-logging side; service-interactions covers the orchestration patterns.

Tiered, local-first inference

Generation is layered L1→L4 — single-document summaries feed filing- and company-level synthesis, which feeds short intro copy — and roughly 98% of it runs on on-prem GPUs. Every stage is declared in one YAML file:

model_configs:
  l1_document_summary:         # one per document — high volume
    model: google/gemma-4-e4b
    max_tokens: 8192
    temperature: 0.2
  l2_topic_diff_summary:
    model: google/gemma-4-e4b
    # gemma-4-e4b is a reasoning model: it spends tokens "thinking" before
    # emitting the answer. 512 was too tight — it hit the cap mid-reasoning
    # and returned empty content. Give it room to finish the summary.
    max_tokens: 2048
    temperature: 0.3
  l3_company_main_content:     # aggregate synthesis over many summaries
    model: google/gemma-4-e4b
    max_tokens: 4096
    temperature: 0.5
  l4_company_intro_content:    # short flavor text — higher temperature
    model: google/gemma-4-e4b
    max_tokens: 1024
    temperature: 0.7

Provider routing is one rule:

def _provider_for(model: str) -> str:
    # Anything whose name starts with "claude" goes to the Anthropic API;
    # everything else goes to the local OpenAI-compatible endpoint.
    return "anthropic" if model.lower().startswith("claude") else "openai"

The "OpenAI endpoint" is not OpenAI: it is an on-prem GPU host serving gemma-4-e4b behind the OpenAI-compatible API that both LM Studio and Ollama speak. Each worker is pinned to an inference host via OPENAI_API_HOST in its compose service, so the fleet scales by pairing a worker with a GPU rather than queueing on one shared endpoint. Temperature still follows function — factual extraction near 0.2, prose that should vary near 0.7 — and because the mapping lives in one YAML file, re-tiering after a model release is a config change, not a code change.

Overflow to external providers

The ~2% that leaves the building is the requests a local context window can't hold — a single very long section (a sprawling risk_factors) can exceed the local model's ceiling. The reroute to Anthropic is decided twice:

  • Preemptively. Before a call, the worker estimates context cost as (prompt_tokens + max_output_tokens) * safety_margin, using a deliberately pessimistic chars-per-token heuristic. If that exceeds OPENAI_OVERFLOW_THRESHOLD_TOKENS (55k in production), the stage config is swapped for an Anthropic one mirroring its options. Counting the reserved output matters: the window must hold prompt and completion, so a prompt that looks small can still overflow once its output is reserved. The same math sizes the local model's dynamic context window, so the reroute decision and the window sizer never disagree about what a request costs.
  • Reactively. When the estimate undershoots and the local endpoint returns its overflow error (HTTP 400 with Context size has been exceeded.), the caller catches it and retries against the larger Anthropic context. Matching on the message rather than the bare 400 keeps genuine bad-request bugs failing fast instead of pointlessly re-routing.

Data quality is engineering work

The unglamorous majority of Symbology's evolution has been data quality: XBRL facts that disagree with the filing text, duplicate filings under amended accessions, companies that change names. Two structural answers emerged. Generated content is content-hashed, so regeneration is idempotent and unchanged output is never rewritten. And FILING_DIFF / DIFF_SUMMARY job types make change detection a first-class pipeline stage rather than an ad-hoc script — when a company amends a filing, the diff is computed, summarized, and surfaced like any other content.

Honest debt

The schema debt called out in earlier revisions is paid: migrations are now versioned with Alembic rather than applied with SQLAlchemy create_all(). End-to-end test coverage remains thin relative to the unit suite — tracked, not hidden. The deployment story is converging on the platform described in sfi-platform: a Helm chart published to the internal OCI registry, with the database moving to CloudNativePG.

References