Architecture

RAG is a Temporal worker. The Go API is the Temporal client that starts these workflows; RAG executes them, and the API surfaces their progress by watching Temporal and streaming status over its …/watch SSE endpoints (which the app consumes as tRPC subscriptions).

Worker

The worker (app/temporal/worker.py) connects with the deployment environment as its Temporal namespace and polls a single task queue (rag-tasks by default). It registers 6 workflows and 22 activities.

• Activities are synchronous (def, not async def) and run on a ThreadPoolExecutor — the heavy work (PDF rendering, OCR, embeddings, DB) is blocking and CPU/IO-bound. Admission is resource-based: WorkerTuner.create_resource_based admits activities against memory/CPU targets (worker_target_memory 0.8, worker_target_cpu 0.9), bounded by configurable slot caps (worker_activity_slots_min 8, worker_activity_slots_max 200; workflow slots capped at 500), and the thread pool is sized to the activity-slot max — not a fixed count. (Resource-based slots replaced the old fixed max_concurrent_* settings; the SDK can’t combine the two.)
• Workflows run in Temporal’s sandbox, with sniffio and opentelemetry passed through (the httpx-based API SDK’s async-client garbage-collector finalizers run on arbitrary threads where a lazy import sniffio inside httpcore would crash, and the OTel imports don’t survive the sandbox otherwise).
• The worker also runs a minimal background health server (start_health_server, port 8080, GET → 200 "ok") in a daemon thread, for the Railway platform healthcheck. It is not the source of the status signal — that comes from a Temporal poller metric, not an HTTP probe.
• Services are imported lazily inside activities, not at module top, to keep the workflow sandbox’s import graph small and defer heavy deps (fitz, numpy, Voyage) off the startup path.

Workflows

Workflow	Arg	What it does
ParseDocumentWorkflow	document_id	Parse → clean → chunk → embed → mark-parsed
GenerateFieldWorkflow	field_id	Retrieve → rerank → LLM extract → ground → evidence
FindBondWorkflow	request_id	Locate the bid/final bond form(s) in the docs
FillBondWorkflow	bond_id	Detect blank fields and fill them (vision-LM)
DeleteDocumentWorkflow	document_id	Remove a document’s chunks/elements
ProcessBondRequestWorkflow	request_id	Parent — orchestrates the above

Each stage is a separate activity with its own timeout and retry, so a failure resumes at the failed stage rather than re-running the whole chain.

ProcessBondRequest: the orchestrator

ProcessBondRequestWorkflow is the durable, server-side replacement for what used to be a fire-and-forget chain kicked off in the browser (which died on a page refresh). It takes only a request_id and spawns the others as child workflows:

graph TD
    P[ProcessBondRequestWorkflow] --> R[resolve_process activity<br/>→ docs to parse + fields to generate]
    R --> PH1{Phase 1 — parse, strict}
    PH1 --> PD1[child: ParseDocumentWorkflow ×N]
    PH1 --> PH2{Phase 2 — find + generate, best-effort}
    PH2 --> FB[child: FindBondWorkflow]
    PH2 --> GF[child: GenerateFieldWorkflow ×M]
    PH2 --> PH3{Phase 3 — fill, best-effort}
    PH3 --> TS[terminate_stale_fills activity]
    TS --> FL[child: FillBondWorkflow ×K]

• Phase 1 (parse — strict): one ParseDocumentWorkflow child per unparsed document, gathered concurrently. It drains all children first (each completed one is durably marked parsed), then fails the run if any failed — so a resubmit re-parses only the failures.
• Phase 2 (find + generate — best-effort): once parsing succeeds, it spawns FindBondWorkflow plus one GenerateFieldWorkflow per generated field, all concurrently. Individual child failures are logged but don’t sink the rest.
• Phase 3 (fill — best-effort): once find + generate finish, the parent fills every located form (FindBondWorkflow’s fillable_form_ids). Fill must run last because its data_prep reads both the Bond records find wrote and the freshly generated field values. Since a child start can’t carry a conflict policy, the parent first runs the terminate_stale_fills activity to kill any in-flight FillBondWorkflow with the same derived ID (e.g. a wizard-triggered fill whose data_prep ran before all generated values landed), then spawns one FillBondWorkflow child per fillable form. A child already running under the same ID is handed off (not double-filled), and one terminated by a Go-side re-trigger is treated as superseded, not fatal.

Deterministic child IDs keep the UI wired up

The parent computes each child’s workflow ID with the same scheme the Go API uses — {task_queue}/{WorkflowName}/sha256("[arg]"). Because the children reuse the exact IDs the API’s watch endpoints already stream, the app’s existing per-document parse and per-field generate status subscriptions observe them unchanged. This ID parity with the API is load-bearing — don’t change one side without the other.

Document ingestion pipeline (parse)

graph LR
    A[Download PDF<br/>presigned URL] --> B{Scanned?}
    B -->|text layer| C[PyMuPDF blocks]
    B -->|scanned| D[Tesseract OCR<br/>ProcessPool per page]
    C --> E[Store elements]
    D --> E
    E --> F[Clean + chunk]
    F --> G[Embed<br/>Gemini · LiteLLM]
    G --> H[Store chunks<br/>pgvector]

• Parser (parser.py): downloads the PDF via a presigned URL, samples pages to decide scanned vs. text-layer. Text PDFs use PyMuPDF get_text("dict") blocks; scanned PDFs render each page to PNG and OCR in parallel via a ProcessPoolExecutor (one process per page — Tesseract isn’t thread-safe). Output is DOMElements with normalized [x1,y1,x2,y2] bboxes.
• Clean + chunk: OCR noise (single-char runs, dot/dash leaders, U+FFFD) is stripped, then elements aggregate into ~1000-char chunks (1500 max, 200 overlap), each keeping its source element_ids for grounding.
• Embed: the EmbeddingService embeds via litellm.embedding (gemini/gemini-embedding-001, 1536 dims, task_type="RETRIEVAL_DOCUMENT") in parallel batches with a shared token-bucket rate limiter; only rows with a NULL embedding are embedded, so it resumes cleanly. Calls route through the LiteLLM gateway and carry the requester org / workflow as x-litellm-customer-id / x-litellm-tags headers for per-org, per-workflow spend attribution.

Field extraction pipeline (generate)

graph LR
    A[Fetch field spec<br/>from API] --> B[Embed query]
    B --> C[Hybrid search<br/>dense + BM25 → RRF]
    C --> D[Rerank<br/>Voyage]
    D --> E[LLM extract<br/>strict JSON schema]
    E --> F[Ground quote<br/>→ page + bbox]
    F --> G[Evidence PNG<br/>+ update field]

• Embed query (EmbeddingService.embed_query, task_type="RETRIEVAL_QUERY"): the query vector is served from a Postgres query-embedding cache (query_embedding_cache) before any API call — a hit returns in-process, spending no tokens and skipping the rate-limit bucket. Only a miss embeds through the gateway and back-fills the row.
• Hybrid retrieval (vectorstore.hybrid_search): a dense arm (pgvector cosine, <=>) and a BM25 arm (ParadeDB pg_search) run on separate pooled connections and are combined with Reciprocal Rank Fusion (with optional anchor-term boosting).
• Rerank: Voyage rerank-2.5-lite via litellm.rerank, routed through the gateway (the alias voyage/rerank-2.5-lite the proxy declares) and gated by the rerank_enabled flag — no longer by the presence of a Voyage key, since the worker no longer holds one.
• LLM extraction: a Gemini chat model (via litellm.completion) with strict JSON-schema structured output (additionalProperties: false, reasoning_effort kept minimal), under an explicit per-call timeout. Confidence is a weighted blend of vector score and the model’s self-reported confidence.
• Grounding (grounding.py): matches the answer/quote back to a source element (exact substring, else rapidfuzz fuzzy match ≥ 85), producing a page number and bbox. The evidence step re-renders that page with a highlight rectangle, uploads the PNG via a presigned URL, and writes the value + evidence reference + confidence back to the API.

OpenRouter SDK → LiteLLM → gateway (#142, then #145)

This happened in two steps. #142 dropped the OpenRouter Python SDK (a single get_openrouter_client()) for module-level litellm.completion(...) / litellm.embedding(...) — no per-instance client object (each service’s close()/__exit__ is now a no-op), model IDs moved from google/…·openai/… to gemini/… prefixes, and the bond finders that ran on openai/gpt-5.4-mini moved to Gemini too. LiteLLM defaults Gemini 3 to temperature 1.0, so the services no longer pass an explicit temperature (structure stays pinned by the strict json_schema).

If you read this aside before, it said calls then went directly to Gemini with a GOOGLE_API_KEY (bring-your-own-key). That is no longer true. #145 routes everything through the self-hosted LiteLLM gateway: setup_litellm() sets litellm.use_litellm_proxy = True, the worker holds no provider key (GOOGLE_API_KEY is gone — LITELLM_PROXY_API_BASE / LITELLM_PROXY_API_KEY are now required), and each call passes gateway_headers() (x-litellm-customer-id = org, x-litellm-tags = workflow) for spend attribution. Three declared-but-dead dependencies remain in pyproject.toml — openrouter (dead since #142), voyageai (dead since reranking moved to litellm.rerank through the gateway), and google-genai (never imported) — flagged for the rag team, no code imports any of them.

Find & fill bond forms

• Find (bond_finder.py, final_bond_finder.py): a fast scan pass produces page references only (no page text crosses the activity boundary — a recurring memory-hygiene rule), then an LLM refine pass re-fetches just those pages to pick the winning document and consecutive page span. The final-bond finder dedups against any bid bond already found. A best-effort find_bid_bond_percent activity runs concurrently with the refine pass to extract the bid-bond penalty percentage; it degrades to 0.0 on failure. Percentages are a strict-number schema coerced through a shared services/pct.py (coerce_pct) and clamped to [0, 100], while performance/payment specs are forced to 100%. write_bonds centralizes a delete-then-insert for idempotency and maps internal refs to the SDK’s bond_type (bid/final).
• Fill (bond_filler.py): a two-pass vision-LM flow — pass 1 detects blank fields (bboxes), pass 2 fills values from known contractor data. Penal sums and their word forms are computed in Python, not by the model. Surety details (name/address/phone) are fetched live from the API by org_id in data_prep rather than hardcoded (phone falls back to a default if the fetch fails).

Calling the Go API

Everything goes through the git-installed huddlesurety-api SDK, built by get_huddle_client() on a custom httpx.Client:

• Auth: a request hook injects Authorization: Bearer <KEY_AUTH_SECRET> for requests to the API host (service-key auth).
• Timeout: an explicit httpx.Timeout(10s, connect=10s) — set wider than httpx’s 5s default because some API calls (e.g. synchronous PDF trimming) take longer than a couple of seconds.
• The client is instrumented so RAG → API calls emit a CLIENT span and inject traceparent, producing the RAG → API edge in the service graph.

HTTP only in activities, never in workflows

resolve_field_context / resolve_bond_request_context (api_client.py) turn a single ID from the API into full context. They run only inside activities — workflow code must stay deterministic and must not make HTTP calls.

Distributed tracing & identity

Baggage doesn’t cross the Temporal boundary, so RAG carries requester identity itself via worker interceptors (app/o11y/identity.py):

• The inbound workflow interceptor reads the org header off the workflow header; the outbound interceptor re-injects it onto every child workflow and activity (mirroring the Go API’s identity propagator, which doesn’t run on the Python side).
• The activity interceptor reads it back and stamps org.id + workflow.type on the OTel span (and a current_org_id context var).

The header key is x-huddle-org-id, JSON-encoded to match the API’s propagator. Only org is propagated (not user), a deliberate cardinality choice. Traces export on a 5s interval (metrics on 15s) so app → api → rag traces assemble quickly in Grafana. Custom rag.* metrics record LLM/embedding tokens, call duration, and USD cost per model, plus a rag.embedding.cache counter with a three-way result (hit/miss/error) and an op (read/write) so a dead cache reads as error, never as a cold-but-healthy miss. Note that provider-side spend and latency now live in the LLM gateway’s own dashboard, not in Tempo — RAG spans end at the litellm call.

Database

Schema document, migrated by goose (app/db/):

• document.chunk — id, document_id, project_id, org_id, raw, embedding vector(1536), element_ids UUID[], plus a BM25 index (USING bm25 … WITH (key_field='id')) for ParadeDB search. The embedding dimension is substituted into the migration from EMBEDDING_DIMENSION.
• document.element — raw text blocks with bbox, page_number, element_type.
• document.query_embedding_cache — query_hash (PK), query, model, dimensions, embedding. Caches query-side embeddings keyed on a SHA-256 of (model, dimensions, task_type, query), so a model swap or EMBEDDING_DIMENSION change misses into fresh rows rather than needing invalidation. Every read/write failure degrades to a miss — a cache outage looks exactly like the pre-cache path plus a log line.

A process-wide psycopg3 ConnectionPool (db_pool_min_size 2 / db_pool_max_size 30, sized independently of the activity-slot cap) is shared across activities. Vectors are passed as pgvector text literals; IDs convert between ULID (Go/S3) and UUID (DB) via idconvert.