Observability

Huddle runs a self-hosted Grafana LGTM stack, driven by OpenTelemetry. It lives in the observability repo (a Railway template) and every service ships telemetry to it over Railway’s private network.

The stack

Grafana

Dashboards + alerting. Google OAuth sign-in; alerts fan out to Discord.

Prometheus

Metrics (remote-write receiver). 7-day / 1 GB retention.

Loki

Logs (OTLP + Faro). 14-day retention, structured-metadata enabled.

Tempo

Traces. 14-day retention (block_retention: 336h); runs the local-blocks processor for TraceQL metrics.

Grafana Alloy

The collector — OTLP receiver, blackbox prober, and the span-metrics / service-graph connectors.

Self-hosted, private network

Prometheus/Loki/Tempo are self-hosted with auth_enabled: false, reachable only over Railway’s private network (*.railway.internal) — there is no Grafana Cloud and no remote-write credential to manage. The public alloy. / otlp. / faro.huddlesurety.co hostnames are Railway public domains mapping to Alloy’s ports.

Alloy: the pipeline

All telemetry flows through Alloy (alloy/config.alloy).

The Alloy config is CI-validated

Because a broken pipeline silently drops telemetry, the repo’s CI runs mise run validate:alloy (alloy fmt + a config check) on every push, so a malformed config.alloy fails the build rather than shipping.

• OTLP receiver on :4318 (HTTP) → routes metrics, logs, and traces to their exporters (Prometheus remote-write, Loki, Tempo).
• Metrics get an instance label promoted from service.instance.id (or replica.id for the app). This is load-bearing: without a per-replica instance label, two replicas emit the same series and Prometheus rejects the batch with “duplicate sample for timestamp … overrides not allowed.”
• Traces are tee’d to two connectors that produce metrics from spans:
  • spanmetrics (namespace traces_spanmetrics_*) — RED metrics, with extra dimensions deployment.environment, org.id, and workflow.type.
  • servicegraph — the service dependency graph. It carries three Temporal-specific tweaks so worker edges form correctly:
    • Postgres otelpgx client spans (which carry db.namespace/db.system, not legacy db.name) are recognized as DB requests so the API → Postgres edge shows up. The API now also traces its S3 (Tigris) client calls (otelaws instrumentation, aws.s3.bucket/aws.s3.key attributes on operation-named spans such as S3.GetObject), so object-storage calls likewise appear as CLIENT spans in the API’s traces.
    • A workflow_span_kind transform promotes INTERNAL RunWorkflow:* spans to SERVER. The TypeScript SDK’s OTel interceptors emit every workflow span as INTERNAL (Go emits CLIENT/SERVER, Python emits SERVER), and the servicegraph connector skips INTERNAL spans — so without this the API → Notification edge never formed. Promoting them matches what the Python workers already emit.
    • virtual_node_peer_attributes and the connector store ttl (15s) are tuned so client/server spans reliably pair (they only pair while both sit in the store) without misclassifying the Notification → API / → Resend HTTP calls as virtual nodes.
• Tail sampling (storage branch only). Traces bound for Tempo first pass through a tail-sampling processor that keeps every error trace, every trace ≥ 1s, and 10% of the rest (decision_wait = 15s, buffering up to 50k traces). The decision is made per whole trace, so a kept trace retains all of its app → api → rag spans rather than shattering into rootless fragments. The spanmetrics and servicegraph connectors deliberately read the unsampled 100% stream — computing span metrics after a 10% sample would undercount request rates ~10× and drop edges.
• Faro receiver for browser RUM (Web Vitals + exceptions) from the app.

Health probes

Alloy’s blackbox exporter probes service health endpoints across all three environments (development included), every 60 seconds:

Target	Endpoint
App (dev / staging / prod)	/api/health
API (dev / staging / prod)	/

This produces probe_success metrics — a reliable UP/DOWN signal decoupled from the services’ own telemetry push. RAG and Notification have no HTTP endpoint, so their “status” is derived from a Temporal worker metric instead — temporal_num_pollers{poller_type="workflow_task"} (> bool 0), which stays positive as long as the worker is polling its task queue. The Status dashboard has a row per service — App, API, RAG, and Notification.

This used to read the worker task-slots metric

RAG/Notification status previously came from temporal_worker_task_slots_available. That metric is only emitted by fixed-size slot suppliers, so once RAG switched to a resource-based WorkerTuner (see RAG › Architecture) it stopped being reported and the status panels/alert pinned RAG to DOWN — and even on a fixed-slot worker it read 0 when fully busy, a false DOWN. temporal_num_pollers is emitted for every supplier type and stays > 0 while the worker polls, so it’s a true liveness signal for both worker services.

Grafana

Provisioned dashboards, datasources, and alert rules (grafana/provisioning/).

• Dashboards: App (process + Faro RUM + exceptions), API (process + request RED + a tracing section with per-route latency, DB p95, and a service graph), RAG (process + LLM token/cost + workflow & activity metrics from span-metrics + a TraceQL run table), Notification (process + an Emails row from the notification.emails.sent counter — sent/failed/by-template — plus workflow & activity metrics and a TraceQL run table, mirroring RAG), and Status (per-service uptime). All are filterable by an $environment variable.
• Datasources: Prometheus (default), Tempo (service map + traces-to-metrics wired to Prometheus, TraceQL streaming on), and Loki.
• Auth: Google OAuth — admin for a configured email, editor for the org domain, viewer otherwise. The password login form is disabled.

Alerting

Production only, 1-day evaluation window

Alert rules fire for production only — staging and development rules were explicitly removed. Each rule evaluates over the last 1 day (86400s), not 7 days (the 7-day figure is Prometheus’s retention, a different setting).

Alert rules exist for App / API / RAG only — Notification has a Status dashboard row (same temporal_num_pollers liveness query) but no alert rules yet. Each covers Status (App/API on probe_success == 0; RAG on temporal_num_pollers{poller_type="workflow_task"} == 0 — for: 3–5m), CPU, Memory, and Network (for: 10m), all notifying a single Discord contact point.

All rules now live in a single Alerts folder grouped by service — the earlier layout used a per-service folder with a production group each. Because the folder no longer disambiguates the alert name in a notification, every rule title now carries a service prefix (e.g. API Status, RAG Memory).

Distributed tracing

A single request is traced browser → app → api → worker (RAG or Notification) into one Tempo trace. Each service exports spans over OTLP to Alloy, and trace context propagates across each boundary by a different mechanism — inject-only at the browser edge, manual traceparent injection app → api, and a Temporal ContextPropagator (plus requester org.id) across the api → worker Temporal boundary.

This has enough moving parts to warrant its own page:

Distributed TracingThe three-leg trace chain and how trace + org identity propagate across each hop.

Export policy

To control egress and cardinality:

• Metrics export on ~60s intervals; traces are exported promptly (~5s) so cross-service spans assemble quickly in Grafana rather than landing a minute apart.
• The instance label is the only per-replica dimension promoted — identity is carried as org (not user) to keep span/metric cardinality bounded.
• Stored traces are tail-sampled (all errors + all traces ≥ 1s + 10% of the rest); span-derived metrics are computed before sampling, so sampling shrinks trace storage without skewing RED metrics or the service graph.