Distributed Tracing

A single user action can touch several tiers — the browser, the Next.js app, the Go API, and the Temporal workers it triggers (RAG in Python and Notification in TypeScript). Distributed tracing stitches those into one trace in Tempo, so you can follow a request end to end and see where time went.

Getting there took solving three separate propagation problems — one per boundary — because each hop uses a different transport. This page documents each.

The trace chain

graph LR
    B[Browser<br/>Faro] -->|W3C traceparent<br/>NOT joined| A[App<br/>Next.js server]
    A -->|W3C traceparent<br/>manual inject| API[API<br/>Go / Echo]
    API -->|Temporal header<br/>OTel interceptor| RAG[RAG worker<br/>Python]
    API -->|Temporal header<br/>OTel interceptor| NOT[Notification worker<br/>TypeScript]
    A -.OTLP.-> T[(Tempo)]
    API -.OTLP.-> T
    RAG -.OTLP.-> T
    NOT -.OTLP.-> T
    B -.Faro.-> T

Every service exports spans over OTLP/HTTP to Alloy, which forwards them to Tempo. Trace context — the traceparent that ties spans into one trace — rides the actual request between tiers via the three mechanisms below.

Trace context vs. telemetry export

Two different channels are in play. Telemetry export is each service pushing its finished spans to Alloy (fire-and-forget, OTLP). Context propagation is the traceparent (and our org identity) traveling with the live request so the next tier knows which trace it belongs to. Tracing only works when both happen — this page is about the second.

Leg 1 — Browser → App (deliberately not joined)

The browser runs Grafana Faro with its tracing instrumentation, which emits a same-origin traceparent on requests to the app.

By default Next would adopt that browser traceparent as the parent of its server span — but the browser span is exported through Faro, not through the server’s OTLP pipeline, so the app’s trace would show <root span not yet received> and never resolve.

So the app’s OpenTelemetry setup (src/instrumentation.ts) wraps the W3C propagator to be inject-only — it writes traceparent outbound but ignores inbound headers:

const injectOnly = (inner: TextMapPropagator): TextMapPropagator => ({
  inject: (ctx, carrier, setter) => inner.inject(ctx, carrier, setter),
  extract: (ctx) => ctx,   // ← drop the inbound browser traceparent
  fields: () => inner.fields(),
});

This makes the App server span a true trace root. Joining browser sessions to backend traces is an explicitly deferred decision; app → api → rag is unaffected, because that context comes from the active server span, not from inbound extraction.

Leg 2 — App → API (manual injection, always sampled)

When the app calls the Go API through the @huddlesurety/api SDK, the SDK’s server-side fetcher (src/trpc/api.ts) manually injects the active trace context into the outbound headers:

propagation.inject(context.active(), headers, {
  set: (h, k, v) => h.set(k, v),
});

Two hard-won details make this reliable — both were real bugs:

• Manual inject, not automatic. @vercel/otel’s automatic fetch propagation is unreliable here: the SDK passes a Request as the fetch input while the app supplies a fresh headers object in init, and the auto-injected header gets dropped on the wire. Injecting into the app’s own headers is deterministic regardless of that merge.
• AlwaysOnSampler, not ParentBased. In the app’s Next.js runtime, the span active in an outbound fetch’s context is often a Next-internal span with traceFlags=0. A ParentBased sampler would defer to that flag and inject a …-00 (unsampled) traceparent, which the API’s own sampler then drops — silently breaking the join. AlwaysOnSampler guarantees every injected traceparent is sampled.

On the receiving side, the API sets a global W3C propagator so its echo-opentelemetry middleware extracts the incoming traceparent instead of starting fresh (internal/o11y/otel.go):

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
))

Without SetTextMapPropagator the API silently starts a new root

OpenTelemetry’s global default is a no-op propagator. If the API doesn’t install a real one, it ignores the caller’s traceparent and roots a brand-new trace — so app and API spans land in two disconnected traces. This one line is load-bearing for the whole app → api → rag chain.

Leg 3 — API → workers (across the Temporal boundary)

This hop is the hard one: the API doesn’t call its workers over HTTP — it starts a Temporal workflow, and a worker (RAG or Notification) picks it up later. HTTP header propagation doesn’t apply, and crucially W3C baggage does not cross the Temporal boundary. Two things travel across it instead, both via the Temporal workflow/activity header:

1. Trace context — carried by Temporal’s OpenTelemetry interceptor. The API dials the Temporal client with that interceptor (and the same TraceContext propagator above), so the workflow it starts continues the request’s trace.

2. Requester identity (org.id) — carried by a custom Temporal ContextPropagator, IdentityPropagator (internal/o11y/temporal_propagator.go). Registering it on the client means every workflow the API starts while handling an authenticated request carries the caller’s org, with no change to any workflow signature:

// Header key carrying the requester's org across the Temporal boundary.
// MUST match byte-for-byte the key the rag worker reads.
const headerOrgID = "x-huddle-org-id"

func (p *IdentityPropagator) Inject(ctx context.Context, w workflow.HeaderWriter) error {
    _, orgID, err := utils.GetUserFromContext(ctx)
    if err != nil {
        return nil   // unauth / key-auth path — nothing to propagate
    }
    return p.setHeader(w, headerOrgID, orgID.String())
}

The value is the org’s ULID, encoded with Temporal’s default (JSON) data converter so a Go-written string decodes cleanly in Python.

RAG receives and re-propagates it

RAG’s Python worker (app/o11y/identity.py) can’t run the Go propagator, so it carries identity forward itself, using worker interceptors:

1. The inbound workflow interceptor reads x-huddle-org-id off the incoming workflow header.

2. The outbound interceptor re-injects it onto every downstream call — child workflows and activities — so the chain survives past the first hop (mirroring the Go propagator’s InjectFromWorkflow).

3. The activity interceptor reads it back and stamps org.id (and workflow.type) onto the current OTel span, and sets a current_org_id context var for the activity’s duration.

The interceptor is registered after Temporal’s TracingInterceptor, so the OTel span is already current when identity tagging runs. See RAG › Distributed tracing & identity.

Notification joins the same way

The Notification worker (TypeScript) is the second consumer of this leg and uses the same mechanism: OTel workflow interceptors baked into its workflow bundle continue the API’s trace, an UndiciInstrumentation emits the CLIENT spans that form the Notification → API service-graph edge, and an inbound activity interceptor reads the same x-huddle-org-id header to stamp org.id + workflow.type. It only needs the inbound side — it spawns no child workflows, so there’s nothing to re-propagate outbound. See Notification › Observability.

Identity on spans: org, not user

Across all three services, traces are tagged with org.id only — never the user ID. This is a deliberate, consistent choice: org is the dimension we slice traces and dashboards by, and user ID would add cardinality with no consumer. Each tier tags it the same way:

• App — tagSpanWithIdentity() (src/lib/trace.ts) reads the org from the session JWT and sets org.id on the active server span (wrapped in React cache() so a fan-out render verifies the token once per request). Best-effort — no span or no session simply no-ops.
• API — the JWT middleware sets org.id on the request span; the IdentityPropagator carries it onward.
• RAG — the activity interceptor stamps org.id + workflow.type on activity spans.

org.id has no standard OTel semantic convention — the attribute key is the same across services by agreement.

Why traces export on a short interval

Each service flushes its span batches on a ~5-second timer, not the default minute (OTEL_BSP_SCHEDULE_DELAY=5000 in the app; matching batch timeouts in the API and RAG). With a 60-second batch, the app/api/rag spans of one trace could land up to a minute apart, so Grafana would show a partial, rootless tree until the slowest batch arrived. Metrics stay on the longer ~60s interval — only traces need to assemble quickly.

Only a sample of traces is stored

Alloy tail-samples traces bound for Tempo — it keeps every error trace, every trace ≥ 1s, and 10% of the rest, deciding per whole trace so a kept trace retains all its legs. A perfectly healthy, fast request may therefore simply not be in Tempo — that’s expected, not a broken trace. Span metrics and the service graph are computed from the unsampled stream, so they still count every request. See Observability › Export policy.

Debugging a broken trace

When spans don’t connect, walk the legs in order:

• App server span is rootless / “root span not yet received” — expected for browser-origin traces (Leg 1 is inject-only by design). A backend-rooted trace that’s rootless points at Leg 2.
• App and API in separate traces — the traceparent isn’t reaching the API, or reaching it as …-00. Check the fetcher’s manual inject (Leg 2) and that the API installed a non-no-op SetTextMapPropagator.
• API and a worker in separate traces — the Temporal OTel interceptor isn’t registered on the client, or the workflow header isn’t carrying context (Leg 3). For Notification specifically, also suspect a stale workflow bundle — the OTel workflow interceptors are baked in at bun run build time, so an un-rebuilt bundle drops trace context even when the API side is correct.
• Trace connected but not filterable by org — the IdentityPropagator / RAG interceptors aren’t stamping org.id; confirm the x-huddle-org-id header key matches byte-for-byte on both sides.