LLM Gateway

Every LLM and embedding call Huddle makes goes through a self-hosted LiteLLM proxy at litellm.huddlesurety.co. Services never hold provider credentials: they authenticate to the gateway with a virtual key, and the gateway holds the Google AI Studio key and does the routing, accounting, and rate limiting.

graph LR
    RAG[RAG worker] -->|virtual key| GW[LiteLLM gateway]
    DEV[Developer laptop] -->|personal key| GW
    GW -->|provider key| G[Google AI Studio]
    GW --> PG[(Postgres — keys, teams, spend)]

One instance serves all environments. That matters for attribution: the gateway’s own identity is fixed, so which environment a call came from has to be carried by the caller’s credentials rather than inferred from the proxy.

Connecting a service

Two variables, both per-environment:

Variable	Value
LITELLM_PROXY_API_BASE	https://litellm.huddlesurety.co
LITELLM_PROXY_API_KEY	that service’s virtual key for that environment

RAG sets litellm.use_litellm_proxy = True, so ordinary litellm.completion and litellm.embedding calls route through the gateway with no per-call-site change. Any OpenAI-compatible client works the same way.

Model routing

Models are configured as a gemini/* wildcard, so any Gemini model is reachable without being listed. check_provider_endpoint: true makes /v1/models return the resolved catalogue instead of the literal gemini/*.

Two naming traps

Use the gemini/ prefix — that’s Google AI Studio, which needs only an API key. google/ is OpenRouter’s naming (a leftover from the pre-gateway SDK) and vertex_ai/ would require full GCP service-account credentials.

Because the wildcard doesn’t validate model names, a typo now 404s at Google rather than 400ing at the proxy, and a model newer than the pinned image’s price map logs $0 spend.

Teams and keys

The convention encodes two axes without duplicating either:

Primitive	Carries	Example
Team	environment	development, staging, production
Service key	which service, in which environment	rag-development, rag-production
Developer key	which service, for which person	rag-ethan

• Team is the environment. The LiteLLM UI renders the team alias as the Env: badge on every request, and team-level budgets are how one environment is stopped from spending another’s allowance.
• A key is a credential, never a reporting dimension. Keys rotate; when they do, the old key’s spend history stays behind under a hash that no longer resolves to an alias. Anything you need to chart over time should come from the team, the customer, or a tag instead.
• Key aliases are globally unique across the whole gateway, so rag-development and rag-ethan can’t collide with anything else.

Adding a service is one key per environment plus nothing else — no new teams, no scheme change.

Local development

Every developer gets their own key per service, named <service>-<your name> — rag-ethan, api-jane. Personal keys make spend attributable per person and revocable individually, and they keep a laptop experiment from being indistinguishable from deployed traffic.

1. Open litellm.huddlesurety.co and sign in with your Huddle Google Workspace account (SSO — the UI redirects automatically).
2. Create a virtual key on the development team, with the alias <service>-<your name>.
3. Put it in that service’s .env.local as LITELLM_PROXY_API_KEY.

Why .env.local

mise run env overwrites .env.json from Railway, which carries the shared deployed key. .env.local is loaded after it, so a personal key set there wins and survives the next env refresh.

Developers are members of the development team only, so there is no path to minting a staging or production key by accident. Personal keys are a good place for a small max_budget and rpm_limit — a runaway loop then costs you a 429 rather than the environment’s budget.

Spend attribution

Four things identify every call, and none of them are derived from the others:

Dimension	Source	Read it via
Environment	team	UI Env: badge, per-team spend
Service or developer	key alias	key spend views
Tenant (org)	x-litellm-customer-id header	/customer/info, /customer/list
Workflow	x-litellm-tags: workflow:<Type> header	/spend/tags, /tag/daily/activity

Headers, not body fields

Send the org and tag as headers. The OpenAI-compatible user body field also registers a customer, but Gemini rejects it on embedding calls (litellm.UnsupportedParamsError) — and embeddings are the highest-volume path in RAG, so it would fail exactly where it matters most.

LiteLLM also auto-tags by User-Agent, so /spend/tags mixes workflow:… entries with User-Agent: OpenAI/Python 2.50.0. Filter on the prefix you want.

Customer display names are set with alias on /customer/new — never format a name into the customer id itself, because spend is keyed on that exact string and a rename would silently start a second customer.

Never set a default customer budget

A default customer budget applies to customers that don’t exist in the database yet, which means it throttles every new organization on its first request.

Observability

Gateway telemetry deliberately does not flow into Alloy, Tempo, or Prometheus. It lives in LiteLLM’s own dashboard and API — spend by team, key, tag, customer, and model — rather than being duplicated into a second system.

The trade-off is that distributed traces end at RAG instead of continuing into the provider call, so gateway latency can’t be correlated with a bond request in Tempo. LiteLLM’s Prometheus callback, which would be the other way to export metrics, is an Enterprise-only feature.

Prompt and completion bodies are stored in the gateway’s own Postgres (store_prompts_in_spend_logs) and nowhere else — nothing exports them to the trace store.

Deployment

Self-hosted on Railway from the litellm repo, in the Infrastructure project alongside Grafana and Temporal, deploying from main. The image pins an exact LiteLLM version and bakes config.yaml in, so configuration changes ship like code. Postgres holds keys, teams, customers, and spend logs.

ObservabilityMetrics, logs, and dashboards for the services themselves.