llm-proxy

OpenAI-compatible LLM privacy proxy. Sits between clients and upstream LLM providers — strips fingerprinting headers, randomizes user-agents and Accept-Language, scrubs PII, and enforces ephemeral-by-default storage so upstream providers learn as little as possible about who is talking to them and what they're saying.

Philosophy: "You don't have to protect what you do not have."

Extracted from the REDACTED swarm monorepo (redacted-proxy) into a standalone service. Any OpenAI-compatible client works — point it at the proxy's base URL with the bearer token.

---

Features

• OpenAI-compatible POST /v1/chat/completions, including SSE streaming (stream: true). Anthropic upstreams are transparently translated to/from the Messages API, chunks included.
• Multi-provider routing by model alias or prefix: xAI, Groq, Anthropic, OpenAI, Venice, and local Ollama (keyless, via OLLAMA_URL).
• Automatic failover — if the primary provider has no key, rate-limits, or 5xxes, the request is retried against equivalent models on other providers. The response carries X-Failover-From when this happens. Pinning with X-Provider disables failover.
• Privacy modes (anonymous / private / maximum / zero / tee / e2ee) controlling logging, PII scrubbing, and retention.
• Hot-reloadable config via POST /config — no redeploy.
• Cost tracking with per-provider pricing tables and real upstream token usage when available.

Endpoints

Method	Path	Auth	Description
POST	/v1/chat/completions	Bearer token	Main proxy — OpenAI-compatible, streaming supported
GET	/v1/models	None	List available model aliases
GET	/health	None	Liveness + provider key status
GET	/privacy	None	Current privacy mode, guarantees, storage policy
GET	/logs?n=100	Bearer token	Recent proxy log (in-memory ring)
GET	/config	Bearer token	Current runtime config
POST	/config	Bearer token	Hot-update config (no redeploy needed)

Provider routing

By model alias (see proxy/providers.py for the full table) or prefix:

Pattern	Provider
grok-*	xAI
llama-*, gemma*, qwen-*, deepseek-*	Groq
claude-*	Anthropic
gpt-*	OpenAI
Venice exact names (venice-uncensored, deepseek-v4-pro, …)	Venice
X-Provider: ollama header	Local Ollama

Friendly aliases track the current model generation: claude-opus → claude-opus-4-8, claude-sonnet → claude-sonnet-5, gpt-5.6 → gpt-5.6-sol, etc. Decommissioned upstream models (Groq mixtral-8x7b, gemma2-9b-it; xAI grok-3*) are remapped to their nearest current equivalents for backward compatibility.

Privacy architecture

Every request: 30+ fingerprinting headers stripped, synthetic randomized User-Agent + Accept-Language, optional PII regex scrub (IDs, @handles, emails, cards, phones), client Authorization never forwarded. Send X-Ephemeral: true to skip all logging for one request. Responses carry X-Privacy-Mode.

Mode	Disk log	Ring TTL	PII scrub
anonymous	opt-in (default on)	unlimited	opt-in
private (default)	off by default	1 hour	on
maximum / zero	never	5 min	forced
tee / e2ee (future)	never	5 min	forced

GET /privacy reports the active guarantees in full.

Running

pip install -r requirements.txt
cp .env.example .env   # fill in PROXY_TOKEN + provider keys
python main.py

Docker:

docker build -t llm-proxy .
docker run --env-file .env -p 7080:7080 llm-proxy

Railway: deploys as-is via railway.toml / Procfile (python main.py).

Client usage

curl http://localhost:7080/v1/chat/completions \
  -H "Authorization: Bearer $PROXY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-sonnet", "messages": [{"role": "user", "content": "hi"}], "stream": true}'

Optional headers: X-Provider (pin provider, disables failover), X-Temperature, X-Top-P, X-Ephemeral.

Environment variables

See .env.example. Highlights: PROXY_TOKEN (required), one or more provider keys (XAI_API_KEY, GROQ_API_KEY, ANTHROPIC_API_KEY, OPENAI_API_KEY, VENICE_API_KEY), OLLAMA_URL, PRIVACY_MODE, RATE_LIMIT_RPM, and REDIS_URL (optional liveness heartbeat — disabled when unset).

Development

pip install -r requirements-dev.txt
pytest
ruff check proxy/ main.py tests/

Layout: thin main.py entry point; implementation in proxy/ (config, privacy, providers, logging_ring, handlers, app). CI runs pytest + ruff on Python 3.9 and 3.12; the code stays 3.9-compatible.