litellm vs respan-gateway
Open-source AI gateway: call 100+ LLM providers in OpenAI format via a Python SDK or self-hosted proxy — with cost tracking, virtual keys, guardrails, load balancing and logging. — versus — Rust LLM gateway in one binary: any OpenAI/Anthropic/Gemini client to any model, cross-vendor realtime voice, MCP tools inside chat calls, hedged requests and a live cost log.
Same job — one endpoint over every provider; LiteLLM is the Python incumbent with the widest provider coverage, Respan is a Rust binary built for throughput with realtime voice and MCP tools in the gateway.
| litellm | respan-gateway | |
|---|---|---|
| Stars | 64k | 954 |
| Forks | 13k | 145 |
| Language | Python | Rust |
| License | MIT | Apache-2.0 |
| Last activity | today | today |
| Topics | gateway | gateway |
| Curated connections | 24 | 4 |
litellm — the curator's take
The default answer to "one API for every LLM." Reach for it when app code or an agent fleet must hit many providers without per-SDK glue, or when a team needs a central proxy with spend caps, virtual keys and logging. The SDK is a thin drop-in; the proxy is the real value (dashboard, budgets, rate limits). NOT an inference engine — it routes to backends like vLLM/Ollama, doesn't run models. Overkill if you only ever call one provider. Guardrails/evals exist but are lighter than dedicated tools.
respan-gateway — the curator's take
Pick it when gateway overhead matters (sub-0.1ms, tiny memory) or you need protocol translation beyond chat — realtime voice across vendors, MCP tools usable by any model, a plugin trait for custom guardrails. Young and tied to the Respan tracing product for deep observability; if you need team keys, budgets and a mature provider list today, LiteLLM or Otari are safer.