StackMap
Subscribe

litellm vs respan-gateway

Open-source AI gateway: call 100+ LLM providers in OpenAI format via a Python SDK or self-hosted proxy — with cost tracking, virtual keys, guardrails, load balancing and logging. — versus — Rust LLM gateway in one binary: any OpenAI/Anthropic/Gemini client to any model, cross-vendor realtime voice, MCP tools inside chat calls, hedged requests and a live cost log.

The curated verdict

Same job — one endpoint over every provider; LiteLLM is the Python incumbent with the widest provider coverage, Respan is a Rust binary built for throughput with realtime voice and MCP tools in the gateway.

litellmrespan-gateway
Stars64k954
Forks13k145
LanguagePythonRust
LicenseMITApache-2.0
Last activitytodaytoday
Topicsgatewaygateway
Curated connections244

litellm — the curator's take

The default answer to "one API for every LLM." Reach for it when app code or an agent fleet must hit many providers without per-SDK glue, or when a team needs a central proxy with spend caps, virtual keys and logging. The SDK is a thin drop-in; the proxy is the real value (dashboard, budgets, rate limits). NOT an inference engine — it routes to backends like vLLM/Ollama, doesn't run models. Overkill if you only ever call one provider. Guardrails/evals exist but are lighter than dedicated tools.

respan-gateway — the curator's take

Pick it when gateway overhead matters (sub-0.1ms, tiny memory) or you need protocol translation beyond chat — realtime voice across vendors, MCP tools usable by any model, a plugin trait for custom guardrails. Young and tied to the Respan tracing product for deep observability; if you need team keys, budgets and a mature provider list today, LiteLLM or Otari are safer.