StackMap
Subscribe
Explore / respan-gateway
respanai

respan-gateway

Rust LLM gateway in one binary: any OpenAI/Anthropic/Gemini client to any model, cross-vendor realtime voice, MCP tools inside chat calls, hedged requests and a live cost log.

954 145 Rust Apache-2.0updated today
View on GitHubDispute this mapping →
Curator's take

Pick it when gateway overhead matters (sub-0.1ms, tiny memory) or you need protocol translation beyond chat — realtime voice across vendors, MCP tools usable by any model, a plugin trait for custom guardrails. Young and tied to the Respan tracing product for deep observability; if you need team keys, budgets and a mature provider list today, LiteLLM or Otari are safer.

Mapped by ShipWithAI editors · links verified

Continue your stack

What teams reach for next — and why each earns a place beside respan-gateway. Ranked by curator confidence.

alternativealternativealternativealternativelitellmotariOmniRoutemcp-context-forgerespan-gateway
pairs wellalternativebuilt withpick a node for the why · open it from the panel
Weekly digest
README.md2 min read
Respan Gateway

Respan Gateway: open-source LLM gateway

The universal adapter for AI.

License: MIT OR Apache-2.0 CI Latest release crates.io PyPI Discord

Respan Gateway is an open-source LLM gateway: a single Rust binary that sits between your code and every AI API. Point the SDK you already use at it, and it connects you to everything: chat models, images, video and voice from every major provider; tools for any model, from web search and memory to code sandboxes and any MCP server; and traffic control (format translation, fallback, retries and hedged requests), with a live log of every call. ▶ Watch the demo


Your app plugs into Respan Gateway, which connects it to models, voice, image and video, tools, embeddings and self-hosted models, with fallback, hedging, translation, tool loop, PII redaction and tracing built in

[!TIP] Starring the repo helps more developers find Respan Gateway.

LLM gateway features

  • Any client, any model
    OpenAI Chat Completions, OpenAI Responses, Anthropic Messages and Gemini API clients can all call any model, with streaming, tools, images and reasoning translated.

  • Multimodal
    Realtime voice across vendors (an OpenAI Realtime client talks to Gemini Live, and back), plus images, speech, transcription, embeddings and video through OpenAI's or Gemini's APIs, to any vendor that has them.

  • MCP gateway and tools for every model
    Put MCP servers or built-in tool vendors behind the gateway, from web search to code sandboxes. It runs their tools inside chat requests, so a plain chat call can use them, whatever the client or model.

  • Traffic control
    Named routes, globs and provider/model; fallback, retries, cheapest or fastest; and hedged requests that race a slow call against a second provider.

  • Observability
    A live request log with cost, latency and tokens for every call, and traces in Respan, where span-02 analyzes how your agents behave.

  • Best-in-class performance
    0.07 ms of added latency, 56,000 requests per second and 31 MB of memory in our benchmarks: about 3× the throughput of the next fastest gateway we tested.

  • Write your own plugin in minutes
    Every feature above is a plugin, built on the same public Rust API you can use: one trait with four hooks. Start from the plugin guide or an example such as guardrails, PII masking o