README.md
vLLM is a fast inference and serving engine for LLMs, using PagedAttention for high throughput — the production choice when you outgrow single-request local inference.
pip install vllm
Explore / vLLM
vllm-project
vLLM
High-throughput, memory-efficient inference and serving engine for LLMs.
89,493 20,944 Python Apache-2.0updated today
Continue your stack
What teams reach for next — and why each earns a place beside vLLM. Ranked by curator confidence.
Pairs well with
litellm
Open-source AI gateway: call 100+ LLM providers in OpenAI format via a Python SDK or self-hosted proxy — with cost tracking, virtual keys, guardrails, load balancing and logging.
Why it fitsLiteLLM fronts a self-hosted vLLM server as one more OpenAI-compatible backend — vLLM does the serving, LiteLLM does routing, keys and spend tracking.
speech-to-speech
Modular local voice-agent pipeline — VAD→STT→LLM→TTS behind an OpenAI Realtime-compatible WebSocket; every stage swappable, the LLM slot takes any OpenAI-compatible server. Powers Reachy Mini robots.
Why it fitsThe README's own recipe for a fully local stack: point the OpenAI-compatible LLM slot at a vLLM server and no token leaves your network.
LlamaFactory
The unified fine-tuning framework: 100+ LLMs and VLMs via LoRA/QLoRA/full-parameter, config-driven or through the LlamaBoard GUI. ACL 2024, 1000+ citations, 73k stars.
Why it fitsFine-tune here, serve fast there — LlamaFactory ships a vLLM backend for high-throughput inference on the models it trains.
Alternative to See all + compare →
Ollama
Run Llama, Mistral and other open models locally with a single command and a clean API.
Why it fitsBoth serve open models locally; vLLM optimizes for throughput, Ollama for one-command simplicity.
sie
Self-hosted inference cluster for everything agents call besides the big LLM: embeddings, rerankers, OCR, NER, guardrails and small LLMs — 100+ models, one OpenAI-compatible API, K8s stack included.
Why it fitsBoth self-hosted OpenAI-compatible inference servers, opposite shapes: vLLM maximizes throughput for one big LLM; SIE serves breadth — 100+ heterogeneous task models (embedders, rerankers, OCR, NER, guards) loaded on demand across a cluster.
airllm
Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families.
Why it fitsOpposite ends of the local-inference spectrum: vLLM maximizes throughput given abundant VRAM (production serving); AirLLM minimizes VRAM given abundant patience (frontier-size models on consumer cards).
Built with / on
LMCache
KV-cache layer for scalable LLM serving: offload and reuse KV across GPU/CPU/disk/remote tiers to cut TTFT and prefill cost. vLLM-first; used by NVIDIA Dynamo and llm-d.
Why it fitsLMCache plugs into vLLM as its KV-connector; vLLM is the primary serving engine it accelerates.
llm-d
Distributed inference stack for Kubernetes from Red Hat, Google and IBM (CNCF) — prefix-cache-aware routing, tiered KV-cache, prefill/decode disaggregation and SLO autoscaling above vLLM/SGLang.
Why it fitsllm-d is explicitly the orchestration layer above model servers: vLLM does the on-accelerator inference, llm-d adds cluster-level routing, KV-cache management, disaggregation and autoscaling.
production-ocr-course
Runnable course: a production OCR pipeline on Kubernetes — Rust ingestion, Qwen 3.5 (4B) served by vLLM at 1.86 pages/s, Redis queues, KEDA autoscaling. Deploy it on AKS/GKE, not a notebook.
Why it fitsThe serving layer of the whole course — Qwen 3.5 runs on vLLM's continuous batching, PagedAttention and multi-token prediction to hit its 1.86 pages/second.