vllm-project

vLLM

High-throughput, memory-efficient inference and serving engine for LLMs.

★ 85,485⑂ 19,022● Python⚖ Apache-2.0updated today

README.md

vLLM is a fast inference and serving engine for LLMs, using PagedAttention for high throughput — the production choice when you outgrow single-request local inference.

pip install vllm

Continue your stack

What teams reach for next — and why each earns a place beside vLLM.

Alternative to

Ollama

Run Llama, Mistral and other open models locally with a single command and a clean API.

Why it fitsBoth serve open models locally; vLLM optimizes for throughput, Ollama for one-command simplicity.

Built with / on

olmocr

Open toolkit that linearizes messy PDFs — scans, tables, equations, handwriting — into clean ordered Markdown with a self-hosted vision-language model. Built for LLM training data and RAG ingestion.

Why it fitsolmocr runs its OCR vision-language model through a high-throughput inference backend — vLLM (or SGLang) — to batch-process PDFs at scale, so vLLM is the serving engine under olmocr's pipeline.