airllm vs vLLM
Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families. — versus — High-throughput, memory-efficient inference and serving engine for LLMs.
Opposite ends of the local-inference spectrum: vLLM maximizes throughput given abundant VRAM (production serving); AirLLM minimizes VRAM given abundant patience (frontier-size models on consumer cards).
| airllm | vLLM | |
|---|---|---|
| Stars | 23k | 86k |
| Forks | 2.6k | 19k |
| Language | Jupyter Notebook | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 4 days ago | today |
| Topics | local | local |
| Curated connections | 2 | 11 |
airllm — the curator's take
The trick is elegant and the tradeoff is brutal, and you should know both: only one layer lives on the GPU at a time, so VRAM scales with layer size instead of model size — that's how 671B fits on a hobbyist card — but every token streams the whole model from disk, so generation runs at seconds-per-token. Use it for batch/offline jobs where 'it fits' beats 'it's fast', for poking at frontier-scale open models on hardware you own, or with block-wise 4/8-bit compression for a ~3x claw-back. NOT a chat or serving solution: Ollama is the fits-in-VRAM daily driver, vLLM the throughput server. Apple-silicon Macs supported via MLX. README carries the author's sponsor/affiliate links — the library stands on its own.
vLLM — the curator's take
High-throughput, memory-efficient inference and serving engine for LLMs.