airllm vs Model-Optimizer
Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families. — versus — NVIDIA's model-compression library: quantization (PTQ/QAT, FP8/NVFP4), pruning, distillation, NAS and speculative decoding over HF/PyTorch/ONNX, exported to TensorRT-LLM, vLLM and SGLang.
Both make an oversized model fit on hardware it shouldn't. AirLLM streams layers and explicitly refuses to quantize, preserving quality at the cost of latency; ModelOpt compresses the weights and keeps throughput.
| airllm | Model-Optimizer | |
|---|---|---|
| Stars | 33k | 3.7k |
| Forks | 3.5k | 574 |
| Language | Jupyter Notebook | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | today | today |
| Topics | local | training, local |
| Curated connections | 7 | 7 |
airllm — the curator's take
The trick is elegant and the tradeoff is brutal, and you should know both: only one layer lives on the GPU at a time, so VRAM scales with layer size instead of model size — that's how 671B fits on a hobbyist card — but every token streams the whole model from disk, so generation runs at seconds-per-token. Use it for batch/offline jobs where 'it fits' beats 'it's fast', for poking at frontier-scale open models on hardware you own, or with block-wise 4/8-bit compression for a ~3x claw-back. NOT a chat or serving solution: Ollama is the fits-in-VRAM daily driver, vLLM the throughput server. Apple-silicon Macs supported via MLX. README carries the author's sponsor/affiliate links — the library stands on its own.
Model-Optimizer — the curator's take
The default answer to 'make this checkpoint cheaper to serve' if you deploy on NVIDIA hardware — the FP8/NVFP4 recipes here are the ones TensorRT-LLM and vLLM actually consume, and QAT recovers accuracy that naive post-training quantization throws away. It is deliberately vendor-shaped: the interesting formats want Hopper or Blackwell, and it optimizes checkpoints rather than serving them, so you still need an engine underneath. Wrong tool for training from scratch, for CPU or Apple-silicon targets, or for anyone who just wants a GGUF.