airllm vs colibri
Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families. — versus — Pure-C, zero-dep MoE runtime that runs GLM-5.2 (744B) on a 25GB-RAM consumer box by streaming experts from disk — VRAM/RAM/NVMe as one tiered hierarchy, never touching precision.
Same job — giant models on tiny hardware — opposite technique: AirLLM streams dense layers through a 4GB GPU from Python/HF, colibrì streams MoE experts from NVMe in pure C with a workload-learning pin cache.
| airllm | colibri | |
|---|---|---|
| Stars | 24k | 17k |
| Forks | 2.7k | 1.5k |
| Language | Jupyter Notebook | C |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 4 days ago | today |
| Topics | local | local |
| Curated connections | 4 | 3 |
airllm — the curator's take
The trick is elegant and the tradeoff is brutal, and you should know both: only one layer lives on the GPU at a time, so VRAM scales with layer size instead of model size — that's how 671B fits on a hobbyist card — but every token streams the whole model from disk, so generation runs at seconds-per-token. Use it for batch/offline jobs where 'it fits' beats 'it's fast', for poking at frontier-scale open models on hardware you own, or with block-wise 4/8-bit compression for a ~3x claw-back. NOT a chat or serving solution: Ollama is the fits-in-VRAM daily driver, vLLM the throughput server. Apple-silicon Macs supported via MLX. README carries the author's sponsor/affiliate links — the library stands on its own.
colibri — the curator's take
Use it when you want a frontier-scale open MoE (GLM-5.2, 744B) on hardware that can't hold it, and you care about fidelity: forward pass is token-exact against the transformers oracle, and placement only ever changes speed, never semantics. The learning cache pins the experts YOUR workload actually routes, so it genuinely gets faster with use. NOT a general runner — it's a one-model engine; for everyday multi-model local use pick ollama. And expect 0.05–6.8 tok/s depending on tier residency: this is patience-ware for correctness fanatics, not latency-sensitive serving.