airllm vs mesh-llm
Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families. — versus — Distributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.
Opposite answers to 'the model doesn't fit': AirLLM streams layers from disk on one small GPU (slow, solo), mesh-llm splits stages across peers' GPUs (faster, needs friends).
| airllm | mesh-llm | |
|---|---|---|
| Stars | 23k | 2.7k |
| Forks | 2.6k | 322 |
| Language | Jupyter Notebook | Rust |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 7 days ago | today |
| Topics | local | local |
| Curated connections | 4 | 4 |
airllm — the curator's take
The trick is elegant and the tradeoff is brutal, and you should know both: only one layer lives on the GPU at a time, so VRAM scales with layer size instead of model size — that's how 671B fits on a hobbyist card — but every token streams the whole model from disk, so generation runs at seconds-per-token. Use it for batch/offline jobs where 'it fits' beats 'it's fast', for poking at frontier-scale open models on hardware you own, or with block-wise 4/8-bit compression for a ~3x claw-back. NOT a chat or serving solution: Ollama is the fits-in-VRAM daily driver, vLLM the throughput server. Apple-silicon Macs supported via MLX. README carries the author's sponsor/affiliate links — the library stands on its own.
mesh-llm — the curator's take
The 'LLM for the people' play: friends or a homelab pool mid-range GPUs and serve models none of them could run alone, with public meshes discoverable via Nostr. The Skippy stage-split design is genuinely clever. Experimental distributed systems — expect rough edges, and never treat a public mesh as private infrastructure. One box that fits your model? Just run Ollama.