airllm vs FreeToken
Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families. — versus — Edge-native MoE serving engine: bandwidth-adaptive CPU-GPU co-execution, global LRU expert caching and elastic VRAM run 290B+ frontier MoE models on a gaming PC at interactive speed.
airLLM slices dense models layer by layer to fit tiny GPUs; FreeToken exploits MoE sparsity instead — narrower model coverage, dramatically better tokens per second on the models it does support.
| airllm | FreeToken | |
|---|---|---|
| Stars | 32k | 4.3k |
| Forks | 3.4k | 388 |
| Language | Jupyter Notebook | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | today | today |
| Topics | local | local |
| Curated connections | 6 | 8 |
airllm — the curator's take
The trick is elegant and the tradeoff is brutal, and you should know both: only one layer lives on the GPU at a time, so VRAM scales with layer size instead of model size — that's how 671B fits on a hobbyist card — but every token streams the whole model from disk, so generation runs at seconds-per-token. Use it for batch/offline jobs where 'it fits' beats 'it's fast', for poking at frontier-scale open models on hardware you own, or with block-wise 4/8-bit compression for a ~3x claw-back. NOT a chat or serving solution: Ollama is the fits-in-VRAM daily driver, vLLM the throughput server. Apple-silicon Macs supported via MLX. README carries the author's sponsor/affiliate links — the library stands on its own.
FreeToken — the curator's take
The research-grade answer to 'frontier MoE on one consumer box' — Berkeley/MIT-flavored author list, an arXiv paper behind it, and the two ideas that matter for agents: a q* policy that splits work across GPU, CPU and host memory by available bandwidth, and semantic anchor checkpoints so tool calls and thinking-block edits don't force full context recompute. Ships Anthropic/OpenAI-compatible APIs, so Claude Code, Codex or OpenCode can point straight at it, plus a desktop GUI. Constraints are real: NVIDIA RTX 30/40/50 on Windows or Linux, MoE architectures only, and a paper-fresh engine that moves fast — pin versions. Serving a dense 8B on a proper GPU? This is the wrong tool.