FreeToken vs vLLM
Edge-native MoE serving engine: bandwidth-adaptive CPU-GPU co-execution, global LRU expert caching and elastic VRAM run 290B+ frontier MoE models on a gaming PC at interactive speed. — versus — High-throughput, memory-efficient inference and serving engine for LLMs.
vLLM is the datacenter default: many GPUs, many concurrent users, model fits in VRAM. FreeToken targets the opposite corner — one person, one box, weights far larger than the VRAM.
| FreeToken | vLLM | |
|---|---|---|
| Stars | 4.3k | 90k |
| Forks | 388 | 21k |
| Language | Python | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | today | today |
| Topics | local | local |
| Curated connections | 8 | 17 |
FreeToken — the curator's take
The research-grade answer to 'frontier MoE on one consumer box' — Berkeley/MIT-flavored author list, an arXiv paper behind it, and the two ideas that matter for agents: a q* policy that splits work across GPU, CPU and host memory by available bandwidth, and semantic anchor checkpoints so tool calls and thinking-block edits don't force full context recompute. Ships Anthropic/OpenAI-compatible APIs, so Claude Code, Codex or OpenCode can point straight at it, plus a desktop GUI. Constraints are real: NVIDIA RTX 30/40/50 on Windows or Linux, MoE architectures only, and a paper-fresh engine that moves fast — pin versions. Serving a dense 8B on a proper GPU? This is the wrong tool.
vLLM — the curator's take
High-throughput, memory-efficient inference and serving engine for LLMs.