BigMoeOnEdge vs FreeToken
Run MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output. — versus — Edge-native MoE serving engine: bandwidth-adaptive CPU-GPU co-execution, global LRU expert caching and elastic VRAM run 290B+ frontier MoE models on a gaming PC at interactive speed.
bigmoeonedge keeps always-needed weights resident and streams each token's experts for byte-identical CPU-only output, even on phones; FreeToken assumes a consumer GPU and trades that purity for interactive throughput.
| BigMoeOnEdge | FreeToken | |
|---|---|---|
| Stars | 397 | 4.3k |
| Forks | 41 | 388 |
| Language | C++ | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 7 days ago | today |
| Topics | local | local |
| Curated connections | 6 | 8 |
BigMoeOnEdge — the curator's take
The clever insight is that a Mixture-of-Experts model already refuses to use most of itself per token, so flash storage can be part of the memory hierarchy instead of a swap disaster. Built on llama.cpp's public API rather than as a fork, so every quantization, tokenizer and chat template works because llama.cpp is still doing that part, and following upstream is a submodule bump. The underrated case isn't the 284B stunt: it's the 18-22 GB model on a 12 GB phone, where ordinary mmap becomes a fault storm that kills other apps, and streaming turns it into a stable 5 tok/s. Set expectations: ~1 tok/s at the extreme, CPU-only with no GPU or NPU path, MoE architectures only, and a 91 GB download is still a 91 GB download.
FreeToken — the curator's take
The research-grade answer to 'frontier MoE on one consumer box' — Berkeley/MIT-flavored author list, an arXiv paper behind it, and the two ideas that matter for agents: a q* policy that splits work across GPU, CPU and host memory by available bandwidth, and semantic anchor checkpoints so tool calls and thinking-block edits don't force full context recompute. Ships Anthropic/OpenAI-compatible APIs, so Claude Code, Codex or OpenCode can point straight at it, plus a desktop GUI. Constraints are real: NVIDIA RTX 30/40/50 on Windows or Linux, MoE architectures only, and a paper-fresh engine that moves fast — pin versions. Serving a dense 8B on a proper GPU? This is the wrong tool.