BigMoeOnEdge vs colibri
Run MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output. — versus — Pure-C, zero-dep MoE runtime that runs GLM-5.2 (744B) on a 25GB-RAM consumer box by streaming experts from disk — VRAM/RAM/NVMe as one tiered hierarchy, never touching precision.
The same trick aimed at different hardware: Colibri is a pure-C MoE runtime streaming experts from NVMe to fit a 744B model on a 25 GB desktop; BigMoeOnEdge targets phones on plain CPU and rides llama.cpp's format support instead of its own runtime.
| BigMoeOnEdge | colibri | |
|---|---|---|
| Stars | 395 | 26k |
| Forks | 41 | 2.8k |
| Language | C++ | C |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 7 days ago | 4 days ago |
| Topics | local | local |
| Curated connections | 3 | 4 |
BigMoeOnEdge — the curator's take
The clever insight is that a Mixture-of-Experts model already refuses to use most of itself per token, so flash storage can be part of the memory hierarchy instead of a swap disaster. Built on llama.cpp's public API rather than as a fork, so every quantization, tokenizer and chat template works because llama.cpp is still doing that part, and following upstream is a submodule bump. The underrated case isn't the 284B stunt: it's the 18-22 GB model on a 12 GB phone, where ordinary mmap becomes a fault storm that kills other apps, and streaming turns it into a stable 5 tok/s. Set expectations: ~1 tok/s at the extreme, CPU-only with no GPU or NPU path, MoE architectures only, and a 91 GB download is still a 91 GB download.
colibri — the curator's take
Use it when you want a frontier-scale open MoE (GLM-5.2, 744B) on hardware that can't hold it, and you care about fidelity: forward pass is token-exact against the transformers oracle, and placement only ever changes speed, never semantics. The learning cache pins the experts YOUR workload actually routes, so it genuinely gets faster with use. NOT a general runner — it's a one-model engine; for everyday multi-model local use pick ollama. And expect 0.05–6.8 tok/s depending on tier residency: this is patience-ware for correctness fanatics, not latency-sensitive serving.