BigMoeOnEdge vs mesh-llm
Run MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output. — versus — Distributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.
When the model exceeds the box you have two moves — borrow more memory or stream from storage. mesh-llm pools GPUs across machines into one endpoint; BigMoeOnEdge assumes you have exactly one device and its flash chip.
| BigMoeOnEdge | mesh-llm | |
|---|---|---|
| Stars | 395 | 3.3k |
| Forks | 41 | 393 |
| Language | C++ | Rust |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 7 days ago | 4 days ago |
| Topics | local | local |
| Curated connections | 3 | 5 |
BigMoeOnEdge — the curator's take
The clever insight is that a Mixture-of-Experts model already refuses to use most of itself per token, so flash storage can be part of the memory hierarchy instead of a swap disaster. Built on llama.cpp's public API rather than as a fork, so every quantization, tokenizer and chat template works because llama.cpp is still doing that part, and following upstream is a submodule bump. The underrated case isn't the 284B stunt: it's the 18-22 GB model on a 12 GB phone, where ordinary mmap becomes a fault storm that kills other apps, and streaming turns it into a stable 5 tok/s. Set expectations: ~1 tok/s at the extreme, CPU-only with no GPU or NPU path, MoE architectures only, and a 91 GB download is still a 91 GB download.
mesh-llm — the curator's take
The 'LLM for the people' play: friends or a homelab pool mid-range GPUs and serve models none of them could run alone, with public meshes discoverable via Nostr. The Skippy stage-split design is genuinely clever. Experimental distributed systems — expect rough edges, and never treat a public mesh as private infrastructure. One box that fits your model? Just run Ollama.