BigMoeOnEdge vs needle
Run MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output. — versus — A 45M-parameter tool-calling model shipped as one 14MB binary that runs a full session in ~28MB RAM — grammar-constrained JSON, calibrated confidence, tool retrieval, LoRA fine-tuning.
Opposite answers to the same on-device memory ceiling. BigMoeOnEdge streams a 284B MoE's experts from flash so a phone can run a frontier model; Needle ships a 45M model purpose-built for tool calling that fits in ~28MB and never touches the network. Capability versus footprint.
| BigMoeOnEdge | needle | |
|---|---|---|
| Stars | 397 | 8.9k |
| Forks | 41 | 577 |
| Language | C++ | Python |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 7 days ago | yesterday |
| Topics | local | local, agents, training |
| Curated connections | 6 | 3 |
BigMoeOnEdge — the curator's take
The clever insight is that a Mixture-of-Experts model already refuses to use most of itself per token, so flash storage can be part of the memory hierarchy instead of a swap disaster. Built on llama.cpp's public API rather than as a fork, so every quantization, tokenizer and chat template works because llama.cpp is still doing that part, and following upstream is a submodule bump. The underrated case isn't the 284B stunt: it's the 18-22 GB model on a 12 GB phone, where ordinary mmap becomes a fault storm that kills other apps, and streaming turns it into a stable 5 tok/s. Set expectations: ~1 tok/s at the extreme, CPU-only with no GPU or NPU path, MoE architectures only, and a 91 GB download is still a 91 GB download.
needle — the curator's take
The right tool when the agent is a phone, wearable or robot and the job is picking a function and filling its arguments — not reasoning. Three design choices make it usable rather than a demo: byte-level grammar compiled from your schema, so output is always valid JSON; a calibrated confidence head you threshold to escalate to a big model; and a retrieval head that renders only the top five tools per turn, so a large catalogue still fits the 256-token window. Fine-tuning is LoRA on a frozen base merged into a single `.cact`, so a tuned model keeps the same one-file deployment. NOT a chat model and NOT a reasoner — there is no room for multi-step planning, and writing good tool descriptions is most of the accuracy.