BigMoeOnEdge

Run MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output.

395 41 C++ Apache-2.0updated 7 days ago

local

View on GitHubDispute this mapping →

Curator's take

The clever insight is that a Mixture-of-Experts model already refuses to use most of itself per token, so flash storage can be part of the memory hierarchy instead of a swap disaster. Built on llama.cpp's public API rather than as a fork, so every quantization, tokenizer and chat template works because llama.cpp is still doing that part, and following upstream is a submodule bump. The underrated case isn't the 284B stunt: it's the 18-22 GB model on a 12 GB phone, where ordinary mmap becomes a fault storm that kills other apps, and streaming turns it into a stable 5 tok/s. Set expectations: ~1 tok/s at the extreme, CPU-only with no GPU or NPU path, MoE architectures only, and a 91 GB download is still a 91 GB download.

Mapped by ShipWithAI editors · links verified

README.md

Run Mixture-of-Experts models bigger than your device's RAM. On a phone, on a PC, CPU only.

A Mixture-of-Experts model is made of many small "experts", and each generated token only uses a few of them. BigMoeOnEdge takes that literally: it keeps the small always-needed part of the model at hand and reads just the experts each token asks for, directly from flash storage, at the moment they are needed. The rest of the model stays on disk. That is what lets a model several times bigger than your RAM generate text on an ordinary phone, losslessly: the output is byte-identical to running the same model fully resident.

It is built on top of llama.cpp's public API, not as a fork. Every quantization format, tokenizer and chat template llama.cpp supports works out of the box, because llama.cpp itself is doing that part: MXFP4 and Q4_K_M stream through the same code. Supporting a new MoE architecture is one row in a registry, and following a new llama.cpp release is a routine submodule bump.

The most extreme thing it can do today: DeepSeek V4 Flash 0731, a 284B-parameter MoE (~91 GB on disk at 2-bit expert quantization), generating on a phone with 12 GB of RAM at about 1 tok/s. More than seven times more model than memory, streamed from flash as the three shard files Hugging Face ships, with no merge step and no PC in the loop.

DeepSeek V4 Flash 0731 (284B, ~91 GB) generating in the demo app on a 12 GB phone, with live tok/s and telemetry

DeepSeek V4 Flash 0731: 284B parameters, ~91 GB on disk, on a 12 GB phone. 0.94 tok/s in the demo app, real time.

It is not one model, either. Below: three of them, one after another on the same phone, each past what it should be able to hold.

https://github.com/user-attachments/assets/f899b93f-c7c4-4ce9-9fb0-5ed1bae13761

Left to right: gpt-oss-120b (~60 GB), Qwen3-30B-A3B (18.5 GB), Gemma-4-26B-A4B (17 GB), recorded in the demo app on a 12 GB phone, real time, not sped up.

Why this exists
Try it on your phone
Features
Supported models
Benchmarks
Quickstart
How it works
Documentation
Prior art
License

Why this exists

The models people actually want to talk to keep growing faster than the RAM in the devices they carry. MoE models offer a way out, because most of their weights sit idle on any given token, but every mainstream runtime still insists on holding (or paging) the whole file in memory. So a 20 GB model on a 12 GB phone either refuses to load or crawls while the OS frantically swaps.

BigMoeOnEdge treats flash storage as part of the memory hierarchy instead. Three situations where that changes what your device can run:

Models far past RAM. A ~60 GB model on a 12 GB phone cannot be resident, full stop. Streamed, it runs at usable speed. This is the headline case, but it is not the only one.

Models just past RAM. An 18 to 22 GB model on a 12 GB phone is where the ordinary way of loading (mmap) tur

Continue your stack

What teams reach for next — and why each earns a place beside BigMoeOnEdge. Ranked by curator confidence.

Alternative to See all + compare →

colibri

Pure-C, zero-dep MoE runtime that runs GLM-5.2 (744B) on a 25GB-RAM consumer box by streaming experts from disk — VRAM/RAM/NVMe as one tiered hierarchy, never touching precision.

Why it fitsThe same trick aimed at different hardware: Colibri is a pure-C MoE runtime streaming experts from NVMe to fit a 744B model on a 25 GB desktop; BigMoeOnEdge targets phones on plain CPU and rides llama.cpp's format support instead of its own runtime.

airllm

Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families.

Why it fitsTwo ways to run a model that doesn't fit: AirLLM streams transformer layers sequentially to squeeze a 70B onto a 4 GB GPU and works on dense models; BigMoeOnEdge streams only the experts a token selects, which is faster but MoE-only.

mesh-llm

Distributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.

Why it fitsWhen the model exceeds the box you have two moves — borrow more memory or stream from storage. mesh-llm pools GPUs across machines into one endpoint; BigMoeOnEdge assumes you have exactly one device and its flash chip.

BigMoeOnEdge

Table of contents

Why this exists

Continue your stack