FreeToken

Edge-native MoE serving engine: bandwidth-adaptive CPU-GPU co-execution, global LRU expert caching and elastic VRAM run 290B+ frontier MoE models on a gaming PC at interactive speed.

4,269 388 Python Apache-2.0updated today

local

View on GitHubDispute this mapping →

Curator's take

The research-grade answer to 'frontier MoE on one consumer box' — Berkeley/MIT-flavored author list, an arXiv paper behind it, and the two ideas that matter for agents: a q* policy that splits work across GPU, CPU and host memory by available bandwidth, and semantic anchor checkpoints so tool calls and thinking-block edits don't force full context recompute. Ships Anthropic/OpenAI-compatible APIs, so Claude Code, Codex or OpenCode can point straight at it, plus a desktop GUI. Constraints are real: NVIDIA RTX 30/40/50 on Windows or Linux, MoE architectures only, and a paper-fresh engine that moves fast — pin versions. Serving a dense 8B on a proper GPU? This is the wrong tool.

Mapped by ShipWithAI editors · links verified

README.md

Unlock datacenter-class intelligence on the hardware you already own — Run 290B+ frontier MoE models locally on your gaming PC at blistering interactive speeds.

About

FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine designed for running frontier-scale open-weight models on personal and consumer hardware. It treats heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform. Its core features include:

Fast Edge-Native Runtime: Provides efficient MoE serving with bandwidth-adaptive CPU–GPU co-execution ($q^\star$ policy), full-layer double-buffered prefill streaming, global LRU expert caching, graph-compatible execution, and the FTW fast weight format.
Semantic-Aware Caching: Features semantic anchor checkpoints for recurrent state and KV caches, allowing agentic context edits (e.g., tool calls, thinking blocks) to avoid redundant context recomputation.
Elastic Memory Management: Supports dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading.
Broad MoE & Ecosystem Support: Supports frontier open-weight MoE models (e.g., DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2) across various parameter scales and quantization formats (e.g., MXFP4, NVFP4, FP8, BF16), with Anthropic/OpenAI-compatible APIs for seamless integration with real-world coding and tool-calling agents (e.g., Codex, Claude Code, OpenCode, OpenClaw, DeepSeek Harness).
Diverse Consumer Hardware: Scales across consumer laptops, gaming desktops, and workstation GPUs, with native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.

Getting Started

Desktop app

Download FreeToken for Windows or Linux at flashml.ai. It sets the engine up for you and gives you a GUI for running models, chatting, and tuning the engine.

CLI

Install FreeToken with uv (recommended) or pip:

uv pip install "freetoken[accel]"

Or build from source:

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"

For More details:

Citation

If you use FreeToken for your research, please cite our paper:

@article{yang2026freetoken,
  title={FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution},
  author={Yang, Shuo and Fan, Xiaoze and Pan, Melis

Continue your stack

What teams reach for next — and why each earns a place beside FreeToken. Ranked by curator confidence.

Pairs well with

sie

Self-hosted inference cluster for everything agents call besides the big LLM: embeddings, rerankers, OCR, NER, guardrails and small LLMs — 100+ models, one OpenAI-compatible API, K8s stack included.

Why it fitsComplementary halves of a self-hosted agent stack: FreeToken serves the big MoE brain, sie serves everything else the agent calls — embeddings, rerankers, OCR, NER, guardrails, small LLMs.

marker

Datalab's 39k-star PDF-to-Markdown/JSON converter: a layout pipeline plus an optional LLM pass for tables, forms and equations, with chunk output and form-value extraction built in.

Why it fitsmarker's --use_llm hybrid mode accepts any OpenAI-compatible endpoint, so FreeToken can serve those per-page model calls off your own GPU instead of metering Gemini across a large corpus.

Alternative to See all + compare →

colibri

Pure-C, zero-dep MoE runtime that runs GLM-5.2 (744B) on a 25GB-RAM consumer box by streaming experts from disk — VRAM/RAM/NVMe as one tiered hierarchy, never touching precision.

Why it fitsSame trick — stream MoE experts so the model can exceed your memory — different engineering: colibri is pure C with zero dependencies treating NVMe/RAM/VRAM as one hierarchy, FreeToken a Python/CUDA server tuned for RTX GPUs with elastic VRAM and OpenAI-compatible APIs.

BigMoeOnEdge

Run MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output.

Why it fitsbigmoeonedge keeps always-needed weights resident and streams each token's experts for byte-identical CPU-only output, even on phones; FreeToken assumes a consumer GPU and trades that purity for interactive throughput.

vLLM

High-throughput, memory-efficient inference and serving engine for LLMs.

Why it fitsvLLM is the datacenter default: many GPUs, many concurrent users, model fits in VRAM. FreeToken targets the opposite corner — one person, one box, weights far larger than the VRAM.