FreeToken vs mesh-llm
Edge-native MoE serving engine: bandwidth-adaptive CPU-GPU co-execution, global LRU expert caching and elastic VRAM run 290B+ frontier MoE models on a gaming PC at interactive speed. — versus — Distributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.
Two ways to run a model one box cannot hold: mesh-llm pools GPUs across machines into a single OpenAI-compatible endpoint, FreeToken streams MoE experts through one machine's memory hierarchy. Scale out vs scale down.
| FreeToken | mesh-llm | |
|---|---|---|
| Stars | 4.3k | 3.3k |
| Forks | 388 | 396 |
| Language | Python | Rust |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | today | today |
| Topics | local | local |
| Curated connections | 8 | 6 |
FreeToken — the curator's take
The research-grade answer to 'frontier MoE on one consumer box' — Berkeley/MIT-flavored author list, an arXiv paper behind it, and the two ideas that matter for agents: a q* policy that splits work across GPU, CPU and host memory by available bandwidth, and semantic anchor checkpoints so tool calls and thinking-block edits don't force full context recompute. Ships Anthropic/OpenAI-compatible APIs, so Claude Code, Codex or OpenCode can point straight at it, plus a desktop GUI. Constraints are real: NVIDIA RTX 30/40/50 on Windows or Linux, MoE architectures only, and a paper-fresh engine that moves fast — pin versions. Serving a dense 8B on a proper GPU? This is the wrong tool.
mesh-llm — the curator's take
The 'LLM for the people' play: friends or a homelab pool mid-range GPUs and serve models none of them could run alone, with public meshes discoverable via Nostr. The Skippy stage-split design is genuinely clever. Experimental distributed systems — expect rough edges, and never treat a public mesh as private infrastructure. One box that fits your model? Just run Ollama.