colibriPure-C, zero-dep MoE runtime that runs GLM-5.2 (744B) on a 25GB-RAM consumer box by streaming experts from disk — VRAM/RAM/NVMe as one tiered hierarchy, never touching precision.
Why switchSame trick — stream MoE experts so the model can exceed your memory — different engineering: colibri is pure C with zero dependencies treating NVMe/RAM/VRAM as one hierarchy, FreeToken a Python/CUDA server tuned for RTX GPUs with elastic VRAM and OpenAI-compatible APIs.
Full comparison → BigMoeOnEdgeRun MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output.
Why switchbigmoeonedge keeps always-needed weights resident and streams each token's experts for byte-identical CPU-only output, even on phones; FreeToken assumes a consumer GPU and trades that purity for interactive throughput.
Full comparison → vLLMHigh-throughput, memory-efficient inference and serving engine for LLMs.
Why switchvLLM is the datacenter default: many GPUs, many concurrent users, model fits in VRAM. FreeToken targets the opposite corner — one person, one box, weights far larger than the VRAM.
Full comparison → airllmLayer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families.
Why switchairLLM slices dense models layer by layer to fit tiny GPUs; FreeToken exploits MoE sparsity instead — narrower model coverage, dramatically better tokens per second on the models it does support.
Full comparison → mesh-llmDistributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.
Why switchTwo ways to run a model one box cannot hold: mesh-llm pools GPUs across machines into a single OpenAI-compatible endpoint, FreeToken streams MoE experts through one machine's memory hierarchy. Scale out vs scale down.
Full comparison → OllamaRun Llama, Mistral and other open models locally with a single command and a clean API.
Why switchollama is the one-command runner for local models that fit and the friendlier daily driver; FreeToken exists precisely for the frontier MoE weights that don't fit.
Full comparison →