colibriPure-C, zero-dep MoE runtime that runs GLM-5.2 (744B) on a 25GB-RAM consumer box by streaming experts from disk — VRAM/RAM/NVMe as one tiered hierarchy, never touching precision.
Why switchSame job — giant models on tiny hardware — opposite technique: AirLLM streams dense layers through a 4GB GPU from Python/HF, colibrì streams MoE experts from NVMe in pure C with a workload-learning pin cache.
Full comparison → BigMoeOnEdgeRun MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output.
Why switchTwo ways to run a model that doesn't fit: AirLLM streams transformer layers sequentially to squeeze a 70B onto a 4 GB GPU and works on dense models; BigMoeOnEdge streams only the experts a token selects, which is faster but MoE-only.
Full comparison → FreeTokenEdge-native MoE serving engine: bandwidth-adaptive CPU-GPU co-execution, global LRU expert caching and elastic VRAM run 290B+ frontier MoE models on a gaming PC at interactive speed.
Why switchairLLM slices dense models layer by layer to fit tiny GPUs; FreeToken exploits MoE sparsity instead — narrower model coverage, dramatically better tokens per second on the models it does support.
Full comparison → OllamaRun Llama, Mistral and other open models locally with a single command and a clean API.
Why switchBoth run open models on your own hardware, on opposite sides of one constraint: Ollama gives fast, polished local inference for models that fit your VRAM; AirLLM runs models that don't fit at all — 70B on 4GB — by streaming one layer at a time, at heavy latency cost.
Full comparison → Model-OptimizerNVIDIA's model-compression library: quantization (PTQ/QAT, FP8/NVFP4), pruning, distillation, NAS and speculative decoding over HF/PyTorch/ONNX, exported to TensorRT-LLM, vLLM and SGLang.
Why switchBoth make an oversized model fit on hardware it shouldn't. AirLLM streams layers and explicitly refuses to quantize, preserving quality at the cost of latency; ModelOpt compresses the weights and keeps throughput.
Full comparison → mesh-llmDistributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.
Why switchOpposite answers to 'the model doesn't fit': AirLLM streams layers from disk on one small GPU (slow, solo), mesh-llm splits stages across peers' GPUs (faster, needs friends).
Full comparison → vLLMHigh-throughput, memory-efficient inference and serving engine for LLMs.
Why switchOpposite ends of the local-inference spectrum: vLLM maximizes throughput given abundant VRAM (production serving); AirLLM minimizes VRAM given abundant patience (frontier-size models on consumer cards).
Full comparison →