English · 繁體中文
Tiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.
Colibrì is a lightweight, quality-preserving MoE runtime that treats VRAM, RAM, and storage as one managed memory hierarchy. Insufficient fast memory may reduce speed, but the default policy never silently changes model precision or router semantics.
$ ./coli chat
🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
✓ ready in 32s · resident 9.9 GB
› ciao!
◆ Ciao! 😊 Come posso aiutarti oggi?
See it running
The web dashboard (./coli web): a 744B model at 4 tok/s, TTFT 1.6 s, disk 0 —
full expert residency on 6× RTX 5090, with live token metrics, the per-turn time breakdown,
the VRAM/RAM/disk tier bar and the live mini-brain in the corner.
The Brain page: all 19,456 experts as a living cortex — colour is the storage tier, brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's measured topic affinity.
The Atlas page: the measured expert atlas as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic (poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.
The idea
A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts):
So the model doesn't need to fit in fast memory — it needs to be placed:
- the dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4 (~9.9 GB);
- the 19,456 routed experts (75 MoE layers × 256 + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier.
The engine is a single C file (c/glm.c) plus small headers. No BLAS, no Python
at runtime, no GPU required.
How it works
The per-token path
Every layer of every token walks the same five steps. The design goal is that placement only ever decides speed — the router's decisions and the weights' precision are the same whether an expert answered from VRAM or from disk.
One memory hierarchy instead of one memory requirement
The same engine spans the whole range: on a 25 GB laptop everything streams from
disk (slow but correct); on a large host the entire expert set becomes resident
(CUDA_EXPERT_GB=auto PIN_GB=all) and disk drops out of the decode path
entirely. Between the tiers sits a learning cache: the engine records which
experts your workload routes to (`.coli_usa