llm-d vs mesh-llm
Distributed inference stack for Kubernetes from Red Hat, Google and IBM (CNCF) — prefix-cache-aware routing, tiered KV-cache, prefill/decode disaggregation and SLO autoscaling above vLLM/SGLang. — versus — Distributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.
Distributed inference at opposite trust levels: llm-d orchestrates vLLM across a Kubernetes cluster you own; mesh-llm federates volunteer boxes over the open internet.
| llm-d | mesh-llm | |
|---|---|---|
| Stars | 3.8k | 2.7k |
| Forks | 613 | 322 |
| Language | Shell | Rust |
| License | Apache-2.0 | Apache-2.0 |
| Last activity | 3 days ago | today |
| Topics | local | local |
| Curated connections | 3 | 4 |
llm-d — the curator's take
The Kubernetes answer when one vLLM box stops scaling: prefix-cache-aware routing, disaggregated prefill/decode and tiered KV offloading deliver real, benchmarked wins (3x throughput, 2x TTFT in partner numbers) at fleet scale, with Red Hat/Google/IBM/NVIDIA behind it. NOT for a single GPU or a laptop — that's Ollama or plain vLLM territory — and not turnkey: you are operating Kubernetes, Helm charts and gateways. If you don't already run K8s, don't start here.
mesh-llm — the curator's take
The 'LLM for the people' play: friends or a homelab pool mid-range GPUs and serve models none of them could run alone, with public meshes discoverable via Nostr. The Skippy stage-split design is genuinely clever. Experimental distributed systems — expect rough edges, and never treat a public mesh as private infrastructure. One box that fits your model? Just run Ollama.