StackMap
Subscribe
Explore / mesh-llm
Mesh-LLM

mesh-llm

Distributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.

3,504 441 Rust Apache-2.0updated today
local
View on GitHubDispute this mapping →
Curator's take

The 'LLM for the people' play: friends or a homelab pool mid-range GPUs and serve models none of them could run alone, with public meshes discoverable via Nostr. The Skippy stage-split design is genuinely clever. Experimental distributed systems — expect rough edges, and never treat a public mesh as private infrastructure. One box that fits your model? Just run Ollama.

Mapped by ShipWithAI editors · links verified

Continue your stack

What teams reach for next — and why each earns a place beside mesh-llm. Ranked by curator confidence.

alternativealternativealternativealternativealternativealternativealternativePersonal-AI-RoutercolibriFreeTokenOllamaairllmllm-dBigMoeOnEdgemesh-llm
pairs wellalternativebuilt withpick a node for the why · open it from the panel
Weekly digest
README.md2 min read

Mesh LLM

Mesh LLM web console

Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. Start one node, add more nodes later, and let the mesh decide whether a model runs locally, routes to a peer, or uses Skippy stage splits for models that are too large for one box.

Quick start

Install the latest release executable:

curl -fsSL https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.sh | bash

On Windows, use PowerShell:

irm https://raw.githubusercontent.com/Mesh-LLM/mesh-llm/main/install.ps1 | iex

Finish setup:

mesh-llm setup

On Windows PowerShell, use mesh-llm.exe setup.

To remove an executable install later, preview the cleanup first:

mesh-llm uninstall --dry-run
mesh-llm uninstall --yes

Uninstall preserves ~/.mesh-llm configuration and identity data unless you explicitly pass --purge-config.

Join the public mesh and start serving:

mesh-llm serve --auto

That command chooses a backend flavor, downloads a suitable model if needed, joins the best discovered public mesh, starts the local API on port 9337, and starts the web console on port 3131.

Check available models:

curl -s http://localhost:9337/v1/models | jq '.data[].id'

Send an OpenAI-compatible request:

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

For server deployments, add --headless to hide the web UI while keeping the management API on the --console port:

mesh-llm serve --auto --headless

Pick the workflow you need

Goal Command Full guide
Try the public mesh mesh-llm serve --auto docs/MESHES.md
Start a private mesh mesh-llm serve --model Qwen3-8B-Q4_K_M docs/MESHES.md
Publish your own mesh mesh-llm serve --model Qwen3-8B-Q4_K_M --publish docs/MESHES.md
Join by invite token mesh-llm serve --join <token> docs/MESHES.md
Run an API-only client mesh-llm client --auto docs/MESHES.md
Run a big model with splits mesh-llm serve --model hf://meshllm/<repo>@<rev> --split docs/SKIPPY_SPLITS.md
Attach a Flash-MoE SSD backend mesh-llm serve with [[plugin]] name = "flash-moe" docs/plugins/flash-moe.md
Fan out one prompt to every model in the mesh curl ... -d '{"model":"mesh", ...}' docs/design/MOA_GATEWAY.md
Use Goose, OpenCode, Claude Code, or Pi mesh-llm goose, mesh-llm opencode, mesh-llm claude, mesh-llm pi docs/AGENTS.md
Build or contribute just build CONTRIBUTING.md

How the mesh works

  • Single-machine fit first. If one node can host the full model, it serves the model locally without stage traffic.
  • Mesh routing. Every node exposes the same /v1 API. Requests are routed by the model field to the peer that can serve that model.
  • Owner-control plane. Operator config and inventory actions use an additive mesh-llm-control/1 lane with explicit endpoint bootstrap, while public mesh join, gossip, routing, and inference stay on the public mesh plane for mixed-version compatibility.
  • Skippy stage splits. Large dense models can load as package-backed layer stages. The coordinator plans contiguous layer ranges, starts downstream stages first, waits for readiness, then publishes the stage-0 route.
  • Layer packages. Package repositories contain model-package.json plus GGUF fragments so peers fetch only the pieces needed for their assigned stage.
  • **Public disc