LMCache — what pairs with it, and why

LMCache

KV-cache layer for scalable LLM serving: offload and reuse KV across GPU/CPU/disk/remote tiers to cut TTFT and prefill cost. vLLM-first; used by NVIDIA Dynamo and llm-d.

10,721 1,582 Python Apache-2.0updated today

local storage

View on GitHubDispute this mapping →

Curator's take

Use when serving LLMs at scale on vLLM with long or repeated contexts — multi-turn agents, RAG, shared system prompts — where prefill dominates and prefix cache hits are money. It moves KV out of GPU into CPU/disk/Redis-style backends and shares it across instances. NOT for single-box hobby setups (Ollama-class) or workloads with no prompt overlap: you'd add a storage tier and ops surface for zero hit rate. PyTorch Foundation project, Apache-2.0.

Mapped by ShipWithAI editors · links verified

README.md

A KV Cache Management Layer for Scalable LLM Inference

Blog | Documentation | Join Slack | Community Meeting | Roadmap

⭐ If LMCache helps you serve LLMs faster and cheaper, give us a star — it helps more teams discover the project.

Updates

[2026/05] 🔥 Agentic workload benchmark on AMD MI300X (blog).
[2026/04] 🔥 LMCache's new multiprocess (MP) architecture release (blog).
[2026/03] LMCache at GTC 2026 (post).
[2026/01] LMCache multi-node P2P CPU memory sharing, from experimental feature to production (blog).

[2025/11] LMCache x CoreWeave accelerate efficient LLM inference for Cohere (blog).
[2025/10] LMCache joins the PyTorch Foundation and Tensormesh unveiled (blog, PyTorch).
[2025/09] NVIDIA Dynamo integrates LMCache, accelerating LLM inference (blog).
[2025/08] 🎉 LMCache hits 5,000+ GitHub stars (blog).
[2025/08] LMCache supports gpt-oss (20B/120B) on day 1 (blog).
[2025/07] Get faster LLM inference and cheaper responses with LMCache and Redis (Redis blog).
[2025/07] LMCache extends its turbo-boost to multimodal models in vLLM V1 (blog).
[2025/06] LLM Production Stack goes cross-hardware: AMD, Arm and Ascend (blog).

About

LMCache is a KV cache management layer for LLM inference. It turns KV cache from a temporary state into reusable AI-native knowledge that can be stored persistently, reused across multiple serving engines, monitored with an observabi

Continue your stack

What teams reach for next — and why each earns a place beside LMCache. Ranked by curator confidence.

Pairs well with

llm-d

Distributed inference stack for Kubernetes from Red Hat, Google and IBM (CNCF) — prefix-cache-aware routing, tiered KV-cache, prefill/decode disaggregation and SLO autoscaling above vLLM/SGLang.

Why it fitsllm-d's tiered KV-cache subsystem in Kubernetes is built around LMCache for offload; deploy together for cluster-scale prefix caching.

Built with / on

vLLM

High-throughput, memory-efficient inference and serving engine for LLMs.

Why it fitsLMCache plugs into vLLM as its KV-connector; vLLM is the primary serving engine it accelerates.