Run and serve open models on your own hardware.
Run Llama, Mistral and other open models locally with a single command and a clean API.
High-throughput, memory-efficient inference and serving engine for LLMs.