Local / Inference
Run and serve open models on your own hardware.
mikehasa 755 · 3 days agoLocal-first work receipts for coding agents: reads the session logs Claude Code, Codex, OpenCode and Hermes already write and reports what each task did, what it cost, and whether a check proved it.
vllm-project 301 · 3 days agoRust gateway that gives vLLM a stateful, OpenAI-compatible Responses API: server-side conversation state, server-side tool loops, SSE and WebSocket streaming, background runs — Codex-ready.
eunomia-bpf 710 · 18 days agoSystem-level observability for AI agents via eBPF and TLS tracing: correlates prompts and model calls with the processes, files and network the agent actually touched — no SDK, no proxy.
lyogavin 35.1k · 3 days agoLayer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families.
nokia-applied-research 998 · 3 days agoNokia research: turn any open LLM into a Jev-style decision model. Typed choice, yes/no or score from one prefill, de-biased with no labels or a fitted head, served on vLLM.
arcboxlabs 4.3k · 3 days agoOpen-source container and VM runtime for macOS in pure Rust: drop-in Docker engine, sub-100ms agent sandboxes (`abctl claude`), full Linux VMs and throwaway macOS guests on one daemon.
Helldez 595 · 3 days agoRun MoE models bigger than your RAM: keep the always-needed weights resident and stream each token's experts from flash — a 284B model on a 12 GB phone, CPU only, byte-identical output.
claraverse-space 3.9k · 1 months agoSelf-hosted private AI workspace: chat, multi-agent crews with human review, a visual workflow builder and Telegram integration in one app — local models via Ollama/llama.cpp or your own keys.
cocoindex-io 2.7k · 9 days agoAST-based semantic code search for coding agents: pipx install, zero config, local embeddings out of the box — a CLI/skill/MCP that cuts agent context ~70% vs grepping. Built on CocoIndex.
DeusData 45.3k · 3 days agoCode intelligence MCP in pure C: tree-sitter knowledge graph over 158 languages, average repo indexed in milliseconds, sub-ms queries, 10x fewer tokens. Single static binary, zero deps.
getagentseal 11.3k · 4 days agoLocal-first cost ledger for AI coding: reads the session files 36 tools already write and breaks every token and dollar down by task, model, project. TUI, web, desktop, menubar — no proxy, no keys.
JustVugg 38.1k · 3 days agoPure-C, zero-dep MoE runtime that runs GLM-5.2 (744B) on a 25GB-RAM consumer box by streaming experts from disk — VRAM/RAM/NVMe as one tiered hierarchy, never touching precision.
apple 2.2k · 5 days agoApple's official Core AI toolkit: recipes exporting Hugging Face models to .aimodel, PyTorch primitives for authoring, Swift runtime for macOS/iOS apps — plus skills for coding agents.
TencentCloud 12.7k · 3 days agoHardware-isolated microVM sandboxes for AI agents — sub-60ms boot, <5MB overhead, E2B-compatible API, self-hosted on your own KVM nodes.
EXXETA 360 · 3 days agoLocal-first agentic runtime with persistent AI rooms and approval-gated memory: every memory write needs your OK; rooms, KB and artifacts are plain files on disk.
deeplethe 2.9k · 17 days agofork() for agent microVMs: children fork copy-on-write from a warm Firecracker parent — 100 KVM-isolated VMs in ~100ms, live-VM branching in ~56ms, portable snapshots from a hub.
FlashML-org 13.9k · 5 days agoEdge-native MoE serving engine: bandwidth-adaptive CPU-GPU co-execution, global LRU expert caching and elastic VRAM run 290B+ frontier MoE models on a gaming PC at interactive speed.
marcelroed 4.1k · 29 days agoTokenization at GB/s: ~1000x faster than HuggingFace tokenizers with drop-in compatibility modes for HF and tiktoken — Rust reading your files directly. pip install gigatoken.
zzet 1.8k · 3 days agoCode-intelligence engine in one static Go binary: tree-sitter graph over 257 languages, compiler-grade resolution for 17, multi-repo, 175 configurable MCP tools — up to 50x fewer tokens. 100% local.
gridex 1.5k · 24 days agoNative database IDE for Postgres, MySQL, SQLite, Redis, MongoDB, SQL Server and ClickHouse, with a built-in MCP server — 13 tools, 3-tier permissions, audit trail — and schema-aware AI chat.
antirez 2.8k · 1 months agoantirez's C/Metal engine running MiniMax-H3 video and audio generation natively on Apple Silicon — interactive prompting, 4-step schedules, SSD block streaming down to ~2 GB DiT residency.
cjpais 32.4k · 3 days agoPush-to-talk offline dictation: hotkey, speak, text lands in whatever field has focus. Whisper or Parakeet fully on-device; cross-platform Rust/Tauri, built to be forked.
jaredpalmer 7.6k · 3 days agoOpen reproduction of TypeSafe's Jev: a LoRA + readout head on Qwen that answers many typed questions about one document in a single prefill pass, returning calibrated probabilities.
FareedKhan-dev 8.8k · 9 days agoKimi K3 (2.78T-parameter MoE) inference in portable C99 on one CPU: the dense trunk stays in RAM, 4-bit experts stream from disk, and output is byte-identical from 8 GB to 224 GB.
rahulnyk 4.1k · 1 months agoNotebook recipe that turns any text corpus into a concept graph with a local Mistral 7B via Ollama — chunk, extract concepts and relations, add proximity edges — for Graph RAG and KG QA.
ovg-project 1.5k · 6 days agoVirtualized, elastic KV cache for LLM serving on shared GPUs: reserve virtual memory, back it with physical GPU memory only when used — vLLM and SGLang, with a memory-limit CLI, router and sleep mode.
jeffhajewski 694 · 8 days agoEmbedded single-file graph database in Zig: graph traversal, HNSW vector search and BM25 full-text in one query language, plus a durable event log — built for Graph RAG and local agent memory.
NandhaKishorM 29.8k · todayNon-autoregressive decision engine: typed choice, score and yes/no answers over text in one forward pass (~33 ms), 100+ languages, calibrated probabilities, a router picking the checkpoint.
ashuttl 563 · 3 days agoSix dependency-free terminal apps for weather, sun, moon, tides, radar and maps, drawn from free public data with no accounts or API keys — mouse-friendly TUIs that try to match your terminal theme.
llm-d 4.7k · 3 days agoDistributed inference stack for Kubernetes from Red Hat, Google and IBM (CNCF) — prefix-cache-aware routing, tiered KV-cache, prefill/decode disaggregation and SLO autoscaling above vLLM/SGLang.
LMCache 11.9k · 3 days agoKV-cache layer for scalable LLM serving: offload and reuse KV across GPU/CPU/disk/remote tiers to cut TTFT and prefill cost. vLLM-first; used by NVIDIA Dynamo and llm-d.
githubnext 792 · 13 days agoGitHub Next's local Jev bridge: a Bun/TypeScript POST /v1/systemone that translates typed decision questions into prompts for DiffusionGemma behind any OpenAI-compatible endpoint.
ysharma3501 5.4k · 3 months agoLightweight voice-cloning TTS — 48kHz speech at 150x realtime, fits in 1GB VRAM and runs on CPU or MPS. SOTA cloning from a ~3s reference sample, rivaling models 10x larger.
Fast-Editor 598 · 8 days agoSelf-hosted LLM gateway wrapping Claude Code, Cursor or Codex with zero code changes — strips unused tools, compresses JSON tool results ~88%, semantic-caches, tier-routes easy work to local models.
magnitudedev 5.4k · 3 days agoLocal inference desktop app + CLI: profiles your hardware, estimates tok/s per model before download, tunes the one you pick, and connects Pi, OpenCode, Hermes, Codex or Claude Code in a click.
Zackriya-Solutions 31.2k · 16 days agoPrivacy-first meeting note-taker that runs 100% on-device: live Whisper/Parakeet transcription, speaker diarization, local Ollama summaries. Desktop app for macOS & Windows — no cloud, no call bots.
moorcheh-ai 2.3k · 3 days agoCompanion memory agent for 20+ coding agents, built on Moorcheh — its own information-theoretic engine, no third-party vector DB to manage. Runs local (Docker + Ollama, keyless) or on their cloud.
memvid 16.6k · 2 months agoSingle-file memory layer for agents: data, embeddings, index and metadata in one portable .mv2 — append-only Smart Frames, time-travel queries, sub-5ms recall, no server. Rust core, Node/Python SDKs.
Mesh-LLM 3.5k · 3 days agoDistributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.
Goekdeniz-Guelmez 271 · 1 months agoNative Mac app for on-device LLM fine-tuning via mlx-lm-lora: pick a model, choose SFT/LoRA/DPO-family algorithms, watch loss fall live, push to Hugging Face. No cloud, no code.
NVIDIA 5k · 3 days agoNVIDIA's model-compression library: quantization (PTQ/QAT, FP8/NVFP4), pruning, distillation, NAS and speculative decoding over HF/PyTorch/ONNX, exported to TensorRT-LLM, vLLM and SGLang.
cactus-compute 12.8k · 3 days agoA 45M-parameter tool-calling model shipped as one 14MB binary that runs a full session in ~28MB RAM — grammar-constrained JSON, calibrated confidence, tool retrieval, LoRA fine-tuning.
Osmantic 6.9k · 3 days agoOne installer that turns a PC, Mac or Linux box into a private AI server: Ollama, Open WebUI, n8n, ComfyUI wired together — inference, chat, voice, agents, RAG and image gen, no cloud.
ollama 181.8k · 4 days agoRun Llama, Mistral and other open models locally with a single command and a clean API.
opensandbox-group 15.6k · 3 days agoCNCF-landscape sandbox platform for AI agents: multi-language SDKs, unified API, CLI and MCP over Docker/Kubernetes runtimes — coding agents, GUI agents, evals and RL training.
NVIDIA 1.5k · 8 days agoNVIDIA's local inference router: pairs home machines running Ollama or LM Studio behind OpenAI-, Anthropic- and Ollama-compatible endpoints, sending each request to the best available node.
kyutai-labs 9.7k · 3 days agoKyutai's 100M-parameter CPU-only TTS — streaming audio in ~200ms, ~6× real-time on two laptop cores, voice cloning, six languages. pip install and it talks.
rtk-ai 81.9k · 3 days agoRust CLI proxy compressing dev-command output 60-90% before your agent reads it — git, tests, linters, docker, 100+ commands; hooks auto-rewrite bash calls. Single binary, <10ms overhead.
hanxiao 53 · 3 months agoAirgapped closed-corpus QA testbed: a local Qwen agent in a Pi harness explores a .zip dataroom with grep/embeddings/rerankers under a token budget — a bed to study search as test-time compute.
superlinked 3.3k · 3 days agoSelf-hosted inference cluster for everything agents call besides the big LLM: embeddings, rerankers, OCR, NER, guardrails and small LLMs — 100+ models, one OpenAI-compatible API, K8s stack included.
kennss 966 · 5 days agoSudoless Apple Silicon monitor: SwiftUI dashboard plus menu-bar suite tracking ANE, Media Engine and memory bandwidth Activity Monitor won't show — with DVR-style record & replay.
kelviq 292 · 1 months agoClaude Code skill that reads your local session logs and answers 'where did my tokens go' in plain English: deduplicated totals, cost charged to the tool that caused it, 5-hour window state.
aovestdipaperino 650 · 4 days agoCode-intelligence MCP server for coding agents — a pre-indexed semantic graph (libSQL + FTS5) they query instead of grepping: symbols, callers, impact radius in one call. 100% local, 50+ languages.
RyanCodrai 17.3k · 18 days agoRust vector index with Python bindings built on Google's TurboQuant: no training step, online ingest, hand-written SIMD kernels — 10M x 1536 vectors in ~4 GB, with allowlist-filtered search.
vllm-project 92.9k · 3 days agoHigh-throughput, memory-efficient inference and serving engine for LLMs.
xberg-io 9.3k · 3 days agoRust-core extraction orchestrator with 15 language bindings: 96 formats — PDF, Office, images, audio, code, web — to clean text, tables and RAG-ready chunks. OCR and structured extraction built in.
xybrid-ai 460 · 3 days agoCross-platform on-device AI toolkit: run LLMs, ASR and TTS natively from Flutter, Unity, Kotlin, Swift or Rust on a llama.cpp and ONNX Runtime core. Private, offline, no cloud.
zvec-ai 3.8k · 3 days agozg: ripgrep, BM25 and vector search behind one local-first CLI for humans and agents — index a workspace once, search code, docs and data by meaning, then verify with exact text or regex.