Local / Inference

lyogavin 31.7k · today

Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families.

claraverse-space 3.9k · 17 days ago

Self-hosted private AI workspace: chat, multi-agent crews with human review, a visual workflow builder and Telegram integration in one app — local models via Ollama/llama.cpp or your own keys.

cocoindex-io 2.7k · 13 days ago

AST-based semantic code search for coding agents: pipx install, zero config, local embeddings out of the box — a CLI/skill/MCP that cuts agent context ~70% vs grepping. Built on CocoIndex.

DeusData 39.6k · yesterday

Code intelligence MCP in pure C: tree-sitter knowledge graph over 158 languages, average repo indexed in milliseconds, sub-ms queries, 10x fewer tokens. Single static binary, zero deps.

getagentseal 9.6k · yesterday

Local-first cost ledger for AI coding: reads the session files 36 tools already write and breaks every token and dollar down by task, model, project. TUI, web, desktop, menubar — no proxy, no keys.

JustVugg 25.5k · today

Pure-C, zero-dep MoE runtime that runs GLM-5.2 (744B) on a 25GB-RAM consumer box by streaming experts from disk — VRAM/RAM/NVMe as one tiered hierarchy, never touching precision.

apple 1.6k · yesterday

Apple's official Core AI toolkit: recipes exporting Hugging Face models to .aimodel, PyTorch primitives for authoring, Swift runtime for macOS/iOS apps — plus skills for coding agents.

TencentCloud 11.3k · today

Hardware-isolated microVM sandboxes for AI agents — sub-60ms boot, <5MB overhead, E2B-compatible API, self-hosted on your own KVM nodes.

EXXETA 334 · yesterday

Local-first agentic runtime with persistent AI rooms and approval-gated memory: every memory write needs your OK; rooms, KB and artifacts are plain files on disk.

deeplethe 2.7k · 5 days ago

fork() for agent microVMs: children fork copy-on-write from a warm Firecracker parent — 100 KVM-isolated VMs in ~100ms, live-VM branching in ~56ms, portable snapshots from a hub.

marcelroed 4k · 14 days ago

Tokenization at GB/s: ~1000x faster than HuggingFace tokenizers with drop-in compatibility modes for HF and tiktoken — Rust reading your files directly. pip install gigatoken.

zzet 1.3k · today

Code-intelligence engine in one static Go binary: tree-sitter graph over 257 languages, compiler-grade resolution for 17, multi-repo, 175 configurable MCP tools — up to 50x fewer tokens. 100% local.

cjpais 30k · yesterday

Push-to-talk offline dictation: hotkey, speak, text lands in whatever field has focus. Whisper or Parakeet fully on-device; cross-platform Rust/Tauri, built to be forked.

llm-d 4.1k · today

Distributed inference stack for Kubernetes from Red Hat, Google and IBM (CNCF) — prefix-cache-aware routing, tiered KV-cache, prefill/decode disaggregation and SLO autoscaling above vLLM/SGLang.

LMCache 11.2k · today

KV-cache layer for scalable LLM serving: offload and reuse KV across GPU/CPU/disk/remote tiers to cut TTFT and prefill cost. vLLM-first; used by NVIDIA Dynamo and llm-d.

ysharma3501 5.3k · 2 months ago

Lightweight voice-cloning TTS — 48kHz speech at 150x realtime, fits in 1GB VRAM and runs on CPU or MPS. SOTA cloning from a ~3s reference sample, rivaling models 10x larger.

Fast-Editor 542 · today

Self-hosted LLM gateway wrapping Claude Code, Cursor or Codex with zero code changes — strips unused tools, compresses JSON tool results ~88%, semantic-caches, tier-routes easy work to local models.

Zackriya-Solutions 29.6k · 2 months ago

Privacy-first meeting note-taker that runs 100% on-device: live Whisper/Parakeet transcription, speaker diarization, local Ollama summaries. Desktop app for macOS & Windows — no cloud, no call bots.

moorcheh-ai 1.8k · yesterday

Companion memory agent for 20+ coding agents, built on Moorcheh — its own information-theoretic engine, no third-party vector DB to manage. Runs local (Docker + Ollama, keyless) or on their cloud.

memvid 16.4k · 1 months ago

Single-file memory layer for agents: data, embeddings, index and metadata in one portable .mv2 — append-only Smart Frames, time-travel queries, sub-5ms recall, no server. Rust core, Node/Python SDKs.

Mesh-LLM 3.3k · today

Distributed LLM inference in Rust: pool GPUs across machines into one OpenAI-compatible endpoint — local fit first, mesh routing, and stage splits for models too large for any single box.

Goekdeniz-Guelmez 257 · 1 months ago

Native Mac app for on-device LLM fine-tuning via mlx-lm-lora: pick a model, choose SFT/LoRA/DPO-family algorithms, watch loss fall live, push to Hugging Face. No cloud, no code.

Osmantic 4.5k · 7 days ago

One installer that turns a PC, Mac or Linux box into a private AI server: Ollama, Open WebUI, n8n, ComfyUI wired together — inference, chat, voice, agents, RAG and image gen, no cloud.

ollama 179k · today

Run Llama, Mistral and other open models locally with a single command and a clean API.

opensandbox-group 14.4k · today

CNCF-landscape sandbox platform for AI agents: multi-language SDKs, unified API, CLI and MCP over Docker/Kubernetes runtimes — coding agents, GUI agents, evals and RL training.

kyutai-labs 8.8k · 3 days ago

Kyutai's 100M-parameter CPU-only TTS — streaming audio in ~200ms, ~6× real-time on two laptop cores, voice cloning, six languages. pip install and it talks.

rtk-ai 76.7k · today

Rust CLI proxy compressing dev-command output 60-90% before your agent reads it — git, tests, linters, docker, 100+ commands; hooks auto-rewrite bash calls. Single binary, <10ms overhead.

hanxiao 52 · 1 months ago

Airgapped closed-corpus QA testbed: a local Qwen agent in a Pi harness explores a .zip dataroom with grep/embeddings/rerankers under a token budget — a bed to study search as test-time compute.

superlinked 2.8k · 10 days ago

Self-hosted inference cluster for everything agents call besides the big LLM: embeddings, rerankers, OCR, NER, guardrails and small LLMs — 100+ models, one OpenAI-compatible API, K8s stack included.

kennss 843 · 2 days ago

Sudoless Apple Silicon monitor: SwiftUI dashboard plus menu-bar suite tracking ANE, Media Engine and memory bandwidth Activity Monitor won't show — with DVR-style record & replay.

aovestdipaperino 582 · yesterday

Code-intelligence MCP server for coding agents — a pre-indexed semantic graph (libSQL + FTS5) they query instead of grepping: symbols, callers, impact radius in one call. 100% local, 50+ languages.

vllm-project 89.5k · today

High-throughput, memory-efficient inference and serving engine for LLMs.

xberg-io 9.2k · yesterday

Rust-core document-intelligence engine with 15 language bindings: turns 96 formats — PDF, Office, images, audio, code — into clean text, tables and RAG-ready chunks. Library, CLI, REST or MCP.