ai-avatar-system

Self-hosted digital human: a photo plus 10s of voice becomes a real-time talking head — Whisper to LLM to Chatterbox TTS to MuseTalk lip-sync, streamed over WebSocket, with barge-in.

451 101 Python MITupdated 1 months ago

voice video

View on GitHubDispute this mapping →

Curator's take

Pick it when you need the whole product rather than the pipeline: JWT auth, Postgres and Alembic, per-user rate limiting, Prometheus, a pytest suite and one-command GPU deploy are what separate it from the research demos in this space. The price is a GPU — MuseTalk at 30 FPS wants something like a g5.xlarge and the FFmpeg CPU fallback is a slideshow. Weigh the obvious too: a photorealistic face plus a voice cloned from ten seconds is a deepfake engine with a login page, and consent is entirely your problem.

Mapped by ShipWithAI editors · links verified

README.md

🎭 AvatarAI — Real-Time AI Avatar Platform

Upload a photo · Clone a voice · Talk to any face in real time

Quick Start · Features · Architecture · GPU / AWS Deploy · API · Roadmap

The most complete open-source AI avatar / digital human system. Real-time talking-head lip-sync · Zero-shot voice cloning · Multi-LLM · Runs 100% locally or on AWS.

🎬 What is AvatarAI?

AvatarAI is an open-source, production-ready platform for building photorealistic AI avatar conversations. Upload any face photo, clone a voice from a 5-second audio clip, and have a real-time conversation — with lip-sync video generated on every single response.

[mic] → Whisper STT → Claude / GPT / Ollama (streaming) → Chatterbox TTS → MuseTalk lip-sync → [video]
                              < 2–4 s to first video chunk on AWS GPU >

What makes AvatarAI different:

🎤 Zero-shot voice cloning — 10 seconds of audio is all you need (Chatterbox Multilingual)
🎭 Any face, any language — upload a JPEG, pick from 23 languages, start talking
⚡ Token-streaming pipeline — the LLM streams live tokens while TTS + lip-sync run per sentence; the first video chunk plays before the model finishes its reply
✋ Barge-in — speak (or hit stop) mid-reply and the avatar yields instantly, like a real conversation
🔒 100% local mode — local storage, local Whisper, local LLM via Ollama: nothing leaves your machine
🔌 Multi-LLM — Claude (with prompt caching), GPT-4o, or any local model via Ollama / vLLM / LM Studio
🚀 AWS GPU deployment — one-command deploy to g5.xlarge for true real-time (~30 FPS)
🏗️ Production-grade — JWT + httpOnly-cookie auth, per-user rate limiting, Postgres + Alembic, S3/CloudFront, Prometheus, CI, a real test suite — the only project in this niche you can ship as a product, not just a demo

⚔️ How AvatarAI compares

	AvatarAI	Duix-Avatar	Linly-Talker	AIAvatarKit
Real-time conversation	✅ WebSocket streaming	❌ offline video gen	✅ (Gradio / WebRTC spin-off)	✅
Lip-sync video	✅ MuseTalk V1.5	✅ proprietary models	✅ multiple engines	❌ (drives external avatars)
Voice cloning	✅ 10 s, 23 languages	✅	✅	❌
Barge-in / interr

Continue your stack

What teams reach for next — and why each earns a place beside ai-avatar-system. Ranked by curator confidence.

Pairs well with

LuxTTS

Lightweight voice-cloning TTS — 48kHz speech at 150x realtime, fits in 1GB VRAM and runs on CPU or MPS. SOTA cloning from a ~3s reference sample, rivaling models 10x larger.

Why it fitsThe TTS slot is the GPU contention point when MuseTalk is already resident. LuxTTS clones from ~3s of reference and fits in 1GB VRAM, leaving the rest of the card for lip-sync.

vLLM

High-throughput, memory-efficient inference and serving engine for LLMs.

Why it fitsThe LLM slot accepts any OpenAI-compatible server. vLLM is what makes the local path fast enough that the first video chunk still arrives before the model finishes its reply.

pocket-tts

Kyutai's 100M-parameter CPU-only TTS — streaming audio in ~200ms, ~6× real-time on two laptop cores, voice cloning, six languages. pip install and it talks.

Why it fitsFor the CPU-only deployment path, Chatterbox is the bottleneck. A 100M CPU TTS streaming in ~200ms keeps the turn conversational even when there is no GPU to spare.

Alternative to See all + compare →

Open-LLM-VTuber

Hands-free voice conversation with any LLM behind a Live2D animated face — voice interruption included, running fully local and cross-platform.

Why it fitsBoth put an animated face on a local voice conversation with interruption support. Open-LLM-VTuber drives a Live2D character and runs comfortably on a laptop; AvatarAI generates lip-synced video of a real photo and ships auth, Postgres and multi-user deployment.

speech-to-speech

Modular local voice-agent pipeline — VAD→STT→LLM→TTS behind an OpenAI Realtime-compatible WebSocket; every stage swappable, the LLM slot takes any OpenAI-compatible server. Powers Reachy Mini robots.

Why it fitsThe identical VAD to STT to LLM to TTS pipeline underneath. speech-to-speech is a modular library with every stage swappable behind an OpenAI Realtime-compatible socket; AvatarAI is an opinionated full stack that adds the video layer and a web app around it.