Speech To Speech: Build voice agents with open-source models

A low-latency, fully modular voice-agent pipeline: VAD -> STT -> LLM -> TTS, exposed through an OpenAI Realtime-compatible WebSocket API. Every component is swappable. The LLM slot speaks OpenAI-compatible protocols, so you can point it at a hosted provider, at HF Inference Providers, or at a vLLM or llama.cpp server on your own hardware for a fully local, fully open stack.

This pipeline runs in production as the conversation backend for thousands of Reachy Mini robots.

Quickstart

pip install speech-to-speech
export OPENAI_API_KEY=...
speech-to-speech

This starts an OpenAI Realtime-compatible server at ws://localhost:8765/v1/realtime using Parakeet TDT for local STT, an OpenAI-compatible LLM, and Qwen3-TTS for local speech output.

From a source checkout, talk to it from a second terminal:

python scripts/listen_and_play_realtime.py --host 127.0.0.1 --port 8765

Prefer to keep the LLM on your own machine? Serve Gemma 4 with llama.cpp:

llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full

Then point the OpenAI-compatible LLM backend at it:

speech-to-speech \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""

Any OpenAI Realtime-compatible client can connect. See Realtime API for the protocol and LLM backends for provider and local-server options.

Index

• How it works
• Installation
• Supported components
• Run modes
• Realtime API
• LLM backends
• Multi-language support
• Pocket TTS
• CLI reference
• Contributing
• Star history
• Citations

How it works

The pipeline is a cascade of four components, each running in its own thread and connected by queues:

1. Voice Activity Detection (VAD): Silero VAD v5 detects speech boundaries and turn-taking.
2. Speech to Text (STT): transcribes the user's turn, with optional live partial transcripts.
3. Language Model (LLM): generates the response, streaming text and tool calls.
4. Text to Speech (TTS): synthesizes audio and streams it back to the client.

Every stage has multiple interchangeable backends, selected via CLI flags. The code is designed for easy modification, with a focus on models available through Transformers and the Hugging Face Hub.

Installation

Requires Python 3.10+.

pip install speech-to-speech

The default install covers the standard realtime path:

• Parak