🦋 Molt
An agentic-first RL framework for research.
Ray · vLLM · NVIDIA AutoModel — the smallest PyTorch-native stack for 1T-class fully-async, multimodal, multi-turn agentic RL.
Paper · Architecture · Why Molt · Quick Start · Agent Contract · Recipes · Scaling
| Package | SFT | RL | Runtime |
|---|---|---|---|
molt |
molt.cli.train_sft |
molt.cli.train_rl_ray |
vLLM |
Molt is agentic-first and PyTorch-native. The agent is the program;
the trainer is a single actor; reward is any Python you write inside an Env
or ChatAgent — graders, multi-turn tools, VLM environments, LLM-as-judge.
Three components carry the rest — Ray for placement and async queues,
vLLM for rollout, NVIDIA AutoModel + FSDP2 for training in pure
PyTorch. That is the whole stack: ~9.2K lines of RL code that scale to
1T-class MoE on vLLM with TP / EP / CP — think DeepSeek-V3 at
--fsdp.ep_size 256, Adam CPU offload for the largest actors. One agent
API, one trainable actor, clean enough to read end-to-end.
🧩 Architecture
Three boxes. One async loop.
Ray owns placement and the async queue between the three boxes — that is the entire runtime. The contract is token-first: token ids, logprobs, action ranges, rewards, and multimodal tensors stay aligned from rollout to training. Anything you can compute in Python is a valid reward, including LLM-as-judge calls back through the same vLLM engines that drive rollout.
✨ Why Molt
| What you get | Why it matters for research | |
|---|---|---|
| 🤖 Agentic-first | One Gymnasium-aligned API — Env.step() or ChatAgent.run() — covers graders, multi-turn tools, VLM environments, and OpenAI/Anthropic-compatible servers |
The agent is the program — iterate on environments in plain Python, the trainer stays untouched |
| ⚙️ Fully-async runtime | Ray placement, async rollout queues, vLLM engines, partial rollout, weight sync | Rollout, training, and weight sync overlap — a DeepSeek-V3-class actor stays fed without bespoke infra |
| 🔥 PyTorch-native, AutoModel-first | FSDP2 + NVIDIA AutoModel, pure PyTorch end-to-end | Hack the model in the language you already write; no backend ceremony |
| 🎯 Single-actor simplicity | One actor, optional KL reference — the whole RL graph fits on a page | Every gradient is explicit; every loss term is one file away |
| 🚀 Frontier-scale MoE | AutoModel + FSDP2 + TP / EP / CP + Adam CPU offload, MoE-native — e.g. DeepSeek-V3 with --fsdp.ep_size 256 |
The same script that trains 8B scales to 1T-class MoE — no rewrite between scales |
| 🔗 Token-first contract | Aligned token ids, logprobs, action ranges, rewards, multimodal ten |