huggingface

trl

Hugging Face's post-training library: SFT, DPO, GRPO, KTO and reward-model trainers on top of Transformers — from a Colab LoRA run to multi-GPU deployments.

18,852 2,845 Python Apache-2.0updated today

training

View on GitHub Dispute this mapping →

Curator's take

The on-ramp for post-training: if your model is on the Hub and your job fits SFT/DPO/GRPO/KTO, a Trainer class gets you a running job in an afternoon — and nothing else scales down to a free Colab as gracefully. PEFT/LoRA, quantized training and accelerate multi-GPU come along for free. NOT for frontier-scale RL dataflows (that's verl/slime territory — no Megatron, no disaggregated rollout engines), and the trainer abstraction that makes it easy also hides the loss mechanics: when results surprise you, read the trainer source before blaming the data.

Mapped by ShipWithAI editors · links verified

README.md

TRL - Transformers Reinforcement Learning

A comprehensive library to post-train foundation models

🎉 What's New

🌍 Multi-environment agentic RL: GRPOTrainer now supports per-example environment selection and environment-owned rewards — mix multiple sandboxed task suites in one run and let each environment define its own scoring, with Harbor and OpenEnv.

🎯 KTO is now stable: KTOTrainer graduates to the stable API after a full alignment pass with DPOTrainer.

Overview

TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the 🤗 Transformers ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.

Highlights

Trainers: Various fine-tuning methods are easily accessible via trainers like SFTTrainer, GRPOTrainer, DPOTrainer, KTOTrainer and more.
Efficient and scalable:
- Leverages 🤗 Accelerate to scale from single GPU to multi-node clusters using methods like DDP and DeepSpeed.
- Full integration with 🤗 PEFT enables training on large models with modest hardware via quantization and LoRA/QLoRA.
- Integrates 🦥 Unsloth for accelerating training using optimized kernels.
Command Line Interface (CLI): A simple interface lets you fine-tune with models without needing to write code.

Installation

Python Package

Install the library using pip:

pip install trl

From source

If you want to use the latest features before an official release, you can install TRL from source:

pip install git+https://github.com/huggingface/trl.git

Repository

If you want to use the examples you can clone the repository with the following command:

git clone https://github.com/huggingface/trl.git

Quick Start

For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL

Continue your stack

What teams reach for next — and why each earns a place beside trl. Ranked by curator confidence.

Pairs well with

vLLM

High-throughput, memory-efficient inference and serving engine for LLMs.

Why it fitsTRL's online RL trainers (GRPO) generate through vLLM — the use_vllm flag is the standard throughput path.

Alternative to See all + compare →

LlamaFactory

The unified fine-tuning framework: 100+ LLMs and VLMs via LoRA/QLoRA/full-parameter, config-driven or through the LlamaBoard GUI. ACL 2024, 1000+ citations, 73k stars.

Why it fitsThe two dominant open fine-tuning stacks: TRL is the code-first Hugging Face library; LlamaFactory the config-driven unified trainer with a GUI and a wider model-coverage matrix.

h2o-llmstudio

H2O's no-code GUI and framework for fine-tuning LLMs — LoRA, 8-bit, DPO and experiment tracking behind a web UI, with CLI and Docker paths for the same configs.

Why it fitsThe same LoRA/DPO fine-tuning jobs behind different interfaces: TRL is the code-first library for Hub-native workflows, LLM Studio the no-code GUI for teams that don't write training loops.

verl

ByteDance's RL post-training library (HybridFlow): PPO/GRPO dataflows in a few lines, FSDP/Megatron training with vLLM/SGLang rollouts, production-proven at frontier scale.

Why it fitsSame goal — a post-trained model — at opposite ends of the infra spectrum: TRL for Hub-native single-node SFT/DPO/GRPO, verl when the job needs disaggregated rollout engines and multi-node RL dataflows.