[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"readme:trl":3},"\u003Ch1>TRL - Transformers Reinforcement Learning\u003C\u002Fh1>\n\u003Cdiv>\n    \u003Cpicture>\n        \u003Csource media=\"(prefers-color-scheme: light)\" srcset=\"https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ftrl-lib\u002Fdocumentation-images\u002Fresolve\u002Fmain\u002Ftrl_banner_light.png\">\u003C\u002Fsource>\n        \u003Cimg src=\"https:\u002F\u002Fhuggingface.co\u002Fdatasets\u002Ftrl-lib\u002Fdocumentation-images\u002Fresolve\u002Fmain\u002Ftrl_banner_dark.png\" alt=\"TRL Banner\" \u002F>\n    \u003C\u002Fpicture>\n\u003C\u002Fdiv>\u003Chr \u002F> \u003Cbr \u002F>\u003Ch3>\n    \u003Cp>A comprehensive library to post-train foundation models\u003C\u002Fp>\n\u003C\u002Fh3>\u003Cp align=\"center\">\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Fblob\u002Fmain\u002FLICENSE\" rel=\"nofollow ugc noopener\">\u003Cimg alt=\"License\" src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Flicense\u002Fhuggingface\u002Ftrl.svg?color=blue\" \u002F>\u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Findex\" rel=\"nofollow ugc noopener\">\u003Cimg alt=\"Documentation\" src=\"https:\u002F\u002Fimg.shields.io\u002Fwebsite?label=documentation&url=https%3A%2F%2Fhuggingface.co%2Fdocs%2Ftrl%2Findex&down_color=red&down_message=offline&up_color=blue&up_message=online\" \u002F>\u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl\u002Freleases\" rel=\"nofollow ugc noopener\">\u003Cimg alt=\"GitHub release\" src=\"https:\u002F\u002Fimg.shields.io\u002Fgithub\u002Frelease\u002Fhuggingface\u002Ftrl.svg\" \u002F>\u003C\u002Fa>\n    \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Ftrl-lib\" rel=\"nofollow ugc noopener\">\u003Cimg alt=\"Hugging Face Hub\" src=\"https:\u002F\u002Fimg.shields.io\u002Fbadge\u002F🤗%20Hub-trl--lib-yellow\" \u002F>\u003C\u002Fa>\n\u003C\u002Fp>\u003Ch2>🎉 What's New\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>🌍 Multi-environment agentic RL:\u003C\u002Fstrong> \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fgrpo_trainer\" rel=\"nofollow ugc noopener\">\u003Ccode>GRPOTrainer\u003C\u002Fcode>\u003C\u002Fa> now supports per-example environment selection and environment-owned rewards — mix multiple sandboxed task suites in one run and let each environment define its own scoring, with \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fharbor\" rel=\"nofollow ugc noopener\">Harbor\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fopenenv\" rel=\"nofollow ugc noopener\">OpenEnv\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>🎯 KTO is now stable:\u003C\u002Fstrong> \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fkto_trainer\" rel=\"nofollow ugc noopener\">\u003Ccode>KTOTrainer\u003C\u002Fcode>\u003C\u002Fa> graduates to the stable API after a full alignment pass with \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fdpo_trainer\" rel=\"nofollow ugc noopener\">\u003Ccode>DPOTrainer\u003C\u002Fcode>\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch2>Overview\u003C\u002Fh2>\n\u003Cp>TRL is a cutting-edge library designed for post-training foundation models using advanced techniques like Supervised Fine-Tuning (SFT), Group Relative Policy Optimization (GRPO), and Direct Preference Optimization (DPO). Built on top of the \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftransformers\" rel=\"nofollow ugc noopener\">🤗 Transformers\u003C\u002Fa> ecosystem, TRL supports a variety of model architectures and modalities, and can be scaled-up across various hardware setups.\u003C\u002Fp>\n\u003Ch2>Highlights\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Cp>\u003Cstrong>Trainers\u003C\u002Fstrong>: Various fine-tuning methods are easily accessible via trainers like \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fsft_trainer\" rel=\"nofollow ugc noopener\">\u003Ccode>SFTTrainer\u003C\u002Fcode>\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fgrpo_trainer\" rel=\"nofollow ugc noopener\">\u003Ccode>GRPOTrainer\u003C\u002Fcode>\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fdpo_trainer\" rel=\"nofollow ugc noopener\">\u003Ccode>DPOTrainer\u003C\u002Fcode>\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Ftrl\u002Fkto_trainer\" rel=\"nofollow ugc noopener\">\u003Ccode>KTOTrainer\u003C\u002Fcode>\u003C\u002Fa> and more.\u003C\u002Fp>\n\u003C\u002Fli>\n\u003Cli>\u003Cp>\u003Cstrong>Efficient and scalable\u003C\u002Fstrong>:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Leverages \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Faccelerate\" rel=\"nofollow ugc noopener\">🤗 Accelerate\u003C\u002Fa> to scale from single GPU to multi-node clusters using methods like \u003Ca href=\"https:\u002F\u002Fpytorch.org\u002Ftutorials\u002Fintermediate\u002Fddp_tutorial.html\" rel=\"nofollow ugc noopener\">DDP\u003C\u002Fa> and \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fdeepspeedai\u002FDeepSpeed\" rel=\"nofollow ugc noopener\">DeepSpeed\u003C\u002Fa>.\u003C\u002Fli>\n\u003Cli>Full integration with \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fpeft\" rel=\"nofollow ugc noopener\">🤗 PEFT\u003C\u002Fa> enables training on large models with modest hardware via quantization and LoRA\u002FQLoRA.\u003C\u002Fli>\n\u003Cli>Integrates \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Funslothai\u002Funsloth\" rel=\"nofollow ugc noopener\">🦥 Unsloth\u003C\u002Fa> for accelerating training using optimized kernels.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003Cli>\u003Cp>\u003Cstrong>Command Line Interface (CLI)\u003C\u002Fstrong>: A simple interface lets you fine-tune with models without needing to write code.\u003C\u002Fp>\n\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>Installation\u003C\u002Fh2>\n\u003Ch3>Python Package\u003C\u002Fh3>\n\u003Cp>Install the library using \u003Ccode>pip\u003C\u002Fcode>:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">pip install trl\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>From source\u003C\u002Fh3>\n\u003Cp>If you want to use the latest features before an official release, you can install TRL from source:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">pip install git+https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl.git\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>Repository\u003C\u002Fh3>\n\u003Cp>If you want to use the examples you can clone the repository with the following command:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\">git clone https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl.git\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>Quick Start\u003C\u002Fh2>\n\u003Cp>For more flexibility and control over training, TRL provides dedicated trainer classes to post-train language models or PEFT adapters on a custom dataset. Each trainer in TRL\u003C\u002Fp>\n",1784240408281]