Model-Optimizer

NVIDIA's model-compression library: quantization (PTQ/QAT, FP8/NVFP4), pruning, distillation, NAS and speculative decoding over HF/PyTorch/ONNX, exported to TensorRT-LLM, vLLM and SGLang.

3,652 574 Python Apache-2.0updated today

training local

View on GitHubDispute this mapping →

Curator's take

The default answer to 'make this checkpoint cheaper to serve' if you deploy on NVIDIA hardware — the FP8/NVFP4 recipes here are the ones TensorRT-LLM and vLLM actually consume, and QAT recovers accuracy that naive post-training quantization throws away. It is deliberately vendor-shaped: the interesting formats want Hopper or Blackwell, and it optimizes checkpoints rather than serving them, so you still need an engine underneath. Wrong tool for training from scratch, for CPU or Apple-silicon targets, or for anyone who just wants a GGUF.

Mapped by ShipWithAI editors · links verified

README.md

NVIDIA Model Optimizer

Documentation | Roadmap

NVIDIA Model Optimizer (referred to as Model Optimizer, or ModelOpt) is a library comprising state-of-the-art model optimization techniques including quantization, pruning, Neural Architecture Search (NAS), distillation, speculative decoding and sparsity to accelerate models.

[Input] Model Optimizer currently supports inputs of a Hugging Face, PyTorch or ONNX model.

[Optimize] Model Optimizer provides Python APIs for users to easily compose the above model optimization techniques and export an optimized quantized checkpoint. Model Optimizer is also integrated with NVIDIA Megatron-Bridge, Megatron-LM and Hugging Face Accelerate for training required inference optimization techniques.

[Export for deployment] Seamlessly integrated within the NVIDIA AI software ecosystem, the quantized checkpoint generated from Model Optimizer is ready for deployment in downstream inference frameworks like SGLang, TensorRT-LLM, TensorRT, or vLLM. The unified Hugging Face export API now supports both transformers and diffusers models.

Latest News

[2026/08/17] BLOG: Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer: Learn how quantization-aware distillation recovers accuracy from aggressive NVFP4 quantization while reducing model size and increasing throughput.
[2026/06/26] BLOG: Creating the NVIDIA Nemotron 3 Ultra NVFP4 Checkpoint with NVIDIA Model Optimizer: How we quantized Nemotron 3 Ultra (550B) to NVFP4 with Model Optimizer — up to 5.9× higher decode-heavy inference throughput than GLM-5.1 754B FP4 while matching BF16 accuracy. NVFP4 Checkpoint on Hugging Face.
[2026/05/27] End-to-end Optimization tutorial for Nemotron-3-Nano-30B-A3B: Pruning + two-phase distillation + FP8 quantization achieving 2.6× vLLM throughput and 2.6× memory reduction.
[2026/05/13] Puzzletron: A new algorithm for heterogeneous pruning & NAS of LLM and VLM models.
[2026/04/15] Customer story: Domyn compresses Colosseum-355B → 260B using ModelOpt's Minitron pruning + distillation
[2026/03/17] Customer story: Bielik.AI builds Bielik Minitron 7B (33% smaller, 50% faster, 90% quality retained) using ModelOpt's Minitron pruning + distillation
[2026/03/11] Model Optimizer quantized Nemotron-3-Super checkpoints are available on Hugging Face for downl

Continue your stack

What teams reach for next — and why each earns a place beside Model-Optimizer. Ranked by curator confidence.

Pairs well with

vLLM

High-throughput, memory-efficient inference and serving engine for LLMs.

Why it fitsModelOpt is the compress half and vLLM the serve half of the same pipeline: quantized checkpoints export directly into vLLM, which is where the FP8/NVFP4 throughput win actually shows up.

trl

Hugging Face's post-training library: SFT, DPO, GRPO, KTO and reward-model trainers on top of Transformers — from a Colab LoRA run to multi-GPU deployments.

Why it fitsTRL post-trains the model (SFT, DPO, GRPO); ModelOpt compresses the result for deployment. Quantization-aware training sits between them when plain PTQ costs too much accuracy.

LlamaFactory

The unified fine-tuning framework: 100+ LLMs and VLMs via LoRA/QLoRA/full-parameter, config-driven or through the LlamaBoard GUI. ACL 2024, 1000+ citations, 73k stars.

Why it fitsOpposite ends of one pipeline: fine-tune with LLaMA-Factory's config-driven LoRA/QLoRA runs, then prune and quantize with ModelOpt before it goes anywhere near production.

Alternative to See all + compare →

airllm

Layer-by-layer inference that runs 70B models on a 4GB GPU — no quantization required; 405B on 8GB, DeepSeek-V3 671B on ~12GB. One AutoModel line for most open model families.

Why it fitsBoth make an oversized model fit on hardware it shouldn't. AirLLM streams layers and explicitly refuses to quantize, preserving quality at the cost of latency; ModelOpt compresses the weights and keeps throughput.