StackMap
Subscribe
Explore / sam2
facebookresearch

sam2

Meta's Segment Anything 2: promptable zero-shot segmentation for images and video with streaming memory — click and box prompts become tracked masks in real time.

19,539 2,507 Jupyter Notebook Apache-2.0updated 1 months ago
vision
View on GitHubDispute this mapping →
Curator's take

Zero-shot masks with no training data: click a thing or pass a detector's box and SAM 2 segments it — and in video, tracks it across frames via streaming memory. Unbeatable for annotation pipelines, data engines and any 'segment whatever the user points at' feature. NOT a classifier — it draws masks but never names them (pair it with a detector for labels), and it's a research-grade repo: checkpoints and notebooks, not a serving stack; plan your own inference infrastructure and a real GPU for video work.

Mapped by ShipWithAI editors · links verified
README.md

SAM 2: Segment Anything in Images and Videos

AI at Meta, FAIR

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer

[Paper] [Project] [Demo] [Dataset] [Blog] [BibTeX]

SAM 2 architecture

Segment Anything Model 2 (SAM 2) is a foundation model towards solving promptable visual segmentation in images and videos. We extend SAM to video by considering images as a video with a single frame. The model design is a simple transformer architecture with streaming memory for real-time video processing. We build a model-in-the-loop data engine, which improves model and data via user interaction, to collect our SA-V dataset, the largest video segmentation dataset to date. SAM 2 trained on our data provides strong performance across a wide range of tasks and visual domains.

SA-V dataset

Latest updates

12/11/2024 -- full model compilation for a major VOS speedup and a new SAM2VideoPredictor to better handle multi-object tracking

  • We now support torch.compile of the entire SAM 2 model on videos, which can be turned on by setting vos_optimized=True in build_sam2_video_predictor, leading to a major speedup for VOS inference.
  • We update the implementation of SAM2VideoPredictor to support independent per-object inference, allowing us to relax the assumption of prompting for multi-object tracking and adding new objects after tracking starts.
  • See RELEASE_NOTES.md for full details.

09/30/2024 -- SAM 2.1 Developer Suite (new checkpoints, training code, web demo) is released

  • A new suite of improved model checkpoints (denoted as SAM 2.1) are released. See Model Description for details.
    • To use the new SAM 2.1 checkpoints, you need the latest model code from this repo. If you have installed an earlier version of this repo, please first uninstall the previous version via pip uninstall SAM-2, pull the latest code from this repo (with git pull), and then reinstall the repo following Installation below.
  • The training (and fine-tuning) code has been released. See training/README.md on how to get started.
  • The frontend + backend code for the SAM 2 web demo has been released. See demo/README.md for details.

Installation

SAM 2 needs to be installed first before use. The code requires python>=3.10, as well as torch>=2.5.1 and torchvision>=0.20.1. Please follow the instructions here to install both PyTorch and TorchVision dep

Continue your stack

What teams reach for next — and why each earns a place beside sam2. Ranked by curator confidence.