Language-Prompted Video Segmentation + Multi-Object Tracking
Segment and track objects through a video using a short text phrase like "person" or "red car". SAM 3.1's Object Multiplex design tracks up to 16 objects jointly in one pass, with ~7× faster multi-object inference vs. SAM 3 at scale and improved VOS performance on 6 of 7 benchmarks.
What it does
SAM 3.1 (Video) extends SAM 3.1's open-vocabulary, text-prompted segmentation to video - every matching object is detected, masked, and tracked across frames with persistent IDs. Type a short concept ("person", "red car", "dog") and the model returns per-instance masks for every frame in the clip. The headline change vs. SAM 3 is Object Multiplex, a shared-memory design that lets the model track many objects jointly in a single forward pass instead of running an independent tracker per object. The result is ~7× faster inference at 128 objects on a single H100 GPU, with no accuracy loss and improved VOS (Video Object Segmentation) performance on 6 of 7 standard benchmarks. The studio node tracks up to 16 objects per pass at ~1 – 3 minutes per 200 frames on typical hardware.
Problem it solves
- Track many objects at once, cheaply – Object Multiplex tracks up to 16 objects (128 internally on bigger hardware) in a single pass, far more efficient than running one tracker per object
- Language-driven video search – Find and isolate every "person", "red car", "dog", etc., across an entire video clip with a single text prompt
- Editing & VFX rotoscoping – Get per-object masks per frame for compositing, blur-out, color grading, or style transfer without manual rotoscoping
- Surveillance & analytics – Automatically track people, vehicles, or other concepts through a scene with consistent IDs
- Sports & broadcast tagging – Track all players, balls, or referees frame-by-frame with one query
- Privacy filtering in video – Continuously blur faces, license plates, or screens across an entire clip
- Dataset generation – Bootstrap labeled video segmentation datasets from minimal text prompts
Input/Output
- Input: Video clip + text prompt (or visual prompts)
- Format: Standard video formats (MP4, MOV, etc.); also accepts a frame sequence
- Prompts: Short text phrases ("person", "red car", "dog") or visual prompts (points, boxes, masks) on a reference frame
- Object capacity: Up to 16 objects tracked jointly per pass in the studio node
- Options:
- Merge to binary mask video (checkbox) - when on, all per-instance masks are flattened into a single binary mask video for the concept (useful for blur-out / privacy / single-class workflows)
- Output: Per-frame segmentation masks with persistent instance IDs
- Masks: Binary or confidence masks for each matching object, per frame, with consistent IDs across the clip (or a single merged binary mask video when "Merge to binary mask video" is on)
- Tracks: Bounding boxes and object boundaries over time
- Confidence scores: Per-mask, per-frame
Performance
Inference time | ~1 – 3 minutes per 200 frames on typical hardware |
Multi-object scaling | ~7× faster at 128 objects on a single H100 GPU vs. SAM 3 — no accuracy loss |
VOS benchmarks | Improved on 6 of 7 standard Video Object Segmentation benchmarks vs. SAM 3 |
Studio node capacity | Up to 16 objects tracked jointly per pass |
Concept vocabulary | 4M+ concepts (inherited from SAM 3) |
Tracking robustness | Persistent IDs across frames, including under occlusion |
Technical Details
Architecture | Promptable video segmentation + tracking foundation model |
Family | SAM 3 (Segment Anything with Concepts) — SAM 3.1 Video is the Object Multiplex refresh |
Key innovation in SAM 3.1 | Object Multiplex — shared-memory approach for joint multi-object tracking; runs all object trackers in one pass through a shared memory rather than independent runs per object |
Speed result | ~7× faster inference at 128 objects on a single H100 GPU (no accuracy sacrifice) |
Quality result | Improved VOS scores on 6 of 7 standard benchmarks |
Components | Vision Encoder → text-conditioned Detector (DETR-based) → Presence Head → shared memory for cross-frame tracking |
Inference Pipeline | Encode frames → process text prompt → detect & track all matching objects across frames via Object Multiplex shared memory → return masks, coordinates, and per-frame IDs |
Concept Coverage | 4M+ unique concepts (inherited from SAM 3) |
Training Data | SAM 3 large-scale concept-segmentation dataset + video tracking data with hard negatives |
Framework | PyTorch — checkpoints distributed via HF; runtime via the SAM 3 GitHub repo (no Transformers integration) |
Compliance & Provenance
Provider | Meta (open) |
Provider type | Specialized |
License | Others |
EU AI Act risk class | Minimal Risk |
Art. 50 transparency | Not applicable |
Region availability | Available globally |
Training data summary |
For more on how we classify models and mark outputs, see our AI Policy.
Model Source
- Huggingface: https://huggingface.co/facebook/sam3.1
- License: others
- Important: You are not the target of Trade Controls and your use of SAM Materials must comply with Trade Controls. You agree not to use, or permit others to use, SAM Materials for any activities subject to the International Traffic in Arms Regulations (ITAR) or end uses prohibited by Trade Controls, including those related to military or warfare purposes, nuclear industries or applications, espionage, or the development or use of guns or illegal weapon.