SAM 2 for Image Annotation: Point Prompts and Polygon Code

Manual polygonal segmentation can be time-consuming, especially around irregular boundaries and occluded objects.

With Meta AI's Segment Anything Model 2 (SAM 2), annotators can propose segmentation masks using point clicks or bounding box prompts. Runtime and mask quality depend on the model, image, and hardware; inspect the result before using it as a label.

This guide shows a Python example that predicts a mask, extracts polygon coordinates suitable for a COCO segmentation field, and prints YOLO segmentation lines. It does not assemble a COCO dataset or write label files.

---

1. Why SAM 2 Changes Dataset Annotation

SAM 2 accepts positive and negative point prompts and can be used with a box prompt. The example below uses points only. Check mask quality on your own images; speed and accuracy depend on the checkpoint and hardware.

---

2. Python Implementation: Running SAM 2 Inference and Exporting Polygons

Follow the official SAM 2 installation and checkpoint instructions first. Install the repository's sam2 package, download a checkpoint and matching config, then adjust the paths and point coordinates in this example. The script predicts a binary mask and extracts polygon coordinates using OpenCV:

#!/usr/bin/env python3
"""
sam2_annotation_pipeline.py
Runs SAM 2 point prompting and converts predicted binary masks into COCO/YOLO-seg polygons.
Requirements: install the official SAM 2 package and checkpoint; pip install opencv-python numpy
"""

import cv2
import numpy as np
import torch
from sam2.build_sam import build_sam2
from sam2.sam2_image_predictor import SAM2ImagePredictor


def mask_to_polygons(mask: np.ndarray, epsilon: float = 1.5) -> list:
    """
    Converts a binary boolean mask into a simplified polygon coordinate list.
    """
    mask_uint8 = (mask * 255).astype(np.uint8)
    contours, _ = cv2.findContours(
        mask_uint8, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE
    )

    polygons = []
    for contour in contours:
        # Filter tiny noise contours
        if cv2.contourArea(contour) < 20:
            continue
        
        # Approximate contour to reduce vertex count while preserving shape
        approx = cv2.approxPolyDP(contour, epsilon, closed=True)
        if len(approx) >= 3:
            polygon = approx.flatten().tolist()
            polygons.append(polygon)

    return polygons


def polygon_to_yolo_seg(polygon: list, img_width: int, img_height: int) -> str:
    """
    Normalizes polygon coordinates [x1, y1, x2, y2, ...] to 0.0-1.0 for YOLOv8-seg.
    """
    normalized = []
    for i in range(0, len(polygon), 2):
        nx = polygon[i] / img_width
        ny = polygon[i + 1] / img_height
        normalized.extend([f"{nx:.6f}", f"{ny:.6f}"])
    return " ".join(normalized)


def main():
    # 1. Load SAM 2 Model
    checkpoint = "checkpoints/sam2.1_hiera_large.pt"
    model_cfg = "configs/sam2.1/sam2.1_hiera_l.yaml"
    device = "cuda" if torch.cuda.is_available() else "cpu"

    print(f"Loading SAM 2 model on {device}...")
    sam2_model = build_sam2(model_cfg, checkpoint, device=device)
    predictor = SAM2ImagePredictor(sam2_model)

    # 2. Load Input Image
    image_path = "sample.jpg"
    image_bgr = cv2.imread(image_path)
    if image_bgr is None:
        raise FileNotFoundError(image_path)
    image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
    h, w, _ = image_rgb.shape

    predictor.set_image(image_rgb)

    # 3. Provide Prompts (e.g. Foreground Click at [500, 350], Background Click at [620, 410])
    input_points = np.array([[500, 350], [620, 410]], dtype=np.float32)
    input_labels = np.array([1, 0], dtype=np.int32)  # 1 = Foreground, 0 = Background

    # 4. Predict Binary Mask
    masks, scores, logits = predictor.predict(
        point_coords=input_points,
        point_labels=input_labels,
        multimask_output=False
    )

    best_mask = masks[0]  # Shape: (H, W), boolean
    print(f"Mask prediction score: {scores[0]:.4f}")

    # 5. Extract Simplified Polygon Coordinates
    polygons = mask_to_polygons(best_mask, epsilon=2.0)
    print(f"Extracted {len(polygons)} polygon contours.")

    # 6. Format for YOLOv8/YOLOv11 Segmentation
    class_id = 0
    for poly in polygons:
        yolo_seg_str = polygon_to_yolo_seg(poly, w, h)
        yolo_line = f"{class_id} {yolo_seg_str}"
        print(f"YOLO-seg annotation line:\n{yolo_line}")


if __name__ == "__main__":
    main()

---

3. Integrating SAM 2 into Annotation Workflows

While local Python scripts are suitable for offline batch pipelines, interactive labeling requires real-time point adjustments on a canvas.

Click segmentation in LabelOp

LabelOp provides click segmentation inside its image annotation workspace. WebGPU accelerates this workflow and Select Similar on compatible devices. Inspect each proposal, refine the annotation with the available tools, and save it before review.

Local detection and classification models, including zero-shot options, and managed Cloud AI batches provide other prelabeling paths. Export the reviewed dataset in a supported format or train through your own Vast.ai account. The Python script above is a separate example, not a recording of the LabelOp export workflow.

---

4. Bounding Box vs. Polygon: When to Use SAM 2

• Use Bounding Boxes (Object Detection): When your downstream model is standard YOLOv8/v11 detection and speed is paramount. Read the full comparison in Polygon vs Bounding Box Annotation.
• Use SAM 2 Polygons (Instance Segmentation): For precision robotics, surface defect inspection, medical anatomy, or autonomous vehicle lane boundary mapping.

---

Summary

SAM 2 bridges the gap between bounding box speed and polygon precision. Try LabelOp Smart Annotation to use click segmentation, review annotations, and export your work.