Auto-Annotate Video with Detection and Object Tracking

Auto-annotation does not label a video for you.

It gives you a draft that is usually right, and wrong in predictable ways. The work is in finding those wrong parts quickly and fixing them without breaking the rest.

This guide explains how auto-annotation works in LabelOp Tracker, a free in-browser video annotation app from the LabelOp family, and how to review and correct its output with the keyboard. Every control named here exists in the app today.

Two stages: detection, then tracking

Auto-annotate in Tracker runs two separate steps on each frame.

1. Detection. A detector model looks at one frame and returns boxes, each with a class and a confidence score. It has no memory of the previous frame.

2. Tracking. ByteTrack links detections across frames into tracks. It predicts where each existing track should be next (a Kalman filter), then matches new detections to those predictions (Hungarian matching on box overlap).

Both steps run in your browser, with the detector on your GPU through WebGPU. Your video is never uploaded. The two stages fail differently:

Problem you see	Usually caused by
Box on something that is not an object	Detection (false positive)
Object missing on some frames	Detection (missed or low-confidence)
One object gets a new ID halfway through	Tracking (ID switch after occlusion or a gap)
Box jumps from one object to another	Tracking (wrong match in a crowd)
Wrong class	Detection

Knowing which stage caused a problem tells you which fix to use.

How confidence controls tracking

In Tracker, the Minimum confidence setting (default 0.35) does two jobs:

A detection must reach it to start a new track.
A detection at or above it is matched to existing tracks first.

Detections between 0.1 and your threshold are not thrown away. They can still keep an existing track alive if they match it and have the same class. This is the core idea of ByteTrack: a partly hidden person often gets a low score, but that weak detection is still useful for keeping the same ID.

A track that loses its detections is kept for up to 30 frames. If the object reappears within that window and matches, it keeps its ID. If it stays hidden longer, it comes back with a new ID, and you will see an ID change to fix.

Practical effect:

Lower confidence finds more objects and keeps tracks through occlusion, but starts more false tracks.
Higher confidence gives fewer false tracks but misses small or partly hidden objects.

Choosing a model

The model picker in the Setup tab shows each model's download size before anything downloads. Downloads happen once and are cached in the browser.

Model	Size	Use it when
RF-DETR nano (default)	~103 MB	Your objects are among the 80 COCO classes: people, vehicles, animals, common items
OWLv2 text prompts	~295 MB	Your objects are not COCO classes. Type what to look for, one per line, such as "traffic cone" or "forklift"
Your own model	Your file	You already trained a detector and exported it to ONNX

For your own model, upload an .onnx file and describe its output layout (YOLOv8, YOLOv5 or DETR style) and input settings. Tracker validates it with a dry run.

RF-DETR nano runs at a fixed 384 x 384 input, so check small and distant objects carefully in the review step.

Settings to check before a full run

Open More options under the model picker:

Test on this frame runs the model on the current frame only. Use it on a busy frame to tune confidence before you spend time on the whole clip.
Only detect these classes limits a COCO run to the classes you care about. Fewer classes means fewer false positives to delete later.
From frame / To frame sets the range. One run covers up to 2,000 frames.

Then click Start auto-annotate. You can cancel during the run; nothing replaces your timeline until the run completes.

One warning: Run again replaces the current AI result, including fixes you made to it. Tracker asks before anything changes, but plan your corrections after you settle on settings, not before.

Reviewing with the QA tab

After a run, open the QA tab. It scans the tracks and builds a review queue with four kinds of suggestions:

ID change: tracks ended and new ones started nearby. One object may have been given a new ID.
Duplicate: two tracks cover almost the same area. The same object may be labeled twice.
Sudden jump: a box moved or resized unusually between frames. It may have jumped to another object.
Unsure: the model was not confident about this object. The threshold for this flag is in Review settings.

These are suggestions, not errors. Keyboard flow in the QA tab:

Key	Action
`J` / `K`	Next / previous suggestion
`F`	Mark as fixed
`X`	Mark as not an issue

Filter by problem type to work through one kind at a time. ID changes first is a good order, because a wrong identity affects every frame of a track.

Each row has a View button that jumps the video to the flagged frame.

Fixing the common problems

Select a track by clicking its box on the video or its lane in the timeline. Most fixes then take one key or one click.

ID switch: one object, two tracks

The object got a new ID after an occlusion.

Note the two track IDs.
In the Tracks tab, open Bulk edit and use Merge one track into another.

If the same track has a gap (it disappears for a few frames and comes back with the same ID), select it and click the small + marker in the gap on its timeline lane. Tracker fills the gap with boxes that follow the motion in the video. Right-clicking a run on the timeline also offers to merge it with the previous run.

Identity swap: two objects, one track

The track jumped from one object to another, usually where they cross.

Step to the first frame where the box is on the wrong object.
Press S to split the track at the playhead.
If the second half belongs to an existing track, merge it into that one.

False positives

A shadow or a reflection was detected as an object.

If it is a short false track, select it and use the delete button on the timeline toolbar to remove the segment under the playhead. You can also hide a track while you review.
For a Duplicate item, the QA row offers a button to remove the duplicate track on the flagged frames.
If only one frame is wrong, press Delete to remove the box on that frame only.
If a section of the clip is mostly noise, Shift-drag on the timeline to select the range, then delete everything inside it.
If a whole class is noise (for example, every "bench"), use Delete every object of a class in Bulk edit. It asks you to confirm.

Wrong class

Select the track and change its class in the selection bar. The change applies to every frame. To relabel a class everywhere, use Change every object of one class to another in Bulk edit.

Loose or drifting boxes

With a track selected:

Drag the box, or nudge it with the arrow keys (Shift for bigger steps).
Redraw the box on a frame to add a keyframe. Frames between keyframes are filled in.

Missed objects

Draw them by hand with 2 (box), 3 (polygon) or 4 (ellipse). Hand-drawn tracks are labeled M1, M2 and so on, so you can tell them apart from AI tracks. Turn on Follow video motion so boxes between your keyframes follow the video instead of moving in a straight line.

On hard frames, Segment assist (5, SAM 2.1) proposes a mask from a click or box, and Select similar (6, DINOv2) finds similar objects on the current frame. Both download their models only after you agree.

A keyboard-first correction loop

Once the shortcuts are familiar, a review pass looks like this:

J to the next suggestion. The video jumps to that frame.
← / → to step frames and see what happened (with no track selected).
Click the track, then S to split, Delete to remove a box, or arrow keys to nudge.
F to mark it fixed, or X if it was fine.
Ctrl+Z (⌘+Z on Mac) if you went too far.

Space plays and pauses from anywhere. Canvas shortcuts need the video focused, so click it once after working in the side panel.

Measuring tracking quality

If you have ground truth in MOT format (frame,id,x,y,w,h), paste or load it under Compare with ground truth in the QA tab. Tracker reports MOTA, HOTA, IDF1, ID switches (IDSW) and MOTP.

Use it to compare settings on the same clip. It is a check on your clip, not a general benchmark of the model.

When auto-annotation is not worth it

Skip it when the clip is only a few frames long, or when your objects are neither COCO classes nor easy to describe in a text prompt. For when model proposals help and when they create rework, read human-in-the-loop prelabeling. For why browser-side inference matters for privacy, see in-browser AI annotation with WebGPU.

From one clip to a team dataset

Tracker is single-user and keeps projects in your browser. When the output becomes a dataset several people label and review, LabelOp adds reviewer roles and team collaboration, plus smart annotation for images.

Before exporting, read exporting tracked video annotations to COCO and YOLO. For the whole app, start with the Tracker walkthrough.

To try the loop on your own footage, open LabelOp Tracker in Chrome or Edge and run Auto-annotate on a short clip.

FAQ

What detector does Tracker use by default?

RF-DETR nano, which detects the 80 COCO classes. It is about 103 MB, downloads once, and is cached in your browser afterwards.

Can Tracker detect objects that are not COCO classes?

Yes. Choose the OWLv2 text-prompt model and type what to look for, one per line, or upload your own ONNX detector.

Why does the same object get a new ID?

Usually because it was hidden or undetected for longer than the tracker keeps lost tracks (30 frames). Merge the two tracks in the Tracks tab and mark the QA item as fixed.

Does auto-annotation need a GPU?

It needs WebGPU, which uses your device's GPU through the browser. The detector has no CPU fallback, so Chrome or Edge with WebGPU support is recommended.

Will running Auto-annotate again keep my fixes?

No. Run again replaces the current AI result, including fixes you made to it. Tracker asks before it changes anything.

Auto-Annotate Video with Object Tracking: Detect, Track, Then Fix