Searching a video by text sounds like it needs a server farm. You type "a dog running on a beach," and somewhere a large model watches your footage.
It does not have to work that way. LabelOp Moments, a free app in the LabelOp family, does the whole search inside a browser tab. The video is read from your disk, analyzed on your GPU, and never uploaded.
This post explains how that works, step by step, using Moments as the concrete example. It also explains why the approach has the limits it has, because those follow directly from the design.
Short answer
On-device video search works in four steps:
- Sample frames from the video at a fixed rate (Moments uses about one per second).
- Embed each frame with an image-and-text model, producing a list of numbers that represents what the frame shows.
- Embed your text description with the same model, into the same kind of list.
- Compare the text with every frame, and return the frames whose numbers are closest.
Everything runs in the browser. The only large download is the model itself.
Step 1: Read the video without uploading it
When you choose an MP4 in Moments, the browser gives the page access to that local file. Moments uses Mediabunny, a media library, to read the file lazily and the browser's WebCodecs API to decode H.264 video frames. Nothing is sent to a server; there is no upload endpoint.
Before any analysis, Moments checks that the file's codec and the browser's capabilities are supported. That is why it asks for current desktop Chrome or Edge with WebGPU, WebCodecs, and hardware acceleration enabled.
Step 2: Sample frames
A video at 30 frames per second has 108,000 frames per hour. Embedding every one of them in a browser would be slow and mostly redundant, since neighboring frames look nearly identical.
Moments samples about one frame per second and scales each sampled frame so its long edge is at most 512 pixels, keeping the aspect ratio. Repeated frame timestamps are skipped before inference, so the model does not do the same work twice.
This choice is the source of a key limitation. If something happens in less than a second and falls between two samples, the index never sees it.
Step 3: Turn frames into embeddings
An embedding is a fixed-length list of numbers that represents content. Models are trained so that similar content produces nearby lists.
Moments uses EmbeddingGemma 2 Text Vision, a model that can embed both images and text into the same space. It runs through the LiteRT-LM runtime on the GPU via WebGPU, inside a Web Worker so the page stays responsive. Audio is disabled; the model only sees pictures and text.
For each sampled frame, Moments produces a 768-dimensional embedding and normalizes it to unit length. Those vectors, plus the actual timestamp of each sample, are kept in the Worker's memory.
How much memory is that? In the project's one-hour test, 3,600 frame embeddings took about 10.55 MiB. The vectors are small; the model and the browser's own overhead are the heavy part.
Step 4: Embed the query
When you type a description, the same model embeds the text into the same 768-dimensional space. Moments adds a short task prefix the model expects for search queries, so the text is encoded as a search query rather than a document. Descriptions are limited to 240 characters.
If you run the exact same description again on the same video, the last text embedding is reused instead of recomputed.
Step 5: Rank by similarity
Because both kinds of vectors are normalized, comparing them is simple: a dot product. Multiply the two lists element by element and add up the results. Higher means more similar.
Moments scores every analyzed frame against the query and sorts them. Then it picks up to five candidates, skipping any frame within three seconds of one already chosen. Without that rule, a long shot that matches well would fill all five slots with nearly the same moment.
Only after ranking does Moments decode thumbnails, and only for the returned candidates.
If you set a time range, frames outside that range are filtered out before ranking.
On the project's test hardware (an NVIDIA RTX 5060 Ti), the median search on a ready index was 238 ms including thumbnail decoding, across 20 queries on a roughly ten-minute video. That is an observed timing on one machine, not a general benchmark.
Step 6: Refine locally
A one-frame-per-second index is coarse. If the best match is a second or two away from the sampled frame, Refine this moment helps: it rescans that candidate's window (up to six seconds) at two frames per second and returns the best local match.
Refinement embeddings are temporary. They do not replace the main index.
What gets downloaded, stored, and kept
This is where on-device search differs most from a cloud service, so it is worth being precise.
The model. About 388 MB, downloaded once after you consent. It streams into the browser's Cache API and is verified against a pinned SHA-256 hash before it is stored; a mismatched file is never cached. After that, the model and app files allow analysis and search to run offline. The browser can evict the cache, so it is not a permanent copy.
The video. Read from your disk. It is not uploaded and not copied into browser storage.
The index. Kept in memory only. There is no persistent video index, so refreshing the page clears it and you choose the file again to search it.
Favorites and history. Stored in local storage as compact metadata: file name and identity, timestamps, query, range. Never the video, a frame, or a thumbnail.
Analytics. The site uses privacy-friendly page analytics (Umami). So "no network requests at all" is not accurate. "Your video and your queries never leave the device" is.
If you want the same reasoning applied to image annotation, the post on WebGPU and in-browser AI annotation covers it.
Why the limits are what they are
Each limitation follows from a design choice above.
Short events can be missed. Sampling at one frame per second means sub-second events may never be embedded. Refinement only helps around candidates that were already found.
Motion is not understood. Every embedding describes one still frame. "A car turning left" and "a car turning right" may look the same in a single frame. The project's documentation says it directly: motion retrieval is not motion understanding.
Order is not understood. "After the door opens" has no meaning to a system that compares your text with frames independently.
There is always an answer. Ranking returns the closest frames, whether or not the scene exists. In the project's tests, all 12 queries for absent scenes still returned candidates. That is why results are labeled Closest moments and why the score is shown as similarity, not confidence.
Preview is not segmentation in time. Preview plays a few seconds around a frame. It does not know where an action begins or ends.
No audio. Spoken words and sounds are not part of the index.
How well does it work in practice?
The project publishes its own measurements, with caveats that matter as much as the numbers.
- On one animated film of about ten minutes, with 20 queries whose expected answers were written down before the model ran, the expected scene appeared in the top five for 19 of 20 queries (12 of 12 static scenes, 7 of 8 motion scenes). Indexing produced 597 embeddings in about 104.7 seconds on an RTX 5060 Ti.
- On three further videos with English and Turkish queries, the expected scene appeared in the top five for 32 of 36 positive queries. English scored 17 of 18, Turkish 15 of 18.
- A one-hour, low-resolution test video produced 3,600 embeddings in about 315.9 seconds.
The documentation is explicit that one animated film and a repeating low-resolution loop do not establish accuracy or speed on arbitrary footage, that other tests ran during part of the benchmark, and that single-frame embeddings can miss short events, direction, or order. Read these as a small, honest test set, not a general accuracy claim.
When on-device search is the right fit
It fits well when:
- the footage is sensitive or simply too large to upload
- you need to find a few visual scenes, not transcribe or summarize a whole video
- you have a reasonably recent GPU and a WebGPU browser
It is the wrong tool when:
- you need to search speech or sound
- the event you care about is defined by motion or sequence
- you need a persistent, shared index across many videos
Where it fits in a labeling workflow
For computer vision teams, the most practical use is finding scenes worth labeling in long footage. That workflow, from search to extracted frames to labels in LabelOp, is in how to find frames worth labeling in long videos. For a hands-on walkthrough of the app itself, see search a video by describing the scene.
The rest of the LabelOp family follows the same in-browser approach for other jobs, such as video annotation with LabelOp Tracker and background removal with LabelOp Retouch.
Try it
The fastest way to understand the trade-offs is to see them on your own footage. Open LabelOp Moments, choose an MP4, and search for one static scene and one action. The difference between the two results will tell you more than any diagram.
FAQ
Does on-device video search send my video to a server?
Not in LabelOp Moments. The video is decoded and embedded in your browser, and neither the video nor your queries are uploaded. The site does load privacy-friendly page analytics.
What is a video embedding?
In Moments, it is a 768-number vector produced for each sampled frame. Frames and text descriptions are embedded into the same space, so they can be compared directly.
How often does Moments sample the video?
About one frame per second, with the long edge capped at 512 pixels. Refine this moment rescans a candidate's window at two frames per second.
Why can it not find fast actions reliably?
Each embedding describes a single still frame, sampled about once per second. Short events can fall between samples, and direction or order of motion is not represented.
Does it work offline?
After the first model download and once the app files are cached, analysis and search can run offline. The browser can evict that cache, in which case the model downloads again.