Prelabeling can save a project or quietly damage it. Set the confidence threshold too low and reviewers drown in bad drafts. Set it too high and the model contributes almost nothing. Pick a model only because it is fast and you may create more correction work than your reviewers can absorb. Meanwhile, annotation cost keeps rising for boring reasons: the same edge cases get relabeled, reviewers fix predictable mistakes, and exports fail late.
The useful question is not which model is best in the abstract. It is which setup produces approved labels for less total effort on your data, under your privacy constraints, with the review capacity you actually have. This guide covers the three decisions that answer it in LabelOp: local or cloud models, batch thresholds, and where the money really goes.
Local or cloud models: start with the real constraint
LabelOp supports both local and cloud model workflows. That matters because prelabeling is never just a speed choice. It is an operating choice.
Local models. Supported local models in LabelOp run in your browser. LabelOp offers click segmentation and Select Similar with WebGPU acceleration on compatible devices. The local options also include object detection, image classification, and zero-shot detection or classification using your own labels. These workflows use different runtimes, and WebGPU is not guaranteed for every local model.
Local inference does not send an image to a cloud inference service. It does not make everything local, though. Model downloads, project storage, and collaboration still have their own network requirements. If the dataset holds sensitive or tightly controlled images, local models are usually the first option to evaluate, but you still need sound access control and review discipline. The trade-off is compute: local processing depends on the hardware available to the annotator or operator.
Cloud models. LabelOp offers managed Cloud AI for detection and classification batches. You choose an image range and a confidence threshold, then follow progress as results are saved. Cloud AI uses your plan's cloud allowance and is separate from training through your own Vast.ai account. It is the better fit when shared throughput, device independence, and larger non-sensitive batches matter more than device-side execution.
Faster inference does not remove review work on either path. Prelabels still need human validation.
Match the model to the task. Dense scenes and unusual class boundaries change the math. Review cost should influence the model choice as much as raw inference speed. A slightly slower setup that produces cleaner drafts often wins over time, but only if the team measures correction effort honestly.
Mixed setups are normal. Many teams use local prelabeling when device-side inference is appropriate, cloud processing for larger non-sensitive batches, and full review on both paths. Some use different paths for different datasets or for different phases of the same pipeline. There is no need to standardize on one forever.
Batch processing and confidence thresholds
LabelOp supports local and cloud batch processing, so the local versus cloud decision is not only about one image at a time. Batch runs are where threshold choices start to matter, and where treating the threshold like a guess gets expensive.
Start with a limited batch. The right threshold is dataset-specific, so do not send the whole project through batch processing first. Run a controlled, representative slice and inspect what the model creates at one or two threshold settings. A pilot feels slower, and it is the cheapest way to avoid filling a larger queue with weak drafts. Local runs may suit privacy-first or workstation-based flows while cloud runs make larger slices quicker, but the threshold still has to be tested against review cost on both.
Confidence is not correctness. Model confidence is directionally useful, but it does not guarantee label quality. Some classes look strong at lower thresholds, while edge cases can stay wrong at high confidence. Teams also tend to overtrust clean-looking results from easy images.
Set thresholds by class risk. If some classes are expensive to correct or important to model quality, use tighter thresholds for them. A threshold that works for easy background objects may be unacceptable for rare or high-impact classes. Blanket rules such as "always use 0.5" are weak policy. One threshold for every class only makes sense when the classes behave similarly and the risk is low.
Judge by net time to approved output. A good threshold reduces manual work overall. A weak one only moves work from annotators to reviewers. Measure batch processing by the time it takes to reach an approved result, not by the number of predictions created. Tracking correction time takes discipline, but ignoring it means optimizing the wrong number. If prelabeling makes review slower, the threshold is not helping.
Recalibrate when the rules change. Thresholds are not one-time settings. If class definitions get tighter, tolerance for borderline predictions should usually tighten too. If the guideline gets simpler, a conservative threshold may now be too strict. Revisit the setting whenever the dataset mix, class definitions, or reviewer feedback pattern changes meaningfully. A stable threshold on top of a changing rule set is not actually stable.
Know when to skip prelabels. Very ambiguous classes, early ontology experiments, and tiny evaluation batches are often faster to label by hand. LabelOp gives you the option to run AI assistance, but it does not require you to use it everywhere.
Where annotation cost actually comes from
Controlling cost in LabelOp is less about squeezing annotator rates and more about removing avoidable waste. In computer vision projects, QA and rework often matter more than the initial per-image quote.
Rework comes first. When the same labels come back from review again and again, time is spent twice on work that should have been clarified once. Repeated rework consumes annotator time, reviewer time, and release time at once, which makes it the biggest hidden cost. Fixing it often means pausing briefly to improve rules. That pause is still cheaper than permanent correction churn.
Use prelabeling where correction is cheaper than drawing. AI assistance lowers cost only when the draft is easier to review than manual labeling would have been. Prelabeling can increase cost if thresholds are too loose or classes are still unstable. Before estimating savings, compare time to approved output on the same sample, including review and rework.
Match review intensity to risk. Uniform review is simple but not always efficient. Early pilots, new annotators, and high-impact classes deserve tighter review. Mature classes with stable agreement may only need targeted sampling. Cutting review too early, before the workflow is stable, usually increases rework later.
Use assignments to find waiting time. A project that feels expensive may just be poorly distributed. Assignment status and workload visibility show where work is waiting, where overdue batches build up, and whether reviewer bandwidth is the real bottleneck. That is often enough to cut delay cost without changing headcount.
Keep releases small and versioned. Large, messy releases are expensive to debug. Smaller checkpoints are easier to validate, compare, and export. Snapshots and comparisons reduce the cost of uncertainty because you can isolate what changed between runs.
Compare the whole infrastructure picture. Cost is not only labor. Some teams use local models to avoid recurring inference cost. LabelOp includes managed storage and cloud AI capacity in each tier, while optional Vast.ai work is billed directly to the connected project owner's account. Cloud cost is visible because it shows up as usage. Local cost is less obvious because it shows up as slower devices, longer operator time, or narrower hardware eligibility. "Free" local inference is not free if it slows the whole queue, and low visible spend can hide higher coordination cost elsewhere.
Track metrics that show waste. Raw labeled-image counts do not tell you much. Track these instead:
- rejection loop rate
- average time to approved work
- export failure rate
- overdue assignment trend
If cost keeps rising while throughput looks flat, rework is usually the first place to look, and small process fixes there tend to pay back before bigger tooling or staffing changes are needed.
A practical starting policy
- Use local models when device-local inference is the main constraint, and cloud batches when shared throughput and scale matter more.
- Pilot both paths, and one or two thresholds, on the same representative sample.
- Compare correction effort and time to approved output, not inference time or prediction volume.
- Tighten thresholds for high-risk classes and revisit them whenever guidelines or the data mix change.
- Keep review mandatory on every path, scaled by risk rather than habit.
- Attack repeated rejection loops before anything else.
If you are not measuring review effort, you are not really evaluating prelabeling, and you are not really controlling cost either.
Related Reading
- Human-in-the-Loop Prelabeling: Speed Without Silent Bias
- Active Learning for Labeling: A Pipeline You Can Actually Run
- WebGPU In-Browser AI Annotation
- BYO GPU Auto-Labeling with Vast.ai
- Annotation Rework Reduction Framework
- Private and Local Image Annotation Options
- Smart Annotation Features
References
FAQ
Should I run prelabeling models locally or in the cloud?
Choose local inference when you want supported models to run on your device, and managed Cloud AI when you want cloud batch processing. Neither removes review work. Performance depends on the selected model and hardware, and local inference does not make project storage or sharing automatically local.
What is the best confidence threshold for LabelOp?
There is no universal best value. The right threshold depends on class difficulty, review capacity, and how expensive correction is. Test one or two settings on a representative pilot batch and compare time to approved output.
How often should we recalibrate thresholds?
Any time the dataset mix, class definitions, or reviewer feedback pattern changes in a meaningful way.
Do local models always reduce cost?
Not always. They can reduce inference spend, but slower hardware or heavier review can offset that advantage.
What is the biggest hidden cost in annotation operations?
Repeated rework. It consumes annotator time, reviewer time, and release time all at once.
How much does data annotation cost?
It depends on task complexity, label type, review coverage, workforce location, and whether work is priced per image, per object, per hour, or per project. For computer vision teams, QA and rework often matter more than the initial per-image quote.