Loading post…
UberExplainer
Validating Bounding Box Annotations
Summary
Video bounding-box annotations can contain ID swaps and position jumps that degrade training data for object-tracking models, especially when long videos are split among annotators and later merged. Uber's system runs in uLabel, combining visual, motion, and coordinate features from an 11-frame sliding window with an XGBoost classifier trained on synthetic anomalies from six open-source datasets. It scores each frame for error probability, then groups predictions with frame-index and confidence clustering into regions that operators can review, fix, or dismiss. The current iteration prioritizes jump and ID-swap detection, while freeze and scale/aspect errors remain planned for future releases; the deployed workflow is intended to reduce the cost and time of sequential edit-review while processing client data statelessly without retaining it or using it to train models.
Context
Video annotations are split into segments to reduce annotator fatigue, but joining segments and relying on human review can introduce or miss tracking errors. Detection difficulty varies with object size, motion, camera movement, scene complexity, visual conditions, frame rate, and resolution, making fixed pixel thresholds unreliable.
Approach / What changed
The pipeline ingests video frames, bounding boxes, and object IDs, processes each tracked object independently, and extracts visual, motion, and coordinate features over an 11-frame sliding window. An XGBoost classifier assigns frame-level error probabilities, and post-processing uses two-dimensional clustering across frame index and confidence to create reviewable error regions. Synthetic jump and ID-swap perturbations from six open-source datasets provide training anomalies.
Takeaways
The current system targets ID swaps and position jumps; freeze errors and scale/aspect distortions are planned for future releases.
An 11-frame sliding window gives the classifier temporal context, including recent motion consistency, variance, and trends.
Operators can fix flagged suggestions and rerun validation or dismiss them when the model is wrong.
FAQ
Why are fixed pixel thresholds inadequate for detecting annotation errors?
The same coordinate change can be normal or erroneous depending on object size, object speed, camera movement, frame rate, and resolution. A fixed threshold can therefore be too aggressive in difficult scenarios and too lenient in straightforward ones.
How does the system turn frame-level predictions into reviewable results?
The XGBoost classifier assigns each frame an error probability from 0.0 to 1.0. Post-processing groups predictions into contiguous error clusters using frame index and confidence, producing actionable regions for human review.
What happens when the model flags an annotation incorrectly?
After validation runs in uLabel, operators can either fix the suggested issue and rerun the workflow or dismiss the suggestion. This keeps human review in the process while avoiding a mandatory second operator for every sample.
Source and freshness
- Source
- uber.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.