TrackEverything: Dense Long-Horizon Tracking by Deduplicating 3D Scene Representations
TrackEverything addresses the sparse-versus-dense tracking trade-off by representing videos as persistent 3D scene tracks. It deduplicates co-located observations, separates static and dynamic points, and samples features from scene clouds instead of storing large 4D correlations. This enables dense long-horizon tracking with memory tied to scene geometry rather than video length.
Key Takeaways
- TrackEverything addresses the long-standing trade-off between sparse long-video tracking and dense short-clip tracking by representing videos as persistent 3D scene tracks in world coordinates.
- It decouples model complexity from video duration by scaling with unique physical scene geometry instead of frame count, helping overcome memory limits in long-horizon video understanding.
- The method combines voxel-based track deduplication, endpoint prediction with static/dynamic classification, lightweight trajectory refinement for dynamic points, and 3D WAFT for efficient feature sampling.
- The authors report that it can track all visible points across videos over 1,000 frames within 40 GB of GPU memory, while outperforming open-source dense 3D trackers on TAPVid-3D by more than 20% APD on short clips.
Point tracking models have long faced a hard choice: track a sparse set of query points across long videos, or track every point but only across short clips. A new model called TrackEverything claims to break that trade-off by representing videos as persistent 3D scene tracks in world coordinates.
Background: Why the Sparse-Dense Tradeoff Exists
Video is a 2D projection of an underlying 3D world. Existing trackers that operate directly on 2D frames accumulate memory in proportion to video duration, which caps how long they can run and how many points they can follow. TrackEverything instead decouples model complexity from video duration, scaling with unique physical scene geometry rather than frame count.
Three Technical Innovations
The approach rests on three components. A voxelization-based de-duplication mechanism operates at sliding-window boundaries, merging co-located tracks so repeated observations of the same surface do not redundantly accumulate. Tracking is then decomposed into an endpoint refiner, which predicts each point's destination plus a static-versus-dynamic classification, followed by a lightweight trajectory refiner that decodes dense trajectories only for dynamic points. Finally, 3D WAFT replaces memory-prohibitive 4D correlation volumes with efficient feature sampling in the scene cloud.
Benchmarks and Memory Claims
According to the authors, TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory. On TAPVid-3D, it outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips, while remaining competitive with state-of-the-art sparse trackers on long sequences, despite tracking far more points.
What to Watch
The results target two practical constraints in video understanding: memory ceilings and the sparse-dense divide. Whether the voxel-based de-duplication holds up on scenes with heavy occlusion or repeated textures remains an open question, and independent reproduction of the 1000-frame, 40 GB claim will be the next signal to watch.
Comments
No comments yet. Be the first to comment.