TT-VidT: Disentangling the Temporal Dimension for Efficient Motion-Focused Video Pretraining
TT-VidT uses per-frame motion tokens and a key frame to separate appearance from temporal change, achieving strong small-scale results.
Key Takeaways
- TT-VidT is a video pretraining framework that separates spatial appearance from temporal motion instead of merging them.
- It represents each frame with a compact motion token and reconstructs frames from a key frame plus that token.
- This motion-focused design encourages the model to learn actual temporal changes rather than static visual shortcuts.
- In mirror-class tests on SSv2, TT-VidT was far less affected by flipping or time reversal than baseline models, suggesting stronger motion understanding.
- The method shows promising small-scale training results, but its scalability and performance on larger benchmarks remain key questions.
A new video pretraining framework called TT-VidT proposes that spatial and temporal information should be modeled separately rather than merged. The approach generates a per-frame "motion token," and a decoder reconstructs each frame from a key frame plus that token. According to the authors, the method reaches strong results with only small-scale training. The project page is available at https://kohakublueleaf.github.io/TTVidT/.
Why Separate Motion From Appearance
Most video models either stack per-frame embeddings or run a single global 3D model over the whole clip. TT-VidT takes a different route: it isolates motion into a compact token per frame, leaving appearance to a key frame. This design forces the model to encode what actually changes over time instead of memorizing static visual cues.
The Mirror-Class Test
The authors tested this by flipping or time-reversing videos from SSv2 mirror classes. TT-VidT kept its original answer on 1% or fewer of those clips, while baselines stayed on 9% to 43%. That gap suggests TT-VidT is genuinely reading motion rather than relying on spatial shortcuts.
What to Watch Next
The team describes the direction as promising with room to grow, and invites others to build on it. For readers tracking video representation learning, the key question is whether the motion-token approach scales beyond small-scale training and how it performs on larger benchmarks.
Comments
No comments yet. Be the first to comment.