WHAT, WHERE, AND HOW: PROBING AND UNDERSTANDING SPATIOTEMPORAL REPRESENTATIONS IN VIDEO FOUNDATION MODELS

Loading...
Thumbnail Image

Authors

Musa, Sharon Simisola

Journal Title

Journal ISSN

Volume Title

Publisher

Abstract

Self-supervised video foundation models learn rich spatiotemporal representations, yet it remains unclear what visual concepts these representations encode, where they emerge across transformer layers, and how they are geometrically organized. We investigate these questions through a layer-wise analysis of V-JEPA 2 and VideoMAE V2 using lightweight probes for three temporally grounded properties: camera motion, intuitive physics, and anomaly detection. Both models encode camera motion, with best results (>90 ROC-AUC) emerging at 60-70% of network depth, and achieve moderate anomaly detection performance (>60 ROC-AUC), but remain near chance on intuitive-physics tasks. Beyond classification, we find that temporal features from individual videos form smooth low-dimensional trajectories in representation space, suggesting that camera motion is both linearly decodable and geometrically organized. We then apply geometry-aware spline-based steering in latent space to interpolate camera motion, yielding steered videos with smoother trajectories and more coherent temporal progression than linear interpolation.

Description

Keywords

Computer science, Artificial intelligence

Citation