WHAT, WHERE, AND HOW: PROBING AND UNDERSTANDING SPATIOTEMPORAL REPRESENTATIONS IN VIDEO FOUNDATION MODELS

dc.contributor.advisorDerpanis, Konstantinos
dc.contributor.authorMusa, Sharon Simisola
dc.date.accessioned2026-11-06T16:20:07Z
dc.date.available2026-11-06T16:20:07Z
dc.date.copyright2026-09-24
dc.date.issued2026-11-06
dc.date.updated2026-11-06T16:20:07Z
dc.degree.disciplineComputer Science
dc.degree.levelMaster's
dc.degree.nameMSc - Master of Science
dc.description.abstractSelf-supervised video foundation models learn rich spatiotemporal representations, yet it remains unclear what visual concepts these representations encode, where they emerge across transformer layers, and how they are geometrically organized. We investigate these questions through a layer-wise analysis of V-JEPA 2 and VideoMAE V2 using lightweight probes for three temporally grounded properties: camera motion, intuitive physics, and anomaly detection. Both models encode camera motion, with best results (>90 ROC-AUC) emerging at 60-70% of network depth, and achieve moderate anomaly detection performance (>60 ROC-AUC), but remain near chance on intuitive-physics tasks. Beyond classification, we find that temporal features from individual videos form smooth low-dimensional trajectories in representation space, suggesting that camera motion is both linearly decodable and geometrically organized. We then apply geometry-aware spline-based steering in latent space to interpolate camera motion, yielding steered videos with smoother trajectories and more coherent temporal progression than linear interpolation.
dc.identifier.urihttps://hdl.handle.net/10315/44365
dc.languageen
dc.rightsAuthor owns copyright, except where explicitly noted. Please contact the author directly with licensing requests.
dc.subjectComputer science
dc.subjectArtificial intelligence
dc.subject.keywordsInterpretability
dc.subject.keywordsVideo models
dc.subject.keywordsComputer vision
dc.subject.keywordsExplainable AI
dc.titleWHAT, WHERE, AND HOW: PROBING AND UNDERSTANDING SPATIOTEMPORAL REPRESENTATIONS IN VIDEO FOUNDATION MODELS
dc.typeElectronic Thesis or Dissertation

Files

Original bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
Musa_Sharon_Simisola_2026_MSc.pdf
Size:
21.35 MB
Format:
Adobe Portable Document Format

License bundle

Now showing 1 - 2 of 2
Loading...
Thumbnail Image
Name:
license.txt
Size:
1.87 KB
Format:
Plain Text
Description:
Loading...
Thumbnail Image
Name:
YorkU_ETDlicense.txt
Size:
3.39 KB
Format:
Plain Text
Description: