Towards Automatic Sports Analytics: Team Affiliation, Jersey Number Recognition and Player Tracking

Abstract

Automatic sports video understanding can enhance performance analysis, coaching, and the viewing experience. A central challenge is reliably identifying and tracking players, who look visually similar, often occlude each other, and have jersey numbers that are only intermittently visible. This dissertation addresses these challenges through three interconnected tasks: team classification, jersey number recognition, and long-term multi-object tracking. We first introduce a self-supervised method for team affiliation classification using contrastive learning, which generalizes to unseen uniforms and reduces burn-in time compared to color-based methods. Second, we propose a jersey number recognition pipeline that combines legibility filtering, torso localization, and sequence-level aggregation. This approach achieves strong results on a new hockey dataset and SoccerNet, and generalizes across sports and camera viewpoints. Third, we present SportsSUSHI, a graph-based tracking framework that integrates team labels and jersey numbers into the association process, improving robustness under occlusion and moving cameras. To support this work, we release a new university hockey dataset annotated for team affiliation, jersey numbers, and tracking. Together, these contributions—unsupervised team classification, transferable number recognition, and identity-aware tracking—form a unified framework for robust and generalizable sports video analysis, laying a foundation for future advances in analytics, coaching, and media.