Self-Supervised Learning Detects Subtle Exercise Errors in Gym Videos

Computer vision systems typically struggle to identify proper exercise technique in real gym footage, where inconsistent lighting, camera angles, and equipment occlusion confound standard body-pose detection. Spotting workout form errors requires capturing motion patterns far more nuanced than what off-the-shelf models can reliably provide.

A research team has developed a self-supervised learning approach that learns exercise-specific visual patterns from unlabeled gym videos, then applies those insights to detect subtle form mistakes with only a small set of expert-annotated examples. The strategy hinges on exploiting the rhythmic, repetitive nature of exercises themselves.

How the approach works#

Rather than relying on generic pose estimators, the researchers built two specialized self-supervised techniques. Pose contrastive learning trains the model to recognize consistent body positions across video frames despite variations in camera position or clothing. Motion disentangling separately learns the underlying movement patterns, allowing the system to isolate the cyclic motion characteristic of each exercise.

These methods capitalize on natural variation—different camera angles, lighting conditions, and gym attire—to build robust representations that generalize better than supervised-only training.

New benchmark dataset#

To validate their approach, the team created Fitness-AQA, a curated dataset covering three barbell exercises: back squats, barbell rows, and overhead presses. Expert trainers annotated the videos with common form errors specific to each movement, providing ground truth for the supervised phase of training.

Broader applications#

Experimental results demonstrated that the self-supervised representations outperformed both 2D and 3D pose estimators as well as several baseline methods. The researchers also showed the framework can extend to other domains, including general pose estimation and dive quality assessment in sports.