A Convolutional Sequence to Sequence Model for Multimodal Dynamics Prediction in Ski Jumps

A convolutional sequence to sequence model for predicting the jump forces of ski jumpers directly from pose estimates is presented. We collect the footage of multiple, unregistered cameras together with the output of force measurement plates and present a spatiotemporal calibration procedure for all modalities which is merely based on the athlete's pose estimates. The synchronized data is used to train a fully convolutional sequence to sequence network for predicting jump forces directly from the human pose. We demonstrate that the best performing networks produce a mean squared error of 0.062 on normalized force time series while being able to identify the moment of maximal force occurrence in the original video at 55% recall within +/- 2 frames around the ground truth.