Motion-based frame interpolation for film and television effects

Frame interpolation is the process of synthesising a new frame in-between existing frames in an image sequence. It has emerged as a key algorithmic module in motion picture effects. In the context of this special issue, this study provides a review of the technology used to create in-between frames and presents a Bayesian framework that generalises frame interpolation algorithms using the concept of motion interpolation. Unlike existing literature in this area, the authors also compare performance using the top industrial toolkits used in the post production industry. They find that all successful techniques employ motion-based interpolation, and the commercial version of the Bayesian approach performs best. Another goal of this study is to compare the performance gains with recent convolutional neural network (CNN) algorithms against the traditional explicit model-based approaches. They find that CNNs do not clearly outperform the explicit motion-based techniques, and require significant compute resources, but provide complementary improvements in certain types of sequences.