Adaptive Future Frame Prediction with Ensemble Network

Project Summary

Video future frame prediction is challenging because videos include complicated appearance changes and complex motion dynamics of objects. This project focuses on developing an accurate future frame prediction network using optical flow based estimation, online-updating, and ensemble learning technique.

Paper Abstract

Future frame prediction in videos is a challenging problem because videos include complicated movements and large appearance changes. Learning-based future frame prediction approaches have been proposed in kinds of literature. A common limitation of the existing learning-based approaches is a mismatch of training data and test data. In the future frame prediction task, we can obtain the ground truth data by just waiting for a few frames. It means we can update the prediction model online in the test phase. Then, we propose an adaptive update framework for the future frame prediction task. The proposed adaptive updating framework consists of a pre-trained prediction network, a continuous-updating prediction network, and a weight estimation network. We also show that our pre-trained prediction model achieves comparable performance to the existing state-of-the-art approaches. We demonstrate that our approach outperforms existing methods especially for dynamically changing scenes.

Paper Summary Video

Publications

Adaptive Future Frame Prediction with Ensemble Network [Paper]
Wonjik Kim, Masayuki Tanaka, Masatoshi Okutomi, Yoko Sasaki
International Conference on Pattern Recognition Workshop, 2020.