Frequency-Based Motion Representation for Video Generative Adversarial Networks

被引:0
作者
Hyun, Sangeek [1 ]
Lew, Jaihyun [2 ]
Chung, Jiwoo [1 ]
Kim, Euiyeon [1 ]
Heo, Jae-Pil [3 ]
机构
[1] Sungkyunkwan Univ, Dept Artificial Intelligence, Suwon 16419, South Korea
[2] Seoul Natl Univ, Interdisciplinary Program Artificial Intelligence, Seoul 08826, South Korea
[3] Sungkyunkwan Univ, Dept Comp Sci & Engn, Suwon 16419, South Korea
关键词
Generative adversarial networks; video generation; sinusoidal motion representation; speed-level motion manipulation;
D O I
10.1109/TIP.2023.3293767
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Videos contain motions of various speeds. For example, the motions of one's head and mouth differ in terms of speed - the head being relatively stable and the mouth moving rapidly as one speaks. Despite its diverse nature, previous video GANs generate video based on a single unified motion representation without considering the aspect of speed. In this paper, we propose a frequency-based motion representation for video GANs to realize the concept of speed in video generation process. In detail, we represent motions as continuous sinusoidal signals of various frequencies by introducing a coordinate-based motion generator. We show, in that case, frequency is highly related to the speed of motion. Based on this observation, we present frequency-aware weight modulation that enables manipulation of motions within a specific range of speed, which could not be achieved with the previous techniques. Extensive experiments validate that the proposed method outperforms state-of-the-art video GANs in terms of generation quality by its capability to model various speed of motions. Furthermore, we also show that our temporally continuous representation enables to further synthesize intermediate and future frames of generated videos.
引用
收藏
页码:3949 / 3963
页数:15
相关论文
共 50 条
  • [41] Test Case Filtering based on Generative Adversarial Networks
    Liu, Zhijuan
    Zhang, Li
    Wu, Xuangou
    Zhao, Wei
    2022 IEEE 23RD INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE SWITCHING AND ROUTING (IEEE HPSR), 2022, : 65 - 69
  • [42] A New Steganography Method Based on Generative Adversarial Networks
    Naito, Hiroshi
    Zhao, Qiangfu
    2019 IEEE 10TH INTERNATIONAL CONFERENCE ON AWARENESS SCIENCE AND TECHNOLOGY (ICAST 2019), 2019, : 495 - 500
  • [43] Semantic image inpainting based on Generative Adversarial Networks
    Wu, Chugang
    Xian, Yanhua
    Bai, Junqi
    Jing, Yuancheng
    2020 INTERNATIONAL CONFERENCE ON ARTIFICIAL INTELLIGENCE AND COMPUTER ENGINEERING (ICAICE 2020), 2020, : 276 - 280
  • [44] Multimodal Image Fusion Based on Generative Adversarial Networks
    Yang Xiaoli
    Lin Suzhen
    Lu Xiaofei
    Wang Lifang
    Li Dawei
    Wang Bin
    LASER & OPTOELECTRONICS PROGRESS, 2019, 56 (16)
  • [45] Face Image Colorization Based on Generative Adversarial Networks
    Han X.-J.
    Liu Y.-L.
    Yang H.-Y.
    Beijing Ligong Daxue Xuebao/Transaction of Beijing Institute of Technology, 2019, 39 (12): : 1285 - 1291
  • [46] SSGAN: Secure Steganography Based on Generative Adversarial Networks
    Shi, Haichao
    Dong, Jing
    Wang, Wei
    Qian, Yinlong
    Zhang, Xiaoyu
    ADVANCES IN MULTIMEDIA INFORMATION PROCESSING - PCM 2017, PT I, 2018, 10735 : 534 - 544
  • [47] Exploring generative adversarial networks and adversarial training
    Sajeeda A.
    Hossain B.M.M.
    Int. J. Cogn. Comp. Eng., (78-89): : 78 - 89
  • [48] Coevolution of Generative Adversarial Networks
    Costa, Victor
    Lourenco, Nuno
    Machado, Penousal
    APPLICATIONS OF EVOLUTIONARY COMPUTATION, EVOAPPLICATIONS 2019, 2019, 11454 : 473 - 487
  • [49] A survey of generative adversarial networks
    Zhu, Kongtao
    Liu, Xiwei
    Yang, Hongxue
    2018 CHINESE AUTOMATION CONGRESS (CAC), 2018, : 2768 - 2773
  • [50] Steganographic Generative Adversarial Networks
    Volkhonskiy, Denis
    Nazarov, Ivan
    Burnaev, Evgeny
    TWELFTH INTERNATIONAL CONFERENCE ON MACHINE VISION (ICMV 2019), 2020, 11433