Combining CNN and transformers for full-reference and no-reference image quality assessment

被引:10
|
作者
Zeng, Chao [1 ]
Kwong, Sam [2 ]
机构
[1] Hubei Univ, Sch Artificial Intelligence, Wuhan, Peoples R China
[2] City Univ Hong Kong, Dept Comp Sci, Hong Kong, Peoples R China
关键词
Image quality assessment; Convolutional neural network; Transformers; Non-local information; STATISTICS;
D O I
10.1016/j.neucom.2023.126437
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Most deep learning approaches for image quality assessment use regression from deep features extracted by CNN (Convolutional Neural Networks). However, non-local information is usually neglected in exist-ing methods. Motivated by the recent success of transformers in modeling contextual information, we propose a hybrid framework that utilizes a vision transformer backbone to extract features and a CNN decoder for quality estimation. We propose a shared feature extraction scheme for both FR and NR set-tings. A two-branch structured attentive quality predictor is devised for quality prediction. Evaluation experiments on various IQA datasets, including LIVE, CSIQ and TID2013, LIVE-Challenge, KADID-10 K, and KONIQ-10 K, show that our proposed models achieve outstanding performance for both FR and NR settings.& COPY; 2023 Published by Elsevier B.V.
引用
收藏
页数:11
相关论文
共 50 条
  • [41] A weighted full-reference image quality assessment based on visual saliency
    Wen, Yang
    Li, Ying
    Zhang, Xiaohua
    Shi, Wuzhen
    Wang, Lin
    Chen, Jiawei
    JOURNAL OF VISUAL COMMUNICATION AND IMAGE REPRESENTATION, 2017, 43 : 119 - 126
  • [42] Machine learning to design full-reference image quality assessment algorithm
    Ling, Wang Yu
    Hu, Yang
    Telkomnika - Indonesian Journal of Electrical Engineering, 2013, 11 (06): : 3439 - 3444
  • [43] No-Reference Stereoscopic Image Quality Assessment
    Akhter, Roushain
    Sazzad, Z. M. Parvez
    Horita, Y.
    Baltes, J.
    STEREOSCOPIC DISPLAYS AND APPLICATIONS XXI, 2010, 7524
  • [44] No-reference Image Denoising Quality Assessment
    Lu, Si
    ADVANCES IN COMPUTER VISION, CVC, VOL 1, 2020, 943 : 416 - 433
  • [45] Automatic no-reference image quality assessment
    Li, Hongjun
    Hu, Wei
    Xu, Zi-neng
    SPRINGERPLUS, 2016, 5
  • [46] Full-Reference Image Quality Metrics: Classification and Evaluation
    Pedersen, Marius
    Hardeberg, Jon Yngve
    FOUNDATIONS AND TRENDS IN COMPUTER GRAPHICS AND VISION, 2011, 7 (01): : 1 - 80
  • [47] Full-Reference and No-Reference Objective Evaluation of Deep Neural Network Speech
    Voran, Stephen
    2021 13TH INTERNATIONAL CONFERENCE ON QUALITY OF MULTIMEDIA EXPERIENCE (QOMEX), 2021, : 85 - 90
  • [48] No-Reference Fingerprint Image Quality Assessment
    Tiwari, Kamlesh
    Gupta, Phalguni
    INTELLIGENT COMPUTING METHODOLOGIES, 2014, 8589 : 846 - 854
  • [49] Full-reference IPTV image quality assessment by deeply learning structural cues
    Kong, YanQiang
    Cui, Liu
    Hou, Rui
    SIGNAL PROCESSING-IMAGE COMMUNICATION, 2020, 83
  • [50] ENCODING DISTORTIONS FOR MULTI-TASK FULL-REFERENCE IMAGE QUALITY ASSESSMENT
    Huang, Chen
    Jiang, Tingting
    Jiang, Ming
    2019 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO (ICME), 2019, : 1864 - 1869