Computer Science

Computer Science ›› 2021, Vol. 48 ›› Issue (9): 174-180.doi: 10.11896/jsjkx.200800014

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Face Image Inpainting with Generative Adversarial Network

LIN Zhen-xian1, ZHANG Meng-kai2, WU Cheng-mao3, ZHENG Xing-ning2   

  1. 1 School of Science,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
    2 School of Communication and Information Engineering,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
    3 School of Electronic Engineering,Xi'an University of Posts & Telecommunications,Xi'an 710121,China
  • Received:2020-08-03 Revised:2020-10-29 Online:2021-09-15 Published:2021-09-10
  • About author:LIN Zhen-xian,born in 1969,Ph.D,professor.Her main research interests include wavelet theory and its application in signal and image.
    ZHANG Meng-kai,born in 1995,postgraduate.His main research interests include deep learning and computer vision.
  • Supported by:
    National Natural Science Foundation of China(61671377)

PDF (PC)

1081

Abstract: Face image inpainting is a hot topic of image processing research in recent years.Due to the loss of excessive sematic information,it is a difficult problem to inpaint large area missing of face images.Aiming at the problem of inpainting face images,a step-by-step image inpainting algorithm based on generative adversarial network is proposed.Face images inpainting task is divided into two steps.Firstly,face images are completed through the pre-completion network,and pre-completion images is enhanced feature through the enhancement network.The discriminator judges the difference between the pre-completion images,the enhanced images and the ideal image respectively.The long-term memory unit is used to connect the information flow of two parts.Secondly,the adversarial loss,content loss and total variation loss are combined to improve the effectively.Experiments are conducted on CelebA dataset,and this algorithm has an improvement of 16.84%~22.85% in PSNR and 10%～12.82% in SSIM compared with others typical image inpainting algorithms

Key words: Deep learning, Face images, Generative adversarial networks, Image inpainting, Long short term memory, Loss areas, Skip connection

CLC Number: 

  • TP183
[1]HE Y T,TANG X H,ZHANG Y,et al.Improved Criminisi algorithm based on structure tensor[J].Journal of Image and Graphics,2018,23(10):1492-1507.
[2]LAN X L,LIU H X,YAO H B.Improved image inpainting algorithm based on texture blocks and gradient feature[J].Computer Engineering and Applications,2018,54(20):172-177.
[3]KRHENBÜHL P,KOLTUN V.Efficient Inference in FullyConnected CRFs with Gaussian Edge Potentials[J].Advances in Neural Information Processing Systems,2012,24(2011):109-117.
[4]XU L M,WU Y J,ZHANG B.Image Inpainting AlgorithmBased on Adaptive High Order Variation in Eight Neighbors[J].Journal of Graphics,2017,38(4):558-565.
[5]BERTALMIO M,SAPIRO G,CASELLES V,et al.Image in-painting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.ACM Press/Addison-Wesley Publishing Co.,2000:417-424.
[6]LEVIN A,ZOMET A,WEISS Y.Learning how to inpaint from global mage statistics[C]//IEEE In International Conference on Computer Vision.2003:305-312.
[7]SHEN J,CHAN T F.Mathematical Models for Local Nontexture Inpaintings[J].SIAM Journal on Applied Mathematics,2001,62:1019-1043.
[8]SHEN J,KANG S H,CHAN T F.Euler's Elastica and Curvature-Based Inpainting[J].SIAM Journal on AppliedMathema-tics,2002,63(2):564-592.
[9]BOUREAU Y L,BACH F,LECUN Y,et al.Learning mid-level features for recognition[C]//2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.IEEE,2010,2559-2566.
[10]LI S,ZHAO M.Image inpainting with salient structure completion and texture propagation[J].Pattern Recognition Letters,2011,32(9):1256-1266.
[11]EFROS A A,LEUNG T K.Texture synthesis by non-parame-tric sampling[C]//Proceedings of the Seventh IEEE Internatio-nal Conference on Computer Vision.1999,2:1033-1038.
[12]CRIMINISI A,PÉREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212.
[13]BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:A randomized correspondence algorithm for structural image editing[C]//ACM Transactions on Graphics (ToG).ACM,2009:1-11.
[14]WEI Y,LIU S.Domain-based structure-aware image inpainting[J].Signal Image & Video Processing,2016,10(5):911-919.
[15]GIRSHICK R,DONAHUE J,DARRELL T,et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2014:580-587.
[16]LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2015:3431-3440.
[17]XIE J,XU L,CHEN E.Image denoising and inpainting withdeep neural networks[C]//Advances in Neural Information Processing Systems.2012:341-349.
[18]KÖHLER R,SCHULER C,SCHÖLKOPF B,et al.Mask.specific inpainting with deep neural networks[C]//German Confe-rence on Pattern Recognition.Cham:Springer,2014:523-534.
[19]GOODFELLOW I,POUGET-ABADIE J,MIRZAM,et al.Gene-rative adversarial nets[C]//Advances in Neural Information Processing Systems.2014:2672-2680.
[20]PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:Feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:2536-2544.
[21]YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:6721-6729.
[22]IIZUKA S S,SERRA E,SHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Gra-phics (TOG),2017,36(4):107:1-2.
[23]YU J H,ZHE L,YANG J M,et al.Generative Image Inpainting with Contextual Attention [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.2018:5505-5514.
[24]LIU G,REDA F A,SHIH K J,et al.Image inpainting for irre-gular holes using partial convolutions [C]//Proceedings of the European Conference on Computer Vision (ECCV).2018:85-100.
[25]HOCHREITER S,SCHMIDHUBER J.Long Short-Term Me-mory[J].Neural Computation,1997,9(8):1735-1780.
[26]YEH R A,CHEN C,LIM T Y,et al.Semantic Image Inpainting with Deep Generative Models [C]//CVPR 2016.2016:5485-5493.
[27]ISOLA P,ZHU J Y,ZHOU T H,et al.Image-to-image translation with conditional adversarial networks[C]//IEEEConfe-rence on Computer Vision and Pattern Recognition (CVPR).2017.
[28]RONNEBERGER O,FISCHER P,BROX T.U-Net:Convolu-tional Networks for Biomedical Image Segmentation[C]//International Conference on Medical Image Computing and Computer-assisted Intervention.Cham:Springer,2015:234-241.
[29]RADFORD A,METZ L,CHINTALA S.Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks [C]//Proceedings of the International Confe-rence on Learning Representations (ICLR).2016.
[30]ZHAO H,GALLO O,FROSIO I,et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging,2016,3(1):47-57.
[31]WHYTE O,SIVIC J,ZISSERMAN A,et al.Non-uniform deblurring for shaken images[J].International Journal of Compu-ter Vision,2012,98(2):168-186.
[32]RUDIN L I,OSHER S,FATEMI E.Nonlinear total variation based noise removal algorithms[J].Physica D Nonlinear Phenomena,1992,60(1/2/3/4):259-268.
[33]NAH S,KIM T H,LEE K M.Deep multi-scale convolutional neural network for dynamic scene deblurring[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2017:3883-3891.
[34]TONG Y B,ZHANG Q S,QI Y P.Image Quality Assessing by Combining PSNR with SSIM[J].Journal of Image and Gra-phics,2006,11(12),1758-1763.
[35]WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality as-sessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612.
[36]ZHENG C X,CHAM T J,CAI J.Pluralistic Image Completion [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).2019:1438-1447.
[37]CIMPOI M,MAJI S,KOKKINOS I,et al.Describing Textures in the Wild[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).2014:3606-3613.
[1] RAO Zhi-shuang, JIA Zhen, ZHANG Fan, LI Tian-rui. Key-Value Relational Memory Networks for Question Answering over Knowledge Graph [J]. Computer Science, 2022, 49(9): 202-207.
[2] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[3] XU Yong-xin, ZHAO Jun-feng, WANG Ya-sha, XIE Bing, YANG Kai. Temporal Knowledge Graph Representation Learning [J]. Computer Science, 2022, 49(9): 162-171.
[4] WANG Jian, PENG Yu-qi, ZHAO Yu-fei, YANG Jian. Survey of Social Network Public Opinion Information Extraction Based on Deep Learning [J]. Computer Science, 2022, 49(8): 279-293.
[5] HAO Zhi-rong, CHEN Long, HUANG Jia-cheng. Class Discriminative Universal Adversarial Attack for Text Classification [J]. Computer Science, 2022, 49(8): 323-329.
[6] JIANG Meng-han, LI Shao-mei, ZHENG Hong-hao, ZHANG Jian-peng. Rumor Detection Model Based on Improved Position Embedding [J]. Computer Science, 2022, 49(8): 330-335.
[7] SUN Qi, JI Gen-lin, ZHANG Jie. Non-local Attention Based Generative Adversarial Network for Video Abnormal Event Detection [J]. Computer Science, 2022, 49(8): 172-177.
[8] HU Yan-yu, ZHAO Long, DONG Xiang-jun. Two-stage Deep Feature Selection Extraction Algorithm for Cancer Classification [J]. Computer Science, 2022, 49(7): 73-78.
[9] CHENG Cheng, JIANG Ai-lian. Real-time Semantic Segmentation Method Based on Multi-path Feature Extraction [J]. Computer Science, 2022, 49(7): 120-126.
[10] HOU Yu-tao, ABULIZI Abudukelimu, ABUDUKELIMU Halidanmu. Advances in Chinese Pre-training Models [J]. Computer Science, 2022, 49(7): 148-163.
[11] ZHOU Hui, SHI Hao-chen, TU Yao-feng, HUANG Sheng-jun. Robust Deep Neural Network Learning Based on Active Sampling [J]. Computer Science, 2022, 49(7): 164-169.
[12] SU Dan-ning, CAO Gui-tao, WANG Yan-nan, WANG Hong, REN He. Survey of Deep Learning for Radar Emitter Identification Based on Small Sample [J]. Computer Science, 2022, 49(7): 226-235.
[13] WANG Jun-feng, LIU Fan, YANG Sai, LYU Tan-yue, CHEN Zhi-yu, XU Feng. Dam Crack Detection Based on Multi-source Transfer Learning [J]. Computer Science, 2022, 49(6A): 319-324.
[14] CHU Yu-chun, GONG Hang, Wang Xue-fang, LIU Pei-shun. Study on Knowledge Distillation of Target Detection Algorithm Based on YOLOv4 [J]. Computer Science, 2022, 49(6A): 337-344.
[15] ZHOU Zhi-hao, CHEN Lei, WU Xiang, QIU Dong-liang, LIANG Guang-sheng, ZENG Fan-qiao. SMOTE-SDSAE-SVM Based Vehicle CAN Bus Intrusion Detection Algorithm [J]. Computer Science, 2022, 49(6A): 562-570.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy