Skip to main content

Advertisement

Springer Nature Link
Log in

Face photo-sketch portraits transformation via generation pipeline

  • Research
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Portrait sketching is widely used in digital art, forensic security and other fields with its unique value. However, existing portrait sketch style transfer techniques often focus on overall style transformation, neglecting the hierarchical drawing steps and the integration of details in the sketching process. The generated images still fall short in terms of layering effects and detail representation. To address this issue, this paper proposes a generative adversarial network-based portrait sketch temporal generation pipeline (PSTG). The pipeline simulates the artist’s layer-by-layer drawing process, sequentially executing from outline to facial features and then to hair, to generate high-quality sketch images with detailed expressiveness. Additionally, we designed a composite network structure that includes both global and local generators. The global generator is responsible for capturing overall contours and proportions, while the local generators focus on the detailed depiction of facial features and hair. This structure excels in capturing both overall proportions and local details. Experimental results demonstrate that the PSTG method not only restores the detail hierarchy of sketches but also achieves significant success in retaining sketch details and brushstroke effects. It also effectively imitates the brushstroke style of artists, generating portrait sketches that visually resemble the works of professional artists, outperforming existing holistic stylization methods overall.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

No datasets were generated or analyzed during the current study.

References

  1. Yi, R., Liu, Y.-J., Lai, Y.-K., Rosin, P.L.: Apdrawinggan: generating artistic portrait drawings from face photos with hierarchical gans. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10735–10744 (2019)

  2. Pinkney, J.N.M., Adler, D.: Resolution dependent gan interpolation for controllable image synthesis between domains. arXiv arXiv:2010.05334 (2020)

  3. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Computer Vision and Pattern Recognition (2019)

  4. Shu, Y., Yi, R., Xia, M., Ye, Z., Zhao, W., Chen, Y., Lai, Y.-K., Liu, Y.-J.: Gan-based multi-style photo cartoonization. IEEE Trans. Vis. Comput. Graph. 27(1), 1–1 (2021)

    Google Scholar 

  5. Song, G., Luo, L., Liu, J., Ma, W.-C., Lai, C.-P., Zheng, C., Cham, T.: Agilegan: stylizing portraits by inversion-consistent transfer learning. ACM Trans. Graph. 40(4), 117:1-117:13 (2021)

    Article  Google Scholar 

  6. Cui, A., McKee, D., Lazebnik, S.: Dressing in order: recurrent person image generation for pose transfer, virtual try-on and outfit editing. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 14618–14627 (2021)

  7. Li, Y.J., Fang, C., Hertzmann, A., Shechtman, E., Yang, M.H.: Im2Pencil: controllable pencil illustration from photographs. In: Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1525–1534. IEEE, Long Beach (2019)

  8. Wang, S.Y., Bau, D., Zhu, J.Y.: Sketch your own GAN. In: Proceedings of IEEE/CVF International Conference on Computer Vision, pp. 14030–14040. IEEE, Montreal (2021)

  9. Yang, S., Wang, Z., Liu, J., Guo, Z.: Deep plastic surgery: robust and controllable image editing with human-drawn sketches. In: Computer Vision—ECCV,: 16th European Conference, Glasgow, UK, August 23–28, 2020. Proceedings, Part XV, pp. 601–617. Springer, Berlin (2020)

  10. Lu, Y.Y., Wu, S.Z., Tai, Y.W., Tang, C.K.: Image generation from sketch constraint using contextual GAN. In: Proceedings of the 15th European Conference on Computer Vision, pp. 213–228. Springer, Munich (2018)

  11. Peng, C., Wang, N., Li, J., Gao, X.: Universal face photo-sketch style transfer via multiview domain translation. IEEE Trans. Image Process. 29, 8519–8534 (2020)

    Article  MATH  Google Scholar 

  12. Wan, W., Yang, Y., Lee, H.J.: Generative adversarial learning for detail-preserving face sketch synthesis. Neurocomputing 438, 107–121 (2021)

    Article  MATH  Google Scholar 

  13. Zhang, M., Wang, N., Li, Y., Gao, X.: Bionic face sketch generator. IEEE Trans. Cybern. 50(6), 2701–2714 (2020)

    Article  MATH  Google Scholar 

  14. Yu, J., Xu, X., Gao, F., Shi, S., Wang, M., Tao, D., Huang, Q.: Toward realistic face photo-sketch synthesis via composition-aided gans. IEEE Trans. Cybern. 51(9), 4350–4362 (2021)

    Article  MATH  Google Scholar 

  15. Li, L., Tang, J., Ye, Z., Sheng, B., Mao, L., Ma, L.: Unsupervised face super-resolution via gradient enhancement and semantic guidance. Vis. Comput. 37, 2855–2867 (2021)

    Article  MATH  Google Scholar 

  16. Li, Y., Chen, X., Wu, F., Zha, Z.-J.: Linestofacephoto: face photo generation from lines with conditional selfattention generative adversarial networks. In: Proceedings of the 27th ACM International Conference on Multimedia, ser. MM’19, pp. 2323–2331. Association for Computing Machinery, New York (2019)

  17. Li, Y., Chen, X., Yang, B., Chen, Z., Cheng, Z., Zha, Z.-J.: DeepFacePencil: Creating Face Images from Freehand Sketches, pp. 991–999. Association for Computing Machinery, New York (2020)

    Google Scholar 

  18. Zhang, S.C., Ji, R.R., Hu, J., Lu, X.Q., Li, X.L.: Face sketch synthesis by multidomain adversarial learning. IEEE Trans. Neural Netw. Learn. Syst. 30(5), 1419–1428 (2019). https://doi.org/10.1109/TNNLS.2018.2869574

    Article  MATH  Google Scholar 

  19. Sheng, B., Li, P., Gao, C., Ma, K.-L.: Deep neural representation guided face sketch synthesis. IEEE Trans. Vis. Comput. Graph. 25(12), 3216–3230 (2019)

    Article  MATH  Google Scholar 

  20. Zhang, M., Wang, R., Gao, X., Li, J., Tao, D.: Dualtransfer face sketch-photo synthesis. IEEE Trans. Image Process. 28(2), 642–657 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  21. Zhang, L.L., Lin, L., Wu, X., Ding, S.Y., Zhang, L.: End-to-end photo-sketch generation via fully convolutional representation learning. In: Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pp. 627–634. ACM, Shanghai (2015). https://doi.org/10.1145/2671188.2749321

  22. Zhu, J.Y., Krähenbühl, P., Shechtman, E., Efros, A.A.: Generative visual manipulation on the natural image manifold. In: Proceedings of the 14th European Conference on Computer Vision, pp. 597–613. Springer, Amsterdam (2016). https://doi.org/10.1007/978-3-31946454-1-36

  23. Zhang, M.J., Wang, N.N., Li, Y.S., Wang, R.X., Gao, X.B.: Face sketch synthesis from coarse to fine. In: Proceedings of the 32nd AAAI Conference on Artificial Intelligence, California, USA, pp. 7558-7565 (2018). https://doi.org/10.1609/aaai.v32i1.12224

  24. Zhang, M.J., Wang, N., Li, Y., Gao, X.: Markov random neural fields for face sketch synthesis. In: Proceedings of International Joint Conferences on Artificial Intelligence, Stockholm, Sweden, pp. 7558–7565 (2018)

  25. Zhang, S.C., Ji, R.R., Hu, J., Gao, Y., Lin, C.W.: Robust face sketch synthesis via generative adversarial fusion of priors and parametric sigmoid. In: Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, Sweden, pp. 1163–1169 (2018)

  26. Wang, L.D., Sindagi, V., Patel, V.: High-quality facial photo-sketch synthesis using multi-adversarial networks. In: Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition, pp. 83–90. IEEE, Xi’an (2018). https://doi.org/10.1109/FG.2018.00022

  27. Kazemi, H., Iranmanesh, M., Dabouei, A., Soleymani, S., Nasrabadi, N.M.: Facial attributes guided deep sketch-to-photo synthesis. In: Proceedings of IEEE Winter Applications of Computer Vision Workshops. IEEE, Lake Tahoe (2018). https://doi.org/10.1109/WACVW.2018.00006

  28. Kazemi, H., Taherkhani, F.: Unsupervised facial geometry learning for sketch to photo synthesis. In: Proceedings of International Conference of the Biometrics Special Interest Group. IEEE, Darmstadt (2018)

  29. Duan, S.C., Chen, Z.X., Wu, Q.M.J., Cai, L., Lu, D.: Multi-scale gradients self-attention residual learning for face photo-sketch transformation. IEEE Trans. Inf. Forensics Secur. 16, 12181230 (2020). https://doi.org/10.1109/TIFS.2020.3031386

    Article  Google Scholar 

  30. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, Montreal, Canada, pp. 2672–2680 (2014)

  31. Peng, C., Zhang, C., Liu, D., Wang, N., Gao, X.: Face photo-sketch synthesis via intra-domain enhancement. Knowl. Based Syst. 259(C), Art. no. 110026 (2023). https://doi.org/10.1016/j.knosys.2022.110026

  32. Yi, R., Xia, M., Liu, Y.-J., Lai, Y.-K., Rosin, P.L.: Line drawings for face portraits from photos using global and local structure based gans. IEEE Trans Pattern Anal Mach Intell 43(10), 3462–3475 (2021)

    Article  MATH  Google Scholar 

  33. Yi, R., Liu, Y.-J., Lai, Y.-K., Rosin, P.: Quality metric guided portrait line drawing generation from unpaired training data. IEEE Trans. Pattern Anal. Mach. Intell. 66, 1 (2022)

    Google Scholar 

  34. Zenoozi, A.D., Navi, K., Majidi, B.: Argan: fast converging gan for animation style transfer. In: International Conference on Machine Vision and Image Processing (MVIP), vol. 2022, pp. 1–5 (2022)

  35. Li, B., Zhu, Y., Wang, Y., Lin, C.-W., Ghanem, B., Shen, L.: Anigan: style-guided generative adversarial networks for unsupervised anime face generation. IEEE Trans. Multimed. 23(4), 1–1 (2021)

    Google Scholar 

  36. Chen, Z., Wang, C., Yuan, B., Tao, D.: Puppeteergan: arbitrary portrait animation with semantic-aware appearance transformation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2020, pp. 13515–13524 (2020)

  37. Huang, Z., Zheng, Z., Yan, C., Xie, H., Sun, Y., Wang, J., Zhang, J.: Real-world automatic makeup via identity preservation makeup net. In: International Joint Conference on Artificial Intelligence (2021)

  38. Zhang, X., Zheng, Z., Gao, D., Zhang, B., Yang, Y., Chua, T.S.: Multi-view consistent generative adversarial networks for compositional 3D-aware image synthesis. Int. J. Comput. Vis. 131(8), 2219–2242 (2023)

    Article  MATH  Google Scholar 

  39. Suo, Y., Zheng, Z., Wang, X., et al.: Jointly harnessing prior structures and temporal consistency for sign language video generation. ACM Trans. Multimed. Comput. Commun. Appl. 18(2), 1–19 (2022)

    MATH  Google Scholar 

  40. Lin, X., Sun, S., Huang, W., Sheng, B., Li, P., Feng, D.D.: EAPT: efficient attention pyramid transformer for image processing. IEEE Trans. Multimed. 25, 50–61 (2021)

    Article  MATH  Google Scholar 

  41. Xie, Z., Zhang, W., Sheng, B., Li, P., Chen, C.P.: BaGFN: broad attentive graph fusion network for high-order feature interactions. IEEE Trans. Neural Netw. Learn. Syst. 34(8), 4499–4513 (2021)

    Article  MATH  Google Scholar 

  42. Zhou, Y., Chen, Z., Li, P., Song, H., Chen, C.P., Sheng, B.: FSAD-Net: feedback spatial attention dehazing network. IEEE Trans. Neural Netw. Learn. Syst. 34, 66 (2022)

  43. Zhang, S., Ji, R., Hu, J., Lu, X., Li, X.: Face sketch synthesis by multidomain adversarial learning. IEEE Trans. Neural Netw. Learn. Syst. 30(5), 1419–1428 (2019)

    Article  MATH  Google Scholar 

  44. Heusel, M., Ramsauer, H., Unterhaltung, T., Nair, V., Urtasun, R.: GANs trained by a two time-scale update rule converge to equilibrium. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 6736–6744

  45. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  MATH  Google Scholar 

  46. Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2017, pp. 2223–2232 (2017)

  47. Shang, M., Gao, F., Li, X., Zhu, J., Dai, L.: Bridging unpaired facial photos and sketches by line-drawings. In: ICASSP 2021—2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2010–2014. IEEE (2021)

  48. Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 5967–5976 (2017). https://doi.org/10.1109/CVPR.2017.632

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Artificial Intelligence, Wuhan Textile University, Wuhan, 430200, China

    Mengsi Guo, Mingfu Xiong, Jin Huang, Xinrong Hu & Tao Peng

Authors
  1. Mengsi Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingfu Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinrong Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tao Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Mingfu Xiong provided critical guidance and fnancial support for this study, Mengsi Guo wrote the main manuscript text and prepared the experimental data, and Jin Huang was responsible for the preliminary algorithm ideas. All authors reviewed the manuscript.

Corresponding author

Correspondence to Mingfu Xiong.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: the assignment of portraits and text in author biographies was not correct.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, M., Xiong, M., Huang, J. et al. Face photo-sketch portraits transformation via generation pipeline. Vis Comput 41, 1183–1196 (2025). https://doi.org/10.1007/s00371-024-03403-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-024-03403-5

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy