Skip to main content
Springer Nature Link
Log in

Virtual view synthesis for the nonuniform illuminated between views in surgical video

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

For high-quality surgical video virtual view synthesis, a Weighted Autoregressive Interpolation (WAI) algorithm and an Adaptively-enhanced Hole Filling (AHF) are proposed to reduce the artifacts caused by up-sampling and relieve the luma difference. First, high quality up-sampled reference views are acquired by the WAI algorithm. A Piecewise Autoregressive (PAR) model is introduced and the distance weight of pixels is also considered. The precision of the virtual view is improved by the WAI and the texture edges are well preserved. Next, for the AHF, the intermediate view with more structure details is selected as the template. The other intermediate view is calibrated to it. And the luma difference is relieved. Then, a Nearest background Holes Filling algorithm (NHF) is adopted to blend these two intermediate views, in which only background pixels are selected to fill the remaining holes. Combining the WAI with AHF, the visual quality of the surgical virtual video is improved. For the objective quality, the experimental results show that the PSNR of the proposed algorithm is 0.5841 dB higher than the VSRS 1D-Fast algorithm on average. For subjective quality, the proposed method can reduce the artifacts and gain higher subjective quality for the synthesized virtual view of the surgical video.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Cai J, Chang L, Wang H et al (2018) Boundary-preserving depth upsampling without texture copying artifacts and holes. IEEE International Symposium on Multimedia:1–5. https://doi.org/10.1109/ISM.2017.11

  2. Cai C, Fan B, Meng H, Zhu Q (2020) Hole-filling approach based on convolutional neural network for depth image-based rendering view synthesis[J]. Journal of Electronic Imaging 29(1)

  3. Campero A, Baldoncini M, Villalonga JF, Abarca-Olivas J (2019) Three-dimensional microscopic surgical videos: a novel and low-cost system[J]. World Neurosurgery 132(12):188–196

    Article  Google Scholar 

  4. Chen X, Liang H, Xu H et al (2020) Virtual view synthesis based on asymmetric bidirectional DIBR for 3D video and free viewpoint video[J]. Applied ences 10(5):1562

    Google Scholar 

  5. Chia-Ming C, Shu-Jyuan L, Shang-Hong L et al (2012) Improved novel view synthesis from depth image with large baseline. International Conference on Pattern Recognition IEEE. https://doi.org/10.1109/ICPR.2008.4761649

  6. Cho JM, Park SY, Chien SI (2020) Hole-filling of RealSense depth images using a color edge map[J]. IEEE Access 8:53901–53914

    Article  Google Scholar 

  7. Criminisi A, Perez P, Toyama K (2004) Region filling and object removal by exemplar-based image inpainting. IEEE Trans Image Process 13(9):1200–1212. https://doi.org/10.1109/TIP.2004.833105

    Article  Google Scholar 

  8. Daribo I, Pesquet-Popescu B (2010) Depth-aided image inpainting for novel view synthesis. IEEE International Workshop on Multimedia Signal Processing IEEE. https://doi.org/10.1109/MMSP.2010.5662013

  9. Daribo I, Saito H (2015) A novel inpainting-based layered depth video for 3DTV. IEEE Trans Broadcast 57(2):533–541. https://doi.org/10.1109/tbc.2011.2125110

    Article  Google Scholar 

  10. Dziembowski A, Grzelka A, Mieloch D et al (2017) Enhancing view synthesis with image and depth map upsampling. International Conference on Systems, Signals and Image Processing Iwssip. https://doi.org/10.1109/IWSSIP.2017.7965598

  11. Gautier J, Meur OL, Guillemot C (2011) Depth-based image completion for view synthesis. 3dtv Conference: the True Vision - Capture, Transmission and Display of 3d Video, IEEE: 1–4. https://doi.org/10.1109/3DTV.2011.5877193.

  12. Gortler SJ, Grzeszczuk R, Szeliski R et al (1996) The lumigraph. Proc Siggraph:43–54. https://doi.org/10.1145/237170.237200

  13. Gwangju Institute of Science and Technology (GIST), 3DV Sequences of GIST [Online]. Available: ftp://203.253.128.142.

  14. Ham B, Min D, Choi J, et al. (2009) Virtual view rendering using super-resolution with multiview images. 16th IEEE international conference on Image processing (ICIP) IEEE. https://doi.org/10.1109/ICIP.2009.5414509.

  15. Hanxiong Y, Liming Z, Guibo L et al (2015) A new disocclusion filling approach in depth image based rendering for stereoscopic imaging. International Conference on Control, IEEE. https://doi.org/10.1109/ICCAIS.2015.7338683

  16. HEVC Test Model, [Online]. Available: https://hevc.hhi.fraunhofer.de/trac/3dhevc/browser/3DVCSoftware.

  17. Hosseinpour H, Mousavinia A (2018) View synthesis for FTV systems based on a minimum spatial distance and correspondence field[J]. Multidim Syst Sign Process 30(7):1–20

    MathSciNet  MATH  Google Scholar 

  18. JCT-VC. Test Model 10 of 3D-HEVC and MV-HEVC. JCT3V-J1003, Joint Collaborative Team on 3D Video Coding Extension Development of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11, 10th Meeting: Strasbourg, FR:18–24 Oct. 2014.

  19. Jiufei X, Ming X, Dongxiao L, et al. (2010) A new virtual view rendering method based on depth image. Asia-Pacific Conference on Wearable Computing Systems, IEEE, 2010. https://doi.org/10.1109/APWCS.2010.43.

  20. Joachimiak M, Hannuksela M, Gabbouj M (2014) View synthesis quality mapping for depth-based super resolution on mixed resolution 3D video. 3dtv-Conference: the True Vision - Capture, Transmission and Display of 3d Video IEEE: 1–4. https://doi.org/10.1109/3DTV.2014.6874740.

  21. Kim HG, Ro YM (2017) Multi-view stereoscopic video hole filling considering spatio-temporal consistency and binocular symmetry for synthesized 3D video. IEEE Transactions on Circuits & Systems for Video Technology 27(7):1435–1449. https://doi.org/10.1109/TCSVT.2016.2515360

    Article  Google Scholar 

  22. Lai Y, Lan X, Liu Y et al (2012) Disocclusion using depth reliability map for view synthesis. IEEE International Conference on Acoustics, Speech and Signal Processing IEEE:1449–1452. https://doi.org/10.1109/ICASSP.2012.6288164

  23. Levoy M, Hanrahan P (1996) Light field rendering. Proc Siggraph:31–42. https://doi.org/10.1145/237170.237199

  24. Linwei Z, Yun Z, Mei Y et al (2013) View-spatial–temporal post-refinement for view synthesis in 3D video systems. Signal Process Image Commun 28(10):1342–1357. https://doi.org/10.1016/j.image.2013.08.005

    Article  Google Scholar 

  25. Luo G, Zhu Y (2018) Hole filling for view synthesis using depth guided global optimization. IEEE Access 6:32874–32889. https://doi.org/10.1109/ACCESS.2018.2847312

    Article  Google Scholar 

  26. Luo G, Zhu Y, Weng Z, Li Z (2020) A Disocclusion Inpainting framework for depth-based view synthesis. IEEE Trans Pattern Anal Mach Intell 42(6):1289–1302

    Article  Google Scholar 

  27. Meng-Sung W, Yung-Yu C, Yen-Tzu L, Cheng-Chung H (2012) P-8: depth-map-based multi-view synthesis using joint bilateral upsampling on GPUs. SID Symposium Digest of Technical Papers 41(1):1252–1255. https://doi.org/10.1889/1.3499895

    Article  Google Scholar 

  28. Mori Y, Fukushima N, Yendo T, Fujii T, Tanimoto M (2009) View generation with 3D warping using depth information for FTV. Signal Process Image Commun 24(1–2):65–72. https://doi.org/10.1016/j.image.2008.10.013

    Article  Google Scholar 

  29. Muddala S. Sjöström M. and Olsson R (2014) Depth-based inpainting for disocclusion filling. 3dtv-Conference: the True Vision - Capture, Transmission and Display of 3d Video, IEEE: 1–4. https://doi.org/10.1109/3DTV.2014.6874752.

  30. Nagoya University, 3DV Sequences of Nagoya University [Online]. Available: http://www.tanimoto.nuee.nagoya-u.ac.jp/mpeg/mpeg-ftv.html.

  31. Nokia, 3DV Sequences of Poznan University [Online]. Available: ftp://mpeg3dv.research.nokia.com.

  32. Po L, Zhang S, Xu X, et al. (2011) A new multidirectional extrapolation hole-filling method for depth-image-based rendering. 18th IEEE International Conference on Image Processing (ICIP) IEEE. https://doi.org/10.1109/ICIP.2011.6116194.

  33. Poznan University, 3DV Sequences of Poznan University [Online]. Available: ftp://multimedia.edu.pl/3DV/.

  34. Quan Q, He F, Li H (2020) A multi-phase blending method with incremental intensity for training detection networks[J]. Vis Comput 6–8

  35. Ramírez R, Jaureguizar F, García N et al (2015) An effective inpainting technique for hole filling in DIBR synthesized images. IEEE International Symposium on Consumer Electronics, IEEE. https://doi.org/10.1109/ISCE.2015.7177846

  36. Schmeing, M. and Jiang X. (2012) Faithful spatio-temporal disocclusion filling using local optimization. Pattern Recognition (ICPR), 21st International Conference on IEEE, 2012.

  37. Schmeing M, Jiang X (2015) Faithful disocclusion filling in depth image based rendering using superpixel-based inpainting. IEEE Transactions on Multimedia 17(12):2160–2173. https://doi.org/10.1109/TMM.2015.2476372

    Article  Google Scholar 

  38. Tezuka T, Tehrani MP, Suzuki K et al (2015) View synthesis using superpixel based inpainting capable of occlusion handling and hole filling, 124-128. Picture Coding Symposium IEEE. https://doi.org/10.1109/PCS.2015.7170060

  39. View Synthesis Reference Software, [Online]. Available: http://wg11.sc29.org/svn/repos/MPEG-4/test/trunk/3D/viewsynthesis/VSRS.

  40. Vosters LPJ, Varekamp C, Haan G (2013) Evaluation of efficient high quality depth upsampling methods for 3DTV. Proceedings of SPIE - The International Society for Optical Engineering 8650(4):865005. https://doi.org/10.1117/12.2005094

    Article  Google Scholar 

  41. Wang L, Hou C, Lei J, Yan W (2015) View generation with DIBR for 3D display system. Multimed Tools Appl 74(21):9529–9545. https://doi.org/10.1007/s11042-014-2133-9

    Article  Google Scholar 

  42. Wu Y, He F, Zhang D et al (2018) Service-oriented feature-based data exchange for cloud-based design and manufacturing. IEEE Trans Serv Comput 11(2):341–353

    Article  Google Scholar 

  43. Xin T, Ping Y, Xiaozhen Z, et al. (2010) A sub-pixel virtual view synthesis method for multiple view synthesis. 28th Picture Coding Symposium, Nagoya: 490-493. https://doi.org/10.1109/PCS.2010.5702544.

  44. Yao L, Lu Q, Li X (2019) View synthesis based on spatio-temporal continuity[J]. EURASIP Journal on Image and Video Processing 1:86

    Article  Google Scholar 

  45. Yao L, Han Y, Li X (2019) Fast and high-quality virtual view synthesis from multi-view plus depth videos. Multimed Tools Appl 78(7):19325–19340

    Article  Google Scholar 

  46. Yu H, He F, Pan Y (2019) A scalable region-based level set method using adaptive bilateral filter for noisy image segmentation[J]. Multimed Tools Appl 79(10):5743–5765

    Google Scholar 

  47. Zhang X, Image WX (2008) Interpolation by adaptive 2-D autoregressive modeling and soft-decision estimation. IEEE Trans Image Process 17(6):887–896. https://doi.org/10.1109/TIP.2008.924279

    Article  MathSciNet  Google Scholar 

  48. Zhang J, He F, Chen Y (2019) A new haze removal approach for sky/river alike scenes based on external and internal clues[J]. Multimed Tools Appl 20:2085–2107

    Google Scholar 

  49. Zhu S , Xu H , Yan L (2019) An improved depth Image based virtual view synthesis method for interactive 3D video[J]. IEEE Access

    Book  Google Scholar 

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No.61672362, 61272255) and the Beijing Natural Science Foundation (No.4172012), the Scientific Research Common Program of Beijing Municipal Commission of Education (No.KM201710025011). Also thanks Beijing Friendship Hospital, affiliated with Capital Medical University for the hernia surgical video.

Author information

Author notes

  1. Boqi Jia and Nan Zhang contributed equally to this work. Nan Zhang is the Co-first Author.

Authors and Affiliations

  1. School of Biomedical Engineering, Capital Medical University, Beijing, China

    Boqi Jia, Nan Zhang, Nan Liang & Bo Wu

  2. Beijing Key Laboratory of Fundamental Research on Biomechanics in Clinical Application, Capital Medical University, Beijing, China

    Boqi Jia, Nan Zhang, Nan Liang & Bo Wu

  3. Department of Computer Science, City University of Hong Kong, Hong Kong, China

    Shiqi Wang

Authors
  1. Boqi Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nan Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shiqi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Bo Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Wu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jia, B., Zhang, N., Liang, N. et al. Virtual view synthesis for the nonuniform illuminated between views in surgical video. Multimed Tools Appl 80, 20619–20639 (2021). https://doi.org/10.1007/s11042-021-10732-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-10732-3

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy