Skip to main content
Springer Nature Link
Log in

Densely-Sampled Light Field Reconstruction

  • Chapter
  • First Online:
Real VR – Immersive Digital Reality

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11900))

  • 2449 Accesses

Abstract

In this chapter, we motivate the use of densely-sampled light fields as the representation which can bring the required density of light rays for the correct recreation of 3D visual cues such as focus and continuous parallax and can serve as an intermediary between light field sensing and light field display. We consider the problem of reconstructing such a representation from few camera views and approach it in a sparsification framework. More specifically, we demonstrate that the light field is well structured in the set of so-called epipolar images and can be sparsely represented by a dictionary of directional and multi-scale atoms called shearlets. We present the corresponding regularization method, along with its main algorithm and speed-accelerating modifications. Finally, we illustrate its applicability for the cases of holographic stereograms and light field compression.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Adelson, E.H., Bergen, J.R.: The plenoptic function and the elements of early vision. In: Computational Models of Visual Processing, pp. 3–20 (1991)

    Google Scholar 

  2. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54(11), 4311–4322 (2006)

    Article  MATH  Google Scholar 

  3. Ahmad, W., Vagharshakyan, S., Sjöström, M., Gotchev, A., Bregovic, R., Olsson, R.: Shearlet transform based prediction scheme for light field compression. In: 2018 Data Compression Conference, pp. 396–396, March 2018

    Google Scholar 

  4. Alperovich, A., Johannsen, O., Strecke, M., Goldluecke, B.: Light field intrinsics with a deep encoder-decoder network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    Google Scholar 

  5. Blumensath, T., Davies, M.E.: Normalized iterative hard thresholding: guaranteed stability and performance. IEEE J. Sel. Top. Signal Process. 4(2), 298–309 (2010)

    Article  Google Scholar 

  6. Bolles, R.C., Baker, H.H., Marimont, D.H.: Epipolar-plane image analysis: an approach to determining structure from motion. Int. J. Comput. Vis. 1(1), 7–55 (1987)

    Article  Google Scholar 

  7. Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3(3), 492–526 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  8. Cai, J.F., Chan, R.H., Shen, Z.: A framelet-based image inpainting algorithm. Appl. Comput. Harmonic Anal. 24(2), 131–149 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  9. Cai, J.F., Shen, Z.: Framelet based deconvolution. J. Comput. Math. 28(3), 289–308 (2010)

    MathSciNet  MATH  Google Scholar 

  10. Camahort, E., Lerios, A., Fussell, D.: Uniformly sampled light fields. In: Drettakis, G., Max, N. (eds.) Rendering Techniques 1998, Eurographics, pp. 117–130. Springer, Vienna (1998). https://doi.org/10.1007/978-3-7091-6453-2_11

  11. Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  12. Candes, E.J., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 51(12), 4203–4215 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  13. Candès, E.J., Donoho, D.L.: Ridgelets: a key to higher-dimensional intermittency? Philos. Trans. R. Soc. Lond. Ser. A: Math. Phys. Eng. Sci. 357(1760), 2495–2509 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  14. Candès, E.J., Donoho, D.L.: New tight frames of curvelets and optimal representations of objects with piecewise \(c^2\) singularities. Commun. Pure Appl. Math. 57(2), 219–266 (2004)

    Article  MathSciNet  MATH  Google Scholar 

  15. Zhang, C., Chen, T.: Spectral analysis for sampling image-based rendering data. IEEE Trans. Circuits Syst. Video Technol. 13(11), 1038–1050 (2003)

    Article  Google Scholar 

  16. Chai, J.X., Tong, X., Chan, S.C., Shum, H.Y.: Plenoptic sampling. In: 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, pp. 307–318 (2000)

    Google Scholar 

  17. Chan, S.H., Wang, X., Elgendy, O.A.: Plug-and-play ADMM for image restoration: fixed-point convergence and applications. IEEE Trans. Comput. Imaging 3(1), 84–98 (2017)

    Article  MathSciNet  Google Scholar 

  18. Do, M.N., Marchand-Maillet, D., Vetterli, M.: On the bandwidth of the plenoptic function. IEEE Trans. Image Process. 21(2), 708–717 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  19. Do, M.N., Vetterli, M.: The contourlet transform: an efficient directional multiresolution image representation. IEEE Trans. Image Process. 14(12), 2091–2106 (2005)

    Article  Google Scholar 

  20. Genzel, M., Kutyniok, G.: Asymptotic analysis of inpainting via universal shearlet systems. SIAM J. Imaging Sci. 7(4), 2301–2339 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  21. Georgiev, T., Intwala, C.: Light field camera design for integral view photography. Adobe Technical report (2006)

    Google Scholar 

  22. Gilliam, C., Dragotti, P., Brookes, M.: On the spectrum of the plenoptic function. IEEE Trans. Image Process. 23(2), 502–516 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  23. Gilliam, C., Dragotti, P.L., Brookes, M.: A closed-form expression for the bandwidth of the plenoptic function under finite field of view constraints. In: IEEE International Conference on Image Processing (ICIP), pp. 3965–3968, September 2010

    Google Scholar 

  24. Gortler, S.J., Grzeszczuk, R., Szeliski, R., Cohen, M.F.: The lumigraph. In: 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, pp. 43–54 (1996)

    Google Scholar 

  25. Heber, S., Pock, T.: Convolutional networks for shape from light field. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3746–3754, June 2016

    Google Scholar 

  26. Heber, S., Yu, W., Pock, T.: Neural EPI-volume networks for shape from light field. In: IEEE International Conference on Computer Vision (ICCV), pp. 2271–2279, October 2017

    Google Scholar 

  27. Heyden, A., Pollefeys, M.: Multiple view geometry. Emerg. Top. Comput. Vis. 3, 45–107 (2005)

    Google Scholar 

  28. Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 328–341 (2008)

    Article  Google Scholar 

  29. Honauer, K., Johannsen, O., Kondermann, D., Goldluecke, B.: A dataset and evaluation methodology for depth estimation on 4D light fields. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10113, pp. 19–34. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54187-7_2

    Chapter  Google Scholar 

  30. Hosni, A., Rhemann, C., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 504–511 (2013)

    Article  Google Scholar 

  31. Ihm, I., Park, S., Lee, R.K.: Rendering of spherical light fields. In: Fifth Pacific Conference on Computer Graphics and Applications, pp. 59–68, October 1997

    Google Scholar 

  32. Jeon, H., et al.: Accurate depth map estimation from a lenslet light field camera. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1547–1555, June 2015

    Google Scholar 

  33. Kalantari, N.K., Wang, T.C., Ramamoorthi, R.: Learning-based view synthesis for light field cameras. ACM Trans. Graph. (TOG) 35(6), 1–10 (2016)

    Article  Google Scholar 

  34. Kim, C., Zimmer, H., Pritch, Y., Sorkine-Hornung, A., Gross, M.: Scene reconstruction from high spatio-angular resolution light fields. ACM Trans. Graph. (TOG) 32(4), 1–12 (2013)

    MATH  Google Scholar 

  35. Krishnan, D., Fergus, R.: Fast image deconvolution using hyper-laplacian priors. In: Advances in Neural Information Processing Systems, vol. 22, pp. 1033–1041 (2009)

    Google Scholar 

  36. Kutyniok, G., Lemvig, J., Lim, W.Q.: Shearlets: Multiscale Analysis for Multivariate Data. Springer, Heidelberg (2012). https://doi.org/10.1007/978-0-8176-8316-0

    Book  MATH  Google Scholar 

  37. Kutyniok, G., Lim, W.Q.: Compactly supported shearlets are optimally sparse. J. Approx. Theory 163(11), 1564–1589 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  38. Lee, H., Battle, A., Raina, R., Ng, A.Y.: Efficient sparse coding algorithms. In: Proceedings of the 19th International Conference on Neural Information Processing Systems, NIPS 2006, pp. 801–808. MIT Press (2006)

    Google Scholar 

  39. Levin, A., Lischinski, D., Weiss, Y.: A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30(2), 228–242 (2008)

    Article  Google Scholar 

  40. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. ACM Trans. Graph. (TOG) 23(3), 689–694 (2004)

    Article  Google Scholar 

  41. Levoy, M., Hanrahan, P.: Light field rendering. In: 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 31–42 (1996)

    Google Scholar 

  42. Lim, W.Q.: Nonseparable shearlet transform. IEEE Trans. Image Process. 22(5), 2056–2065 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  43. Lin, Z., Shum, H.Y.: A geometric analysis of light field rendering. Int. J. Comput. Vis. 58(2), 121–138 (2004)

    Article  Google Scholar 

  44. Mallat, S.: A Wavelet Tour of Signal Processing: The Sparse Way. Academic Press, Cambridge (2008)

    MATH  Google Scholar 

  45. McMillan, L., Bishop, G.: Plenoptic modeling: an image-based rendering system. In: 22nd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1995, pp. 39–46 (1995)

    Google Scholar 

  46. Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., Hanrahan, P., et al.: Light field photography with a hand-held plenoptic camera. Comput. Sci. Techn. R. (CSTR) 2(11), 1–11 (2005)

    Google Scholar 

  47. Overbeck, R.S., Erickson, D., Evangelakos, D., Pharr, M., Debevec, P.: A system for acquiring, processing, and rendering panoramic light field stills for virtual reality. ACM Trans. Graph. (TOG) 37, 1–15 (2018)

    Article  Google Scholar 

  48. Sahin, E., Vagharshakyan, S., Mäkinen, J., Bregovic, R., Gotchev, A.: Shearlet-domain light field reconstruction for holographic stereogram generation. In: IEEE International Conference on Image Processing (ICIP), pp. 1479–1483 (2016)

    Google Scholar 

  49. Shen, Z., Toh, K., Yun, S.: An accelerated proximal gradient algorithm for frame-based image restoration via the balanced approach. SIAM J. Imaging Sci. 4(2), 573–596 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  50. Shi, L., Hassanieh, H., Davis, A., Katabi, D., Durand, F.: Light field reconstruction using sparsity in the continuous fourier domain. ACM Trans. Graph. (TOG) 34(1), 1–13 (2014)

    Article  Google Scholar 

  51. Shin, C., Jeon, H., Yoon, Y., Kweon, I.S., Kim, S.J.: EPINET: a fully-convolutional neural network using epipolar geometry for depth from light field images. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4748–4757, June 2018

    Google Scholar 

  52. Shum, H.Y., He, L.W.: Rendering with concentric mosaics. In: 26th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1999, pp. 299–306 (1999)

    Google Scholar 

  53. Tao, M.W., Hadap, S., Malik, J., Ramamoorthi, R.: Depth from combining defocus and correspondence using light-field cameras. In: IEEE International Conference on Computer Vision (ICCV), pp. 673–680, December 2013

    Google Scholar 

  54. Vagharshakyan, S., Bregovic, R., Gotchev, A.: Accelerated shearlet-domain light field reconstruction. IEEE J. Sel. Top. Signal Process. 11(7), 1082–1091 (2017)

    Article  Google Scholar 

  55. Vagharshakyan, S., Bregovic, R., Gotchev, A.: Light field reconstruction using shearlet transform. IEEE Trans. Pattern Anal. Mach. Intell. 40(1), 133–147 (2018)

    Article  Google Scholar 

  56. Vagharshakyan, S.: Densely-sampled light field reconstruction. Ph.D. thesis, Tampere University (2020)

    Google Scholar 

  57. Vaish, V., Adams, A.: The (new) stanford light field archive (2008). http://lightfield.stanford.edu

  58. Wanner, S., Goldluecke, B.: Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36(3), 606–619 (2014)

    Article  Google Scholar 

  59. Wen, Z., Goldfarb, D., Yin, W.: Alternating direction augmented lagrangian methods for semidefinite programming. Math. Program. Comput. 2(3–4), 203–230 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  60. Wilburn, B., et al.: High performance imaging using large camera arrays. ACM Trans. Graph. (TOG) 24(3), 765–776 (2005)

    Article  MathSciNet  Google Scholar 

  61. Wilburn, B.S., Smulski, M., Lee, H.H.K., Horowitz, M.A.: Light field video camera. In: Media Processors 2002, vol. 4674, pp. 29–37 (2001)

    Google Scholar 

  62. Wood, D.N., et al.: Surface light fields for 3D photography. In: 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2000, pp. 287–296 (2000)

    Google Scholar 

  63. Wu, G., Liu, Y., Fang, L., Dai, Q., Chai, T.: Light field reconstruction using convolutional network on EPI and extended applications. IEEE Trans. Pattern Anal. Mach. Intell. 41(7), 1681–1694 (2018)

    Article  Google Scholar 

  64. Wu, G., Zhao, M., Wang, L., Dai, Q., Chai, T., Liu, Y.: Light field reconstruction using deep convolutional network on EPI. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1638–1646, July 2017

    Google Scholar 

  65. Yu, Z., Guo, X., Ling, H., Lumsdaine, A., Yu, J.: Line assisted light field triangulation and stereo matching. In: IEEE International Conference on Computer Vision (ICCV), pp. 2792–2799, December 2013

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Tampere University, Tampere, Finland

    Suren Vagharshakyan, Robert Bregovic & Atanas Gotchev

Authors
  1. Suren Vagharshakyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert Bregovic
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Atanas Gotchev
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Atanas Gotchev .

Editor information

Editors and Affiliations

  1. TU Braunschweig, Brunswick, Germany

    Marcus Magnor

  2. Facebook Zurich, Zürich, Switzerland

    Alexander Sorkine-Hornung

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Vagharshakyan, S., Bregovic, R., Gotchev, A. (2020). Densely-Sampled Light Field Reconstruction. In: Magnor, M., Sorkine-Hornung, A. (eds) Real VR – Immersive Digital Reality. Lecture Notes in Computer Science(), vol 11900. Springer, Cham. https://doi.org/10.1007/978-3-030-41816-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-41816-8_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-41815-1

  • Online ISBN: 978-3-030-41816-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy