Skip to main content
Springer Nature Link
Log in

3D text segmentation and recognition using leap motion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we present a method of Human-Computer-Interaction (HCI) through 3D air-writing. Our proposed method includes a natural way of interaction without pen and paper. The online texts are drawn on air by 3D gestures using fingertip within the field of view of a Leap motion sensor. The texts consist of single stroke only. Hence gaps between adjacent words are usually absent. This makes the system different as compared to the conventional 2D writing using pen and paper. We have collected a dataset that comprises with 320 Latin sentences. We have used a heuristic to segment 3D words from sentences. Subsequently, we present a methodology to segment continuous 3D strokes into lines of texts by finding large gaps between the end and start of the lines. This is followed by segmentation of the text lines into words. In the next phase, a Hidden Markov Model (HMM) based classifier is used to recognize 3D sequences of segmented words. We have used dynamic as well as simple features for classification. We have recorded an overall accuracy of 80.3 % in word segmentation. Recognition accuracies of 92.73 % and 90.24 % have been recorded when tested with dynamic and simple features, respectively. The results show that the Leap motion device can be a low-cost but useful solution for inputting text naturally as compared to conventional systems. In future, this may be extended such that the system can successfully work on cluttered gestures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Agarwal C, Dogra DP, Saini R, Roy PP (2015) Segmentation and recognition of text written in 3d using leap motion interface. In: 3rd Asian Conference on Pattern Recognition, pages 539–543

  2. Aggarwal R, Swetha S, Namboodiri AM, Sivaswamy J, Jawahar C (2015) Online handwriting recognition using depth sensors. In: 13th International Conference on Document Analysis and Recognition, pages 1061–1065

  3. Amma C, Georgi M, Schultz T (2014) Airwriting: a wearable handwriting recognition system. Pers Ubiquit Comput 18(1):191–203

    Article  Google Scholar 

  4. Bharath A, Madhvanath S (2012) Hmm-based lexicon-driven and lexicon-free word recognition for online handwritten indic scripts. IEEE Trans Pattern Anal Mach Intell 34(4):670–682

    Article  Google Scholar 

  5. Bianne-Bernard A-L, Menasri F, Mohamad RA-H, Mokbel C, Kermorvant C, Likforman-Sulem L (2011) Dynamic and contextual information in hmm modeling for handwritten word recognition. IEEE Trans Pattern Anal Mach Intell 33(10):2066–2080

    Article  Google Scholar 

  6. Charles DK, Pedlow K, McDonough S, Shek K, Charles T (2013) An evaluation of the leap motion depth sensing camera for tracking hand and fingers motion in physical therapy. In: Interactive Technologies and Games Conference

  7. Cho O-H, Lee S-T (2014) A study about honey bee dance serious game for kids using hand gesture. Int J Multi Ubiquitous Eng 9(6):397–404

    Article  Google Scholar 

  8. Espana-Boquera S, Castro-Bleda MJ, Gorbe-Moya J, Zamora-Martinez F (2011) Improving offline handwritten text recognition with hybrid HMM/ANN models. IEEE Trans Pattern Anal Mach Intell 33(4):767–779

    Article  Google Scholar 

  9. Feng Z, Xu S, Zhang X, Jin L, Ye Z, Yang W (2012) Real-time fingertip tracking and detection using kinect depth sensor for a new writing-in-the air system. In: 4th International Conference on Internet Multimedia Computing and Service, pages 70–74

  10. Ghods V, Kabir E, Razzazi F (2013) Decision fusion of horizontal and vertical trajectories for recognition of online farsi subwords. Eng Appl Artif Intell 26(1):544–550

    Article  Google Scholar 

  11. Graves A, Liwicki M, Fernández S., Bertolami R, Bunke H, Schmidhuber J (2009) A novel connectionist system for unconstrained handwriting recognition. IEEE Trans Pattern Anal Mach Intell 31(5):855–868

    Article  Google Scholar 

  12. Jaeger S, Manke S, Waibel A (2001) Online handwriting recognition: the NPen++ recognizer. Int J Doc Anal Recognit 3:169–180

    Article  Google Scholar 

  13. Kavallieratou E, Fakotakis N, Kokkinakis G (2002) An unconstrained handwriting recognition system. Int J Doc Anal Recognit 4(4):226–242

    Article  Google Scholar 

  14. Kim SH, Jeong S, Lee G-S, Suen CY (2001) Word segmentation in handwritten korean text lines based on gap clustering techniques. In: 6th International Conference on Document Analysis and Recognition, pages 189–193

  15. Liwicki M, Scherz M, Bunke H (2006) Word extraction from on-line handwritten text lines. In: 18th International Conference on Pattern Recognition, volume 2, pages 929–933

  16. Louloudis G, Gatos B, Pratikakis I, Halatsis C (2009) Text line and word segmentation of handwritten documents. Pattern Recogn 42(12):3169–3183

    Article  MATH  Google Scholar 

  17. Manmatha R, Rothfeder JL (2005) A scale space approach for automatically segmenting words from historical handwritten documents. IEEE Trans Pattern Anal Mach Intell 27(8):1212–1225

    Article  Google Scholar 

  18. Marin G, Dominio F, Zanuttigh P (2015) Hand gesture recognition with jointly calibrated leap motion and depth sensor. Multimedia Tools Appl:1–25

  19. Marti U-V, Bunke H (2001) Text line segmentation and word recognition in a system for general writer independent handwriting recognition. In: 6th International Conference on Document Analysis and Recognition, pages 159–163

  20. Murata T, Shin J (2014) Hand gesture and character recognition based on kinect sensor. Int J Distrib Sens Netw:2014

  21. Nigam I, Vatsa M, Singh R (2014) Leap signature recognition using hoof and hot features. In: International Conference on Image Processing, pages 5012–5016

  22. Palacios JM, Sagues C, Montijano E, Llorente S (2013) Human-computer interaction based on hand gestures using rgb-d sensors. Sensors 13(9):11842–11860

    Article  Google Scholar 

  23. Papavassiliou V, Stafylakis T, Katsouros V, Carayannis G (2010) Handwritten document image segmentation into text lines and words. Pattern Recogn 43(1):369–377

    Article  MATH  Google Scholar 

  24. Paquet T, Heutte L et al (2004) Text line segmentation in handwritten document using a production system. In: 9th International Workshop on Frontiers in Handwriting Recognition, pages 245–250

  25. Plamondon R, Srihari SN (2000) On-Line and Off-Line Handwriting Recognition: A Comprehensive Survey. IEEE Trans Pattern Anal Mach Intell 22:63–84

    Article  Google Scholar 

  26. Rabiner L (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286

    Article  Google Scholar 

  27. Rahman M, Ahmed M, Qamar A, Hossain D, Basalamah S (2014) Modeling therapy rehabilitation sessions using non-invasive serious games. In: International Symposium on Medical Measurements and Applications, pages 1–4

  28. She Y, Wang Q, Jia Y, Gu T, He Q, Yang B (2014) A real-time hand gesture recognition approach based on motion features of feature points. In: International Conference on Computational Science and Engineering, pages 1096–1102

  29. Tagougui N, Kherallah M, Alimi AM (2013) Online arabic handwriting recognition: a survey. Int J Doc Anal Recognit 16(3):209–226

    Article  Google Scholar 

  30. Tian J, Qu C, Xu W, Wang S (2013) Kinwrite: Handwriting-based authentication using kinect. In: NDSS

  31. Vamsikrishna K, Dogra D, Desarkar M (2015) Computer vision assisted palm rehabilitation with supervised learning. IEEE Trans Biomed Eng 63(5):991–1001

    Article  Google Scholar 

  32. Vikram S, Li L, Russell S (2013) Handwriting and gestures in the air, recognizing on the fly. In: Proceedings of the CHI, volume 13, pages 1179–1184

  33. Wang D-H, Liu C-L, Zhou X-D (2012) An approach for real-time recognition of online chinese handwritten sentences. Pattern Recogn 45(10):3661–3675

    Article  Google Scholar 

  34. Wang J-S, Hsu Y-L, Chu C-L (2013) Online handwriting recognition using an accelerometer-based pen device. In: 2nd International Conference on Advances in Computer Science and Engineering, pages 229–232

  35. Wang Q, Xu Y, Bai X, Xu D, Chen Y, Wu X (2014) Dynamic gesture recognition using 3d trajectory

  36. Xu N, Wang W, Qu X (2015) On-line sample generation for in-air written chinese character recognition based on leap motion controller. In: Pacific Rim Conference on Multimedia, pages 171–180. Springer

  37. Zhang X, Ye Z, Jin L, Feng Z, Xu S (2013) A new writing experience: finger writing in the air using a kinect sensor. IEEE MultiMedia 20(4):85–93

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institutes of Technology, Roorkee, India

    Pradeep Kumar, Rajkumar Saini & Partha Pratim Roy

  2. School of Electrical Sciences, Indian Institutes of Technology, Bhubaneswar, India

    Debi Prosad Dogra

Authors
  1. Pradeep Kumar
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Rajkumar Saini
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Partha Pratim Roy
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Debi Prosad Dogra
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Debi Prosad Dogra.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kumar, P., Saini, R., Roy, P.P. et al. 3D text segmentation and recognition using leap motion. Multimed Tools Appl 76, 16491–16510 (2017). https://doi.org/10.1007/s11042-016-3923-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3923-z

Keywords

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy