Skip to main content
Springer Nature Link
Log in

Skeleton-based human activity recognition with wifi CSI using a hybrid approach combining convolutional neural network and long short term memory

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Pose estimation based on visual images is evolving, but it is also limited by environmental factors such as occlusion and darkness. Due to its non-intrusive and ubiquitous characters, WiFi Channel State Information (CSI)-based human activity recognition attracts immense attention. In this paper, a CSI-based passive sensing system is proposed to predict joint points of human skeleton for activity recognition. The system leverages a pair of ESP32 based CSI sensors with bidirectional link that can prevent unidirectional link from missing important activity information to collect the amplitude of CSI signals. The Kinect 2.0 is employed to obtain skeleton data as ground truth label synchronously. A hybrid deep neural network composed of Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) is utilized to extract features of CSI signal and map to corresponding human skeleton. K-means clustering algorithm is incorporated to cull the outliers. Experimental results demonstrate that the proposed system achieves satisfactory results with 3.489% average error.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Availability of data and materials

The datasets generated during the current study are available from the corresponding author on reasonable request.

References

  1. Yan, K., Wang, F., Qian, B., et al : Person-in-wifi 3d: End-to-end multi-person 3d pose estimation with wi-fi. In: (ed) 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Seattle, WA, USA, p 969–978 (2024)

  2. Yang ,Z., Lin, M., Zhong, X., et al : Good is bad: Causality inspired cloth-debiasing for cloth-changing person re-identification. In: (ed) 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Vancouver, BC, Canada, p 1472–1481 (2023)

  3. You, Y., Liu, H., Wang, T., et al : Co-evolution of pose and mesh for 3d human body estimation from video. In: (ed) 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, Paris, France, p 14917–14927 (2023)

  4. Tina, Sharma, AK., Tomar, S., et al : Various approaches of human activity recognition: A review. In: (ed) 2021 5th International Conference on Computing Methodologies and Communication (ICCMC). IEEE, Erode, India, pp. 1668–1676 (2021)

  5. Yin, C., Chen, J., Miao, X., et al.: Device-free human activity recognition with low-resolution infrared array sensor using long short-term memory neural network. Sensors 21, 3551 (2021). https://doi.org/10.3390/s21103551

    Article  Google Scholar 

  6. Yin, C., Miao, X., Chen, J., et al.: Human activity recognition with low-resolution infrared array sensor using semi-supervised cross-domain neural networks for indoor environment. IEEE Internet Things J. 10, 11761–11772 (2023). https://doi.org/10.1109/JIOT.2023.3243944

    Article  Google Scholar 

  7. D. Gian, T., Dac Lai, T., Van Luong, T., et al : Hpe-li: Wifi-enabled lightweight dual selective kernel convolution for human pose estimation. In: (ed) Computer Vision – ECCV 2024. Springer Nature Switzerland, Cham, p 93–111 (2024)

  8. Zhou, Y., Huang, H., Yuan, S., et al.: Metafi++: Wifi-enabled transformer based human pose estimation for metaverse avatar simulation. IEEE Internet Things J. 10, 14128–14136 (2023). https://doi.org/10.1109/JIOT.2023.3262940

    Article  Google Scholar 

  9. Liu, J., Chen, Y., Wang, Y., et al.: Monitoring vital signs and postures during sleep using wifi signals. IEEE Internet Things J. 5, 2071–2084 (2018). https://doi.org/10.1109/JIOT.2018.2822818

    Article  Google Scholar 

  10. Chen, C., Han, Y., Chen, Y., et al.: Tr-breath: Time-reversal breathing rate estimation and detection. IEEE Trans. Biomed. Eng. 65, 489–501 (2018). https://doi.org/10.1109/TBME.2017.2699422

    Article  Google Scholar 

  11. Yang, X., Yin, Y., Chen, P., et al.: A device-free intelligent alarm system based on the channel state information. IEEE Trans. Veh. Technol. 69, 11419–11427 (2020). https://doi.org/10.1109/TVT.2020.3010645

    Article  Google Scholar 

  12. Xu, Q., Wang, B., Zhang, F., et al.: Wireless ai in smart car: How smart a car can be? IEEE Access 8, 55091–55112 (2020). https://doi.org/10.1109/ACCESS.2020.2978531

    Article  Google Scholar 

  13. Zhuo, H., Wu, X., Zhong, Q., et al.: Position-free breath detection during sleep via commodity wifi. IEEE Sens. J. 23, 24874–24884 (2023). https://doi.org/10.1109/JSEN.2023.3309839

    Article  Google Scholar 

  14. Wang, H., Zhang, D., Wang, Y., et al : Rt-fall: A real-time and contactless fall detection system with commodity wifi devices. IEEE Transactions on Mobile Computing 16:511–526. https://doi.org/10.1109/TMC.2016.2557795 (2017a)

  15. Wang, Y., Wu, K., Ni LM Wifall: Device-free fall detection by wireless networks. IEEE Transactions on Mobile Computing 16:581–594. https://doi.org/10.1109/TMC.2016.2557792 (2017b)

  16. Ding, J., Wang, Y.: A wifi-based smart home fall detection system using recurrent neural network. IEEE Trans. Consum. Electron. 66, 308–317 (2020). https://doi.org/10.1109/TCE.2020.3021398

    Article  Google Scholar 

  17. Chu, Y., Cumanan, K., Sankarpandi, S.K., et al.: Deep learning-based fall detection using wifi channel state information. IEEE Access 11, 83763–83780 (2023). https://doi.org/10.1109/ACCESS.2023.3300726

    Article  Google Scholar 

  18. Wang, X., Gao, L., Mao, S.: Biloc: Bi-modal deep learning for indoor localization with commodity 5ghz wifi. IEEE Access 5, 4209–4220 (2017). https://doi.org/10.1109/ACCESS.2017.2688362

    Article  Google Scholar 

  19. Song, Q., Guo, S., Liu, X., et al.: Csi amplitude fingerprinting-based nb-iot indoor localization. IEEE Internet Things J. 5, 1494–1504 (2018). https://doi.org/10.1109/JIOT.2017.2782479

    Article  Google Scholar 

  20. Han, S., Li, Y., Meng, W., et al.: Indoor localization with a single wi-fi access point based on ofdm-mimo. IEEE Syst. J. 13, 964–972 (2019). https://doi.org/10.1109/JSYST.2018.2823358

    Article  Google Scholar 

  21. Ali, K., Liu, A.X., Wang, W., et al.: Recognizing keystrokes using wifi devices. IEEE J. Sel. Areas Commun. 35, 1175–1190 (2017). https://doi.org/10.1109/JSAC.2017.2680998

    Article  Google Scholar 

  22. Niu, K., Zhang, F., Jiang, Y., et al.: Wimorse: A contactless morse code text input system using ambient wifi signals. IEEE Internet Things J. 6, 9993–10008 (2019). https://doi.org/10.1109/JIOT.2019.2934904

    Article  Google Scholar 

  23. Shah SW, Kanhere SS (2018) Wi-access: Second factor user authentication leveraging wifi signals. In: (ed) 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops). IEEE, Athens, Greece, p 330–335

  24. Zheng, X., Wang, J., Shangguan, L., et al.: Design and implementation of a csi-based ubiquitous smoking detection system. IEEE/ACM Trans. Networking 25, 3781–3793 (2017). https://doi.org/10.1109/TNET.2017.2752367

    Article  Google Scholar 

  25. Xiao, F., Guo, Z., Ni, Y., et al.: Artificial intelligence empowered mobile sensing for human flow detection. IEEE Network 33, 78–83 (2019). https://doi.org/10.1109/MNET.2018.1700356

    Article  Google Scholar 

  26. Abdelnasser, H., Harras, K., Youssef, M.: A ubiquitous wifi-based fine-grained gesture recognition system. IEEE Trans. Mob. Comput. 18, 2474–2487 (2019). https://doi.org/10.1109/TMC.2018.2879075

    Article  Google Scholar 

  27. Yang, J., Zou, H., Zhou, Y., et al.: Learning gestures from wifi: A siamese recurrent convolutional architecture. IEEE Internet Things J. 6, 10763–10772 (2019). https://doi.org/10.1109/JIOT.2019.2941527

    Article  Google Scholar 

  28. Zhao, Y., Gao, R., Liu, S., et al.: Device-free secure interaction with hand gestures in wifi-enabled iot environment. IEEE Internet Things J. 8, 5619–5631 (2021). https://doi.org/10.1109/JIOT.2020.3032623

    Article  Google Scholar 

  29. Yang, J., Liu, Y., Liu, Z., et al.: (2021) A framework for human activity recognition based on wifi csi signal enhancement. International Journal of Antennas and Propagation 1, 1–18 (2021)

    Google Scholar 

  30. Yan, H., Zhang, Y., Wang, Y., et al.: Wiact: A passive wifi-based human activity recognition system. IEEE Sens. J. 20, 296–305 (2020). https://doi.org/10.1109/JSEN.2019.2938245

    Article  Google Scholar 

  31. Hao, Z., Duan, Y., Dang, X., et al Csi-hc: A wifi-based indoor complex human motion recognition method. Mobile Information Systems 2020():1–20 (2020)

  32. Dang, X., Huang, Y., Hao, Z., et al Pca-kalman: device-free indoor human behavior detection with commodity wi-fi. EURASIP Journal on Wireless Communications and Networking 2018(1): (2018)

  33. Wang, F., Feng, J., Zhao, Y., et al.: Joint activity recognition and indoor localization with wifi fingerprints. IEEE Access 7, 80058–80068 (2019). https://doi.org/10.1109/ACCESS.2019.2923743

    Article  Google Scholar 

  34. Jin, R., Chen, Z., Wu, K., et al Multi-feature fused bidirectional long short-term memory for remaining useful life prediction. In:2021 International Conference on Sensing, Measurement and Data Analytics in the era of Artificial Intelligence (ICSMD).pp.1-5 (2021)

  35. Xiao, C., Lei, Y., Ma, Y., et al.: Deepseg: Deep-learning-based activity segmentation framework for activity recognition using wifi. IEEE Internet Things J. 8, 5669–5681 (2021). https://doi.org/10.1109/JIOT.2020.3033173

    Article  Google Scholar 

  36. Wang, W., Liu, A.X., Shahzad, M., et al.: Device-free human activity recognition using commercial wifi devices. IEEE J. Sel. Areas Commun. 35, 1118–1131 (2017). https://doi.org/10.1109/JSAC.2017.2679658

    Article  Google Scholar 

  37. Feng, C., Arshad, S., Zhou, S., et al.: Wi-multi: A three-phase system for multiple human activity recognition with commercial wifi devices. IEEE Internet Things J. 6, 7293–7304 (2019). https://doi.org/10.1109/JIOT.2019.2915989

    Article  Google Scholar 

  38. Chen, Z., Zhang, L., Jiang, C., et al.: Wifi csi based passive human activity recognition using attention based blstm. IEEE Trans. Mob. Comput. 18, 2714–2724 (2019). https://doi.org/10.1109/TMC.2018.2878233

    Article  Google Scholar 

  39. Bulugu, I.: Gesture recognition system based on cross-domain csi extracted from wi-fi devices combined with the 3d cnn. Signal, Image and Video Processing, Springer Nature 17, 3201–3209 (2023). https://doi.org/10.1007/s11760-023-02545-8

    Article  Google Scholar 

  40. Khan, I.U., LJAfzal S,: Human activity recognition via hybrid deep learning based model. Sensors 22(1), 323 (2022)

  41. Guo, L., Lu, Z., Wen, X., et al.: From signal to image: Capturing fine-grained human poses with commodity wi-fi. IEEE Commun. Lett. 24, 802–806 (2020). https://doi.org/10.1109/LCOMM.2019.2961890

    Article  Google Scholar 

  42. Gao, Q., Wang, J., Ma, X., et al.: Csi-based device-free wireless localization and activity recognition using radio image features. IEEE Trans. Veh. Technol. 66, 10346–10356 (2017). https://doi.org/10.1109/TVT.2017.2737553

    Article  Google Scholar 

  43. Lee, H., Ahn, C.R., Choi, N.: Fine-grained occupant activity monitoring with wi-fi channel state information: Practical implementation of multiple receiver settings. Adv. Eng. Inform. 46, 101147 (2020). https://doi.org/10.1016/j.aei.2020.101147

    Article  Google Scholar 

  44. Daqing, Z., Hao, W., Dan, W.: Toward centimeter-scale human activity sensing with wi-fi signals. Computer 50(1), 48–57 (2017)

    Google Scholar 

  45. Wang, Z., Huang, Z., Zhang, C., et al.: Csi-based human sensing using model-based approaches: a survey. Journal of Computational Design and Engineering 8(2), 510–523 (2021)

    Article  Google Scholar 

  46. Sheng, B., Fang, Y., Xiao, F., et al.: An accurate device-free action recognition system using two-stream network. IEEE Trans. Veh. Technol. 69, 7930–7939 (2020). https://doi.org/10.1109/TVT.2020.2993901

    Article  Google Scholar 

  47. Wang, F., Gong, W., Liu, J.: On spatial diversity in wifi-based human activity recognition: A deep learning-based approach. IEEE Internet Things J. 6, 2035–2047 (2019). https://doi.org/10.1109/JIOT.2018.2871445

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the Natural Science Foundation of Fujian Province, China (No. 2022J01566)

Author information

Author notes

  1. Zhouwang Wei, Yixuan Tong, Hao Jiang, Xiren Miao and Cunyi Yin have contributed equally to this work.

Authors and Affiliations

  1. College of Electrical Engineering and Automation, Fuzhou University, Fuzhou, 350108, China

    Jing Chen, Zhouwang Wei, Yixuan Tong, Hao Jiang, Xiren Miao & Cunyi Yin

Authors
  1. Jing Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhouwang Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yixuan Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hao Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiren Miao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Cunyi Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: Jing Chen; Methodology: Jing Chen, Zhouwang Wei, Yixuan Tong; Formal analysis and investigation: Zhouwang Wei, Yixuan Tong; Writing - original draft preparation: Zhouwang Wei, Yixuan Tong; Writing - review and editing: Jing Chen, Hao Jiang; Funding acquisition: Jing Chen, Hao Jiang, Xiren Miao; Resources: Jing Chen, Hao Jiang, Xiren Miao; Supervision: Jing Chen, Hao Jiang. All authors reviewed the manuscript.

Corresponding author

Correspondence to Hao Jiang.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, J., Wei, Z., Tong, Y. et al. Skeleton-based human activity recognition with wifi CSI using a hybrid approach combining convolutional neural network and long short term memory. Multimedia Systems 30, 381 (2024). https://doi.org/10.1007/s00530-024-01586-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01586-4

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy