Abstract
In medical domain, huge amounts of data are generated at all times. These data are usually difficult to access, with poor data quality and many data islands. Besides, with a wide range of sources and complex structure, these data contain essential information and are difficult to manage. However, few existing data management frameworks based on Data Lake excel in solving the persistence and the analysis efficiency for medical multi-source heterogeneous data. In this paper, we propose an efficient Multi-source Heterogeneous Data Lake Platform (MHDP) to realize the efficient medical data management. Firstly, we propose an efficient and unified method based on Data Lake to store data of different types and different sources persistently. Secondly, based on the unified data store, an efficient multi-source heterogeneous data fusion is implemented to effectively manage data. Finally, an efficient data query strategy is carried out to assist doctors in medical decision-making. In-depth analysis on applications shows that MHDP delivers better performance for data management in medical domain.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Lee, C., Yoon, H.: Medical big data: promise and challenges. Kidney Res. Clin. Pract. 36(1), 3–11 (2017)
Kalkman, S., Mostert, M., Beauvisage, N., et al.: Responsible data sharing in a big data-driven translational research platform: lessons learned. BMC Med. Inform. Decis. Mak. 19(1), 283 (2019)
Mitchell, J., Naddaf, R., Davenport, S.: A medical microcomputer database management system. Methods Inf. Med. 24(2), 73–78 (1985)
Mohamad, B., Orazio, L., Gruenwald, L.: Towards a hybrid row-column database for a cloud-based medical data management system. In: 1st International Workshop on Cloud Intelligence, pp. 1–4. ACM, New York (2012)
Sebaa, A., Chikh, F., Nouicer, A., et al.: Medical big data warehouse: architecture and system design, a case study: improving healthcare resources distribution. J. Med. Syst. 42, 59 (2018)
Farooqui, N., Mehra, R.: Design of a data warehouse for medical information system using data mining techniques. In: 5th International Conference on Parallel Distributed and Grid Computing, pp. 199–203. IEEE, New York (2018)
Farid, M., Roatis, A., LLyas, F., et al.: CLAMS: bringing quality to Data Lakes. In: 2016 International Conference on Management of Data, pp. 2089–2092. ACM, New York (2016)
Alserafi, A., Abello, A., Romero, O., et al.: Towards information profiling: data lake content metadata management. In: 16th International Conference on Data Mining Workshops, pp. 178–185. IEEE, New York (2016)
Dixon, J.: Pentaho, Hadoop, and data lakes. https://jamesdixon.woedpress.com/2010/10/14/pentaho-hadoop-and-data-lakes/. Accessed 25 May 2021
Mesterhazy, J., Olson, G., Datta, S.: High performance on-demand de-identification of a petabyte-scale medical imaging data lake. In: CoRR abs/2008.01827 (2020)
Hai, R., Geisler, S., Quix, C.: Constance: an intelligent data lake system. In: 2016 International Conference on Management of Data, pp. 2097–2100. ACM, New York (2016)
Walker, C., Alrehamy, H.: Personal Data Lake with data gravity Pull. In: 5th International Conference on Big Data and Cloud Computing, pp. 160–167. IEEE, New York (2015)
Bozena, M., Marek, S., Dariusz, M.: Soft and declarative fishing of information in big data lake. IEEE Trans. Fuzzy Syst. 26(5), 2732–2747 (2018)
Alhgaish, A., Alzyadat, W., Alfayoumi, M., et al.: Preserve quality medical drug data toward meaningful data lake by cluster. Int. J. Recent Technol. Eng. 8(3), 270–277 (2019)
Maini, E., Venkateswarlu, B., Gupta, A.: Data lake-an optimum solution for storage and analytics of big data in cardiovascular disease prediction system. Int. J. Comput. Eng. Manag. 21(6), 33–39 (2018)
Kachaoui, J., Larioui, J., Belangour, A.: Towards an ontology proposal model in data lake for real-time COVID-19 cases prevention. Int. J. Online Biomed. Eng. 16(9), 123–136 (2020)
Acknowledgements
This work was supported by National Key R&D Program of China (2019YFC0119600).
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ren, P. et al. (2021). MHDP: An Efficient Data Lake Platform for Medical Multi-source Heterogeneous Data. In: Xing, C., Fu, X., Zhang, Y., Zhang, G., Borjigin, C. (eds) Web Information Systems and Applications. WISA 2021. Lecture Notes in Computer Science(), vol 12999. Springer, Cham. https://doi.org/10.1007/978-3-030-87571-8_63
Download citation
DOI: https://doi.org/10.1007/978-3-030-87571-8_63
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87570-1
Online ISBN: 978-3-030-87571-8
eBook Packages: Computer ScienceComputer Science (R0)
