Abstract
Lately, Machine Learning has taken a crucial role in the society in different vertical sectors. For complex problems with high-dimensionality, Deep Learning has become an efficient solution for learning in the context of supervisioned learning. Deep Learning [1] consists in using Artificial Neural Networks (ANN or NN) with several hidden layers, typically also with a large number of nodes in each layer.
Similar content being viewed by others
References
J. Schmidhuber, Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015)
W.S. McCulloch, W. Pitts, A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 5(4), 115–133 (1943)
A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (2012), pp. 1097–1105
G.L. Martin, J.A. Pittman, Recognizing hand-printed letters and digits using backpropagation learning. Neural Comput. 3(2), 258–267 (1991)
G. Zhang, M.Y. Hu, B.E. Patuwo, D.C. Indro, Artificial neural networks in bankruptcy prediction: general framework and cross-validation analysis. Eur. J. Oper. Res. 116(1), 16–32 (1999)
M.H. Ebell, Artificial neural networks for predicting failure to survive following in-hospital cardiopulmonary resuscitation. J. Family Pract. 36(3), 297–304 (1993)
D.A. Pomerleau, Efficient training of artificial neural networks for autonomous navigation. Neural Comput. 3(1), 88–97 (1991)
W.S. Sarle, On Computing Number of Neurons in Hidden Layer (1995)
A. Jacobs, The pathologies of big data. Commun. ACM 52(8), 36–44 (2009)
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D.G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, X. Zheng, Tensorflow: a system for large-scale machine learning, in Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. OSDI’16, Berkeley, CA, USA, USENIX Association (2016), pp. 265–283
J. Bergstra, O. Breuleux, P. Lamblin, R. Pascanu, O. Delalleau, G. Desjardins, I. Goodfellow, A. Bergeron, Y. Bengio, P. Kaelbling, Theano: Deep Learning on GPUs with Python
R. Collobert, S. Bengio, J. Marithoz, Torch: A Modular Machine Learning Software Library (2002)
B.J. Erickson, P. Korfiatis, Z. Akkus, T. Kline, K. Philbrick, Toolkits and libraries for deep learning. J. Digit. Imaging 30(4), 400–405 (2017)
S. Lawrence, C.L. Giles, A.C. Tsoi, What size neural network gives optimal generalization? convergence properties of backpropagation (1998)
F. Cao, T. Xie, The construction and approximation for feedforword neural networks with fixed weights, in International Conference on Machine Learning and Cybernetics, ICMLC 2010, Qingdao, China, 11–14 July 2010, Proceedings (2010), pp. 3164–3168
N. Guliyev, V. Ismailov, On the approximation by single hidden layer feedforward neural networks with fixed weights (2017)
S. Koturwar, S. Merchant, Weight initialization of deep neural networks (DNNs) using data statistics. arXiv:1710.10570 (2017)
S.K. Kumar, On weight initialization in deep neural networks. arXiv:1704.08863 (2017)
D. Mishkin, J. Matas, All you need is a good init. arXiv:1511.06422 (2015)
I. Goodfellow, Y. Bengio, A. Courville, Y. Bengio, Deep Learning, vol. 1 (MIT Press, Cambridge, 2016)
V. Nair, G.E. Hinton, Rectified linear units improve restricted Boltzmann machines, in Proceedings of the 27th International Conference on Machine Learning (ICML-10) (2010), pp. 807–814
D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning representations by back-propagating errors. Nature 323(6088), 533 (1986)
M. Lichman, UCI Machine Learning Repository (2013)
K.O. Stanley, D.B. D’Ambrosio, J. Gauci, A hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185–212 (2009)
W. McKinney et al., Data structures for statistical computing in python, in Proceedings of the 9th Python in Science Conference, vol. 445 (2010), pp. 51–56
A. Trask, A Neural Network in 11 Lines of Python (2013)
L. Gu, H. Li, Memory or time: performance evaluation for iterative operation on hadoop and spark, in 2013 IEEE 10th International Conference on High Performance Computing and Communications and 2013 IEEE International Conference on Embedded and Ubiquitous Computing (HPCC\(\_\)EUC) (IEEE, 2013), pp. 721–727
G. Dahl, A. McAvinney, T. Newhall et al., Parallelizing neural network training for cluster systems, in Proceedings of the IASTED International Conference on Parallel and Distributed Computing and Networks (ACTA Press, 2008), pp. 220–225
S. Sanfilippo, P. Noordhuis, Redis (2009)
J. Reese, S. Zaranek, GPU programming in MATLAB, in MathWorks News & Notes (The MathWorks Inc., Natick, MA, 2012), pp. 22–25
M.M. Chakravarty, G. Keller, S. Lee, T.L. McDonell, V. Grover, Accelerating Haskell array codes with multicore GPUs, in Proceedings of the Sixth Workshop on Declarative Aspects of Multicore Programming (ACM, 2011), pp. 3–14
A. Fonseca, B. Cabral, Æminiumgpu: an intelligent framework for GPU programming, in Facing the Multicore-Challenge III (Springer, 2013), pp. 96–107
B. Catanzaro, M. Garland, K. Keutzer, Copperhead: compiling an embedded data parallel language. ACM SIGPLAN Not. 46(8), 47–56 (2011)
S.K. Lam, A. Pitrou, S. Seibert, Numba: a LLVM-based python JIT compiler, in Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC (ACM, 2015), p. 7
B. Snyder, D. Bosnanac, R. Davies, ActiveMQ in Action, vol. 47 (Manning, 2011)
E.D. Karnin, A simple procedure for pruning back-propagation trained neural networks. IEEE Trans. Neural Netw. 1(2), 239–242 (1990)
D.B. Fogel, L.J. Fogel, V. Porto, Evolving neural networks. Biol. Cybern. 63(6), 487–493 (1990)
J.F. de Freitas, M. Niranjan, A.H. Gee, A. Doucet, Sequential Monte Carlo methods to train neural network models. Neural Comput. 12(4), 955–993 (2000)
V. Sze, Y. Chen, T. Yang, J.S. Emer, Efficient processing of deep neural networks: a tutorial and survey. arXiv:1703.09039 (2017)
Acknowledgements
The first author was supported by the LASIGE Research Unit (UID/CEC/00408/2013).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Fonseca, A., Cabral, B. (2019). Designing a Neural Network from Scratch for Big Data Powered by Multi-node GPUs. In: Balas, V., Roy, S., Sharma, D., Samui, P. (eds) Handbook of Deep Learning Applications. Smart Innovation, Systems and Technologies, vol 136. Springer, Cham. https://doi.org/10.1007/978-3-030-11479-4_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-11479-4_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-11478-7
Online ISBN: 978-3-030-11479-4
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)