Skip to main content

Advertisement

Springer Nature Link
Log in

Catalysis of neural activation functions: Adaptive feed-forward training for big data applications

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Deep Learning in the field of Big Data has become essential for the analysis and perception of trends. Activation functions play a crucial role in the outcome of these deep learning frameworks. The existing activation functions are hugely focused on data translation from one neural layer to another. Although they have been proven useful and have given consistent results, they are static and mostly non-parametric. In this paper, we propose a new function for modified training of neural networks that is more flexible and adaptable to the data. The proposed catalysis function works over Rectified Linear Unit (ReLU), sigmoid, tanh and all other activation functions to provide adaptive feed-forward training. The function uses vector components of the activation function to provide variational flow of input. The performance of this algorithm is tested on Modified National Institute of Standards and Technology (MNIST) and Canadian Institute for Advanced Research (CIFAR10) datasets against the conventional activation functions. Visual Geometry Group (VGG) blocks and Residual Neural Network (ResNet) architectures are used for experimentation. The proposed function has shown significant improvements in comparison to the traditional functions with a 75 ± 2.5% acuuracy across activation functions. The adaptive nature of training has drastically decreased the probability of under-fitting. The parameterization has helped increase the data learning capacity of models. On performing sensitivity analysis, the catalysis activation show slight or no changes on varying initialization parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Reddy GT, Kumar Reddy MP, Lakshmanna K, Kaluri R, Rajput DS, Srivastava G, Baker T (2020) Analysis of dimensionality reduction techniques on big data. IEEE Access 8:54776–54788

  2. Agrawal S, Sarkar S, Srivastava G, Maddikunta PKR, Gadekallu TR (2021) Genetically optimized prediction of remaining useful life. Sustainable Computing: Informatics and Systems 31:100565

  3. Chen Y, Dai X, Liu M, Chen D, Yuan L, Liu Z (2020) Dynamic ReLU. arXiv preprint arXiv:2003.10027

  4. Si J, Harris SL, Yfantis E (2018) A dynamic ReLU on neural network. In: 2018 IEEE 13th Dallas Circuits and Systems Conference (DCAS), IEEE, pp 1–6

  5. Ahn H, Chung B, Yim C (2019) Super-resolution convolutional neural networks using modified and bilateral ReLU. In: 2019 International Conference on Electronics, Information, and Communication (ICEIC), IEEE, pp 1–4

  6. Hu X, Niu P, Wang J, Zhang X (2019) A dynamic rectified linear activation units. IEEE Access 7:180409–16

    Article  Google Scholar 

  7. Kim J, Kim S, Lee M (2015) Convolutional neural network with biologically inspired on/off relu. In: International Conference on Neural Information Processing, Springer, Cham, pp 316–323

  8. Chung H, Lee SJ, Park JG (2016) Deep neural network using trainable activation functions. In: 2016 International Joint Conference on Neural Networks (IJCNN), IEEE, pp 348–352

  9. Nwankpa C, Ijomah W, Gachagan A, Marshall S (2018) Activation functions: Comparison of trends in practice and research for deep learning. arXiv preprint arXiv:1811.03378

  10. Pedamonti D (2018) Comparison of non-linear activation functions for deep neural networks on MNIST classification task. arXiv preprint arXiv:1804.02763

  11. Yamashita R, Nishio M, Do RK, Togashi K (2018) Convolutional neural networks: an overview and application in radiology. Insights Into Imaging 9(4):611–29

    Article  Google Scholar 

  12. Kumar C, Punitha R (2020) YOLOv3 and YOLOv4: Multiple object detection for surveillance applications. In: 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), IEEE, pp 1316–1321

  13. Sharma N, Mandal R, Sharma R, Pal U, Blumenstein M (2018) Signature and logo detection using deep CNN for document image retrieval. In: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), IEEE, pp 416–422

  14. Xie Q, Luong MT, Hovy E, Le QV (2020) Self-training with noisy student improves imagenet classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 10687–10698

  15. Kolesnikov A, Beyer L, Zhai X, Puigcerver J, Yung J, Gelly S, Houlsby N (2019) Big transfer (BiT): General visual representation learning. arXiv preprint arXiv:1912.11370

  16. Kirillov A, Wu Y, He K, Girshick R (2020) Pointrend: Image segmentation as rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 9799–9808

  17. Trottier L, Gigu P, Chaib-draa B (2017) Parametric exponential linear unit for deep convolutional neural networks. In: 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), IEEE, pp 207–214

  18. Svozil D, Kvasnicka V, Pospichal J (1997) Introduction to multi-layer feed-forward neural networks. Chemometrics and Intelligent Laboratory Systems 39(1):43–62

    Article  Google Scholar 

  19. Narayan S (1997) The generalized sigmoid activation function: Competitive supervised learning. Inform Sci 99(1–2):69–82

    Article  MathSciNet  Google Scholar 

  20. Redmon J, Farhadi A (2018) Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767

  21. Bochkovskiy A, Wang CY, Liao HY (2020) YOLOv4: Optimal Speed and Accuracy of Object Detection. arXiv preprint arXiv:2004.10934

  22. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 770–778

  23. Chen YT, Chen TS, Chen J (2018) A LeNet Based Convolution Neural Network for Image Steganalysis on Multiclass Classification. DEStech Transactions on Computer Science and Engineering, (ccme)

  24. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  25. Krizhevsky, A., Nair, V. and Hinton, G (2009) FAR-10 (canadian institute for advanced research). http://www.cs.toronto.edu/kriz/cifar.html, 5

  26. LeCun, Y (998) The MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/

  27. Azmoodeh A, Dehghantanha A, Choo KK (2019) Big data and internet of things security and forensics: challenges and opportunities. In: Handbook of Big Data and IoT Security, Springer, Cham, pp 1–4

  28. Zhang Q, Yang LT, Chen Z, Li P (2018) A survey on deep learning for big data. Inform Fusion 1(42):146–57

    Article  Google Scholar 

  29. Banerjee C, Mukherjee T, Pasiliao E (2020) Feature representations using the reflected rectified linear unit (RReLU) activation. Big Data Mining and Analytics 3(2):102–120. https://doi.org/10.26599/BDMA.2019.9020024

    Article  Google Scholar 

  30. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, pp 249–256

  31. He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1026–1034

  32. Klambauer G, Unterthiner T, Mayr A, Hochreiter S (2017) Self-normalizing neural networks. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp 972–981

  33. Biswas K, Banerjee S, Pandey AK (2021) TanhSoftÃ-Dynamic trainable activation functions for faster learning and better performance. IEEE Access 9:120613–120623. https://doi.org/10.1109/ACCESS.2021.3105355

    Article  Google Scholar 

  34. Saha S, Mathur A, Pandey A, Arun Kumar H (2021) DiffAct: A unifying framework for activation functions. In: 2021 International Joint Conference on Neural Networks (IJCNN), pp 1–8, https://doi.org/10.1109/IJCNN52387.2021.9534391.

  35. Pratama K, Kang DK (2021) Trainable activation function with differentiable negative side and adaptable rectified point. Appl Intell 51(3):1784–1801

    Article  Google Scholar 

  36. Zhao H, Liu F, Li L, Luo C (2018) A novel softplus linear unit for deep convolutional neural networks. Appl Intell 48(7):1707–1720

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Vellore Institute of Technology, School of Computer Science and Engineering, Vellore, Tamil Nadu, India

    Sagnik Sarkar & Shaashwat Agrawal

  2. Department of Computer Science, College of Computing and Informatics, University of Sharjah, Sharjah, UAE

    Thar Baker

  3. Vellore Institute of Technology, School of Information Technology and Engineering, Vellore, Tamil Nadu, India

    Praveen Kumar Reddy Maddikunta & Thippa Reddy Gadekallu

Authors
  1. Sagnik Sarkar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaashwat Agrawal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thar Baker
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Praveen Kumar Reddy Maddikunta
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thippa Reddy Gadekallu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Thar Baker or Thippa Reddy Gadekallu.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Table 3 Symbol representations
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sarkar, S., Agrawal, S., Baker, T. et al. Catalysis of neural activation functions: Adaptive feed-forward training for big data applications. Appl Intell 52, 13364–13383 (2022). https://doi.org/10.1007/s10489-021-03082-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-03082-y

Keywords

Search

Navigation

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy