Academic Journal of Science and Technology

ISSN: 2771-3032 | Vol. 9, No. 1, 2024

Comprehensive Review of Backpropagation Neural

Networks
Mingfeng Li
College of Mechanical Engineering, Xi'an Shiyou University, Xi’an, Shaanxi 710065, China

Abstract: The Backpropagation Neural Network (BPNN) is a deep learning model inspired by the biological neural network.
Introduced in the 1980s, the BPNN quickly became a focal point in neural network research due to its outstanding learning
capability and adaptability. The network structure consists of input, hidden, and output layers, and it optimizes weights through
the backpropagation algorithm, widely applied in image recognition, speech processing, natural language processing, and more.
The mathematical model of neurons describes the relationship between input and output, and the training process involves
adjusting weights and biases using optimization algorithms like gradient descent. In applications, BPNN excels in image
recognition, speech processing, natural language processing, and financial forecasting. Researchers continuously experiment
with optimization algorithms, including the Grey Wolf Algorithm, Genetic Algorithm, Particle Swarm Algorithm, Simulated
Annealing Algorithm, as well as comprehensive strategies and improved gradient descent algorithms. In the future, with the
ongoing development of deep learning, BPNN is poised to play a crucial role in tasks such as image recognition and speech
processing.
Keywords: Backpropagation Neural Network (BPNN); Deep learning; Network structure; Optimization algorithms.

focal point in the field of neural network research. With the

1. Introduction improvement of computing power and the rise of the deep
As technology continues to evolve, the Backpropagation learning trend, the BP neural network has evolved
Neural Network (BPNN) is emerging as a key driver in the continuously. From the initial single-layer network to the
field of deep learning, playing a pivotal role in the emergence of deep neural networks, BP networks have made
development of artificial intelligence. Since its introduction significant progress in various domains. Its powerful
in the early 1980s, BPNN has gained prominence in both nonlinear modeling capability positions the BP neural
research and practical applications, evolving from a single- network as a crucial element in machine learning. In
layer network to a deep neural network with the rise of numerous fields such as image processing, speech recognition,
computational capabilities and the advent of deep learning. and natural language processing, BP neural networks
BPNN has demonstrated outstanding performance in demonstrate outstanding performance. The neurons in the
various domains such as image processing, speech network are the fundamental units that constitute the entire
recognition, and natural language processing. Its flexibility system. Each neuron receives the output from the neurons in
and powerful fitting capabilities position it as a robust tool for the previous layer, performs a weighted sum through the
addressing complex problems. Neurons, serving as the assigned weights, and generates an output through an
fundamental units of the network, weight the outputs of the activation function. The entire BP neural network consists of
previous layer and produce an output through activation multiple layers of neurons, including the input layer, hidden
functions. The network structure comprises input, hidden, and layers, and output layer. These layers work collaboratively,
output layers, with collaborative efforts and the adjusting weights continuously through the backpropagation
backpropagation algorithm adjusting weights to optimize the algorithm to optimize the network for specific tasks. With its
network for specific tasks. profound research foundation and successful applications, the
This article delves into various aspects of BPNN, including BP neural network has become a crucial pillar in the field of
its mathematical model, network structure, feedforward and deep learning. In the future, as technology advances and
backpropagation algorithms, weight and bias updates, as well application scenarios expand, the BP neural network will
as training and optimization. The goal is to provide readers continue to play a key role in the field of machine learning.
with a clear and in-depth understanding of BPNN's core 2.1.1. Mathematical Model of Neurons
elements and offer insights into its future development. In this The mathematical model of neurons describes the
era of information explosion, BPNN, with its learning relationship between their inputs and outputs:
capabilities and extensive application areas, is at the forefront
of driving innovations in artificial intelligence. i 1
y  f ( wi  xi  b) (1)
2. Basic Concepts of Backpropagation n

(BP) Neural Networks

As shown in Equation (1): xi , represents the input, wi
2.1. Neurons and Network Structure
corresponds to the weights, b is the bias,
The BP neural network, initially proposed in the 1980s[1], f is the activation function, and y is the output of the neuron.
draws inspiration from biological neural networks. Its
superior learning ability and adaptability quickly made it a

150
2.1.2. Network Structure hidden layers and the activation function type for each neuron
BP neural networks generally consist of three main can be adjusted based on the specific task and network design.
layers[2]: 3.Output Layer: The output layer generates the final output
1.Input Layer: The input layer receives external input data result of the neural network. Neurons in this layer integrate
and passes it to the next layer of the network. The neurons in the features passed from the hidden layer, forming the
this layer are responsible for receiving and transmitting the network's overall understanding of the input data. The choice
raw input information. of the activation function in the output layer often depends on
2.Hidden Layer: The hidden layer is a core component of the nature of the problem; for example, the Sigmoid function
the neural network, responsible for processing input data and may be used in binary classification problems, while the
extracting features. Each neuron in the hidden layer gradually Softmax function might be employed in multi-class
adjusts its weights through the learning process, capturing classification problems. The structural diagram is shown in
patterns and correlations in the input data. The number of Figure 1.

Input layer Hidden layer Output layer

h1
X1 Y1

h2
X2 Y2
.
. . .
. . .
. . .
. . .
. . .
. . .
. .

Xw hq Ye

Figure 1. Schematic diagram of BP neural network structure

2.2. Feedforward and Backpropagation E

Δb(jl )   (4)
Algorithms  b(jl )
Feedforward propagation is the process of information
transmission in a neural network from the input layer to the As shown in Equations (3) and (4): s the learning rate,
output layer. For a neuron j in the l-th layer，the calculation
and E represents the loss function.
formula for its output a (jl ) is given by:
2.3. Weight and Bias Updates
The learning process involves adjusting weights and biases
 i 1 
a  f  wij(l )  ai(l 1)  b(jl ) 
(l )
j (2) using optimization algorithms such as gradient descent,
n  where the learning rate (\(\eta\)) determines the step size of
parameter updates.
(l )
As shown in Equation (2): wij is the weight connecting
( l 1)
wij( l )  wij( l )  Δwij( l ) (5)
neuron I and j，a i is the output of neuron i in the previous
(l )
layer, b j is the bias of neuron j。 b(j l )  b(jl )  Δb(jl ) (6)
Backpropagation optimizes the network weights and biases
by computing the gradient of the loss function with respect to 2.4. Training and Optimization of BP Neural
the network parameters. The gradient formulas for weights Networks
and biases are given by: In the training process of BP neural networks, the loss
function is used to evaluate the difference between the
E model's output and the actual labels. Commonly used loss
Δwij(l )   (3) functions include Mean Squared Error (MSE) and cross-
 wij(l ) entropy loss. MSE is suitable for regression problems, while
cross-entropy is typically used for classification problems.
The formulas for Mean Squared Error (MSE) and cross-

151
entropy loss are as follows: the hierarchical structure of deep neural networks
automatically learns abstract features from images, thereby
1 i 1 improving classification accuracy.
E ( yi  yˆ i ) 2 (7)
2n n 3.2. Speech Processing and Speech Recognition
BP neural networks play a crucial role in the field of speech
1 i 1 processing. The temporal nature and complexity of speech
E    yi log( yˆi )  (1  yi ) log(1  yˆi )  (8)
signals make traditional methods challenging, but BP neural
n n
networks, especially those with Long Short-Term Memory
As shown in Equations (7) and (8): n is the number of (LSTM) structures, can better capture the temporal
information in speech. They are widely used in tasks such as
samples, yi is the actual label, and yˆi is the model's speech recognition and speaker identification. Their
predicted output. successful applications have greatly advanced speech
Gradient descent is a common method for adjusting processing technology in areas such as intelligent assistants
network parameters[3], but there are some variants that can and voice search.
improve performance. Among them, Stochastic Gradient In speech recognition tasks, BP neural networks can
Descent (SGD), Batch Gradient Descent, and Mini-Batch accurately identify words and speech features by learning the
Gradient Descent are the three most common. temporal patterns in speech signals. The LSTM structure
1. Stochastic Gradient Descent (SGD): In each iteration, enables the network to handle long-term dependencies in
SGD updates parameters using a single sample. This method speech signals, effectively capturing contextual information
has a lower computational cost, but the noise from individual in speech signals.
samples may lead to unstable parameter updates. Despite this, For speaker identification, BP neural networks model the
SGD is often widely applied, especially on large datasets. speech features of speakers and distinguish between different
2. Batch Gradient Descent: Batch Gradient Descent speakers through weight adjustments during the learning
updates parameters using the entire training set, calculating process. This is valuable in applications such as voice
the average gradient. The advantage of this method is that the assistants, voice search, and secure authentication.
gradient calculation is relatively stable, but it comes with a These successful applications not only drive the
higher computational cost, especially for large datasets. Batch development of speech processing technology but also
Gradient Descent is typically used for small datasets or provide robust support for the practical application of
situations where computational resources are abundant. intelligent assistants. Through BP neural networks,
3. Mini-Batch Gradient Descent: Mini-Batch Gradient advancements have been made in speech interaction
Descent is a compromise between the above two methods, technology, including speech command recognition and
updating parameters using a small subset of samples in each speech synthesis, enabling users to engage more naturally and
iteration. This approach balances computational efficiency conveniently with smart devices.
and relatively stable parameter updates, making it the
preferred method for most deep learning tasks. Mini-Batch
3.3. Natural Language Processing
Gradient Descent often exhibits good convergence In the field of Natural Language Processing (NLP), BP
performance, especially in the case of large datasets and deep neural networks have made significant progress. The
networks. introduction of structures like Recurrent Neural Networks
The choice among these three variants of gradient descent (RNNs) enables networks to process text data more efficiently,
depends on task requirements and dataset sizes in practical achieving success in tasks such as sentiment analysis, text
applications. In the field of deep learning, Mini-Batch generation, and machine translation. The successful
Gradient Descent is a common and effective optimization application of deep learning models in the NLP domain has
method, often leading to good training results by combining significantly improved the efficiency and accuracy of
the advantages of the previous two approaches. automatically processing textual information.
In sentiment analysis tasks, BP neural networks, by
2.5. Applications of BP Neural Networks learning the semantic and emotional information in textual
BP neural networks find wide applications in various fields, data, can accurately determine the sentiment tendency of the
including image recognition, speech processing, natural text. This provides a reliable solution for applications such as
language processing, and more. Their flexibility and powerful social media sentiment analysis and sentiment evaluation in
fitting capabilities make them essential tools for solving product reviews.
complex problems. In text generation, BP neural networks, through learning
the language patterns of large amounts of textual data, can
3. Applications and Performance generate text content with a certain semantic structure. This
Optimization of BP Neural Networks finds widespread applications in automatic text
summarization and dialogue systems.
3.1. Image Recognition and Classification In machine translation tasks, BP neural networks, by
In the field of image recognition, BP neural networks have learning the correspondence between different languages, can
achieved significant success. The introduction of achieve high-quality text translation. Their role in
Convolutional Neural Networks (CNNs) enables the network multilingual communication and international collaboration
to effectively capture spatial features in images, leading to is crucial.
excellent performance in tasks such as image classification These successful applications not only elevate the level of
and object detection. Typical applications in image NLP technology but also provide robust support for the
recognition include face recognition, object recognition, and practical application of text information processing. Through

152
BP neural networks, the NLP field has made significant Algorithm into the training process of BP neural networks
progress in sentiment analysis, text generation, and machine helps the network escape from local optima and better
translation, laying the foundation for more intelligent and converge to global optimal solutions. This method, by
efficient text processing. simulating the gradual cooling process of annealing, makes
the network more flexible in searching parameter space,
3.4. Sequence Data Analysis increasing the probability of finding global optimal solutions
BP neural networks demonstrate distinct advantages in and enhancing the training effectiveness of the neural network.
handling sequential data. In the financial sector, these 3.5.2. Comprehensive Optimization Strategies
networks find extensive applications in tasks like predicting Researchers have also attempted to combine multiple
stock prices and optimizing trading strategies. By learning optimization algorithms to form comprehensive optimization
from historical market data, BP neural networks can capture strategies. By integrating Genetic Algorithm, Particle Swarm
the changing trends in stock prices, providing robust support Algorithm, and Simulated Annealing Algorithm, for example,
for investment decisions. researchers can fully leverage their respective strengths,
In meteorology, BP neural networks enhance the accuracy improving the efficiency of parameter search. This
of predicting future climate changes by assimilating temporal comprehensive strategy allows for a more comprehensive
patterns from meteorological data. This application makes exploration of parameter space, effectively enhancing the
weather forecasts more reliable, aiding in addressing the performance of BP neural networks. Through the
challenges posed by climate change and offering crucial combination of multiple algorithms, researchers can more
information for decision-makers. flexibly address different network structures and types of
3.5. Performance Evaluation and Optimization problems, further optimizing the training process of neural
networks for better performance.
3.5.1. Introduction of Optimization Algorithms 3.5.2.1 Improved Gradient Descent Algorithms
To enhance the performance of BP neural networks, Gradient Descent is a commonly used optimization
researchers have explored various optimization methods. In algorithm in deep learning. However, it has some issues, such
addition to improving network structures and adjusting as slow convergence and susceptibility to local optima. To
hyperparameters, introducing different optimization address these problems, researchers have introduced methods
algorithms has become a key means of improving with adaptive learning rates, such as Adagrad and Adam.
performance. Some common optimization algorithms include: These algorithms dynamically adjust the learning rate to adapt
3.5.1.1 Grey Wolf Algorithm Optimization more flexibly to different parameter update situations during
The Grey Wolf Algorithm simulates the hunting behavior training, improving the convergence speed and stability of the
of grey wolves and is introduced into the optimization process algorithm. This adaptive learning rate strategy makes
of BP neural networks. This algorithm includes stages such as Gradient Descent more suitable for deep learning tasks,
searching for prey, chasing and surrounding prey until the accelerating the model training process and reducing the risk
prey stops fleeing, and besieging prey. By applying the Grey of getting stuck in local optima.
Wolf Algorithm for optimization, the network can converge 3.5.2.2 Reinforcement Learning Algorithms
more effectively during training, improving learning Reinforcement learning algorithms are introduced to
efficiency. optimize the parameters of BP neural networks. By
3.5.1.2 Genetic Algorithm Optimization establishing an interactive model between the network and the
Genetic Algorithm[4] is an optimization algorithm that environment, reinforcement learning algorithms can
simulates the biological evolution process and is applied to continuously optimize the network's parameters through trial
optimize the parameters of BP neural networks. This and error, improving its performance. This approach is often
algorithm optimizes the weights and biases of the network used to handle complex tasks and enhance the network's
through operations such as selection, crossover, and mutation. generalization ability. Through reinforcement learning, the
Due to the advantage of genetic algorithms in global search, network can adjust its parameters based on environmental
the neural network is more likely to find global optimal feedback, gradually learning and optimizing its behavior
solutions. This approach helps improve the performance of strategy. This learning method helps the network make more
neural networks, enabling faster convergence and better flexible and intelligent decisions when facing unknown
results during training. environments or complex tasks, enhancing the adaptability
3.5.1.3 Particle Swarm Algorithm Optimization and performance of BP neural networks.
The Particle Swarm Algorithm[5] simulates the collective
behavior of birds or fish and is applied to parameter 4. Advances and Future Prospects in
adjustment in BP neural networks. By simulating the flight
and collective cooperation of particles, this algorithm can
Deep Learning
search for the optimal solution in parameter space. The In recent years, significant progress has been made in the
advantage of the Particle Swarm Algorithm lies in its balance field of deep learning, including innovative activation
between local search and global search, making it easier for functions, regularization methods, multimodal fusion, and
BP neural networks to converge to good solutions. This way, cross-domain applications. These advancements not only
the network can more effectively adjust parameters during enhance the performance of deep learning models but also
training, enhancing performance. expand their application domains.
3.5.1.4 Simulated Annealing Algorithm Optimization Firstly, in terms of activation functions, traditional ReLU
The Simulated Annealing Algorithm simulates the physical has been widely adopted, but the introduction of new-
principles of metal annealing, gradually searching for the generation activation functions such as Leaky ReLU and ELU
global optimal solution in the solution space as the has enhanced the nonlinear expressive power of neural
temperature decreases. Introducing the Simulated Annealing networks, effectively mitigating the vanishing gradient

153
problem. These innovations provide more stable and efficient continuous innovation and efforts, we are confident in
solutions for the training of deep learning models. Secondly, welcoming broader development opportunities in the field of
in regularization methods, the evolution of L1 and L2 deep learning and making greater contributions.
regularization, Dropout technique, and Batch Normalization
improves the model's generalization ability, accelerates References
convergence speed, and enhances robustness to initial weights. [1] Yang S, Luo L, Tan B. Research on Sports Performance
Furthermore, multimodal fusion and cross-domain Prediction Based on BP Neural Network[J]. Mobile
applications are essential directions for the development of Information Systems, 2021,2021:1-8.
deep learning technology. The fusion of deep learning and
[2] Bai Y, Luo M, Pang F. An Algorithm for Solving Robot Inverse
Convolutional Neural Networks allows for more flexible Kinematics Based on FOA Optimized BP Neural Network[J].
processing of data in different domains, such as images, text, Applied Sciences, 2021,11(15):7129.
and speech, achieving significant results and providing
possibilities for the widespread application of deep learning [3] Liu Y, Dai J, Zhao S, et al. A bidirectional reflectance
distribution function model of space targets in visible spectrum
in practical scenarios. based on GA-BP network[J]. Applied Physics B, 2020,126(6).
However, challenges remain in the future, such as model
interpretability, data privacy issues, and computational [4] Yang J, Hu Y, Zhang K, et al. An improved evolution algorithm
resource requirements. Future research needs to focus on the using population competition genetic algorithm and self-
correction BP neural network based on fitness landscape[J].
interpretability of deep learning models, develop more robust Soft Computing, 2021,25(3):1751-1776.
data privacy protection strategies, and strive to develop more
efficient computing models. Overall, as a core technology in [5] Quan G, Zhang Y, Lei S, et al. Characterization of Flow
artificial intelligence, the future development prospects of Behaviors by a PSO-BP Integrated Model for a Medium
Carbon Alloy Steel[J]. Materials, 2023,16(8):2982.
deep learning are full of infinite possibilities. Through

154