Perceptron Networks and Applications

M. Ali Akcayol
Gazi University
Department of Computer Engineering
Administrative
 Grading
 Midterm: 35%
 Homeworks: 25%
 Final: 40%

 Textbook
 K. Mehrotra, C. Mohan and S. Ranka, Elements of Artificial Neural
Networks, The MIT Press, 1996.

 Other helpful texts

 M. Hagan, H. Demuth and M. Beale, Neural Network Design, PWS
Publishing Company, 1996.
 S. Haykin, Neural Networks: A Comprehensive Foundation 2nd edition,
Prentice Hall, 1999.
 L. V. Fausett, Fundamentals of Neural Networks: Architectures,
Algorithms and Applications, Prentice Hall, 1993.
2
Administrative
 Notes for homeworks & reports:
 This is an individual assignment. All the work should be the student‘s
and in accordance with the ethical policies.
 All resources should be cited in the text and the bibliographic
information should be given end of the report.
 All assignments are due one week.
 Late homeworks will not be accepted.
 Submit a single pdf file which contains the report and the all
attachments.
 The homework file name should be formatted as:
StudentNumber_CourseCode_HomeworkNumber.pdf
 The midterm report file name should be formatted as:
StudentNumber_CourseCode_Midterm.pdf
 The final report file name should be formatted as:
StudentNumber_CourseCode_Final.pdf

3
Course outline
 Introduction
 Neural network architectures
 Perceptrons
 Single layer neural networks
 Multilayer neural networks
 Learning rules
 Backpropagation
 Recurrent neural networks
 Self organising maps
 Hopfield neural networks

4
Content

 Introduction
 History of neural networks
 Biological neurons
 Artificial neuron models

5
Introduction
 A system must have at least three abilities in order to be
intelligent:
 it must be able to receive information by itself,
 it must have a flexible structure to represent and
integrate information,
 it must have a mechanism to adapt itself to the
environment using the acquired information.

6
Introduction
 The goal of neural network research is to realize an
artificial intelligent system using the human brain as the
model.
 There are three basic problems in this area:
 What kind of structure or model should we use?
 How to train or design the neural networks?
 How to use neural networks for knowledge
acquisition?

7
Introduction
 This course introduces:
 the basic models
 learning algorithms
 applications of neural networks
 After this course, you should be able to know how to use
neural networks for solving different problems.

8
Introduction
 The artificial neural networks are all variations on the
parallel distributed processing idea.
 Many tasks involving intelligence or pattern recognition are
extremely difficult to automate.
 Animals recognize various objects and make sense out of
the large amount of visual information in their
surroundings, apparently requiring very little effort.
 The neural network of a human contains a large number
of interconnected neurons.
 Artificial neural networks refer to computing systems inspired
from the analogy of biological neural networks.

9
Introduction
 ANN as a directed graph consists of a set of nodes
(vertices) and a set of connections (edges/links/arcs)
connecting pairs of nodes.
 Each node performs some simple computations, and each
connection conveys a signal from one node to another.
 Connection strength or weight indicates that a signal is
amplified or diminished by a connection.
 Different weights result in different functions in neural
networks.

10
Introduction
 Generally weights are initialized randomly.
 A learning algorithm must be used to determine weights for
the desired task.

x1 x2 o
0 0 0
0 1 0
1 0 0
1 1 1
11
Content

 Introduction
 History of neural networks
 Biological neurons
 Artificial neuron models

12
History of neural networks
 The studies about the neural networks date back to a
century ago.
 The roots of all work on neural networks are in
neurobiological studies.
 Psychologists tried to understand how learning, forgetting,
recognition are accomplished by humans.
 McCulloch and Pitts developed the first mathematical model
of a neuron.
 Neural network learning rules mostly use gradient descent
search procedures.

13
History of neural networks
 1949 Hebb's learning rule modifies weights by examining
whether two connected nodes are simultaneously ON or
OFF.
 1958 Rosenblatt's perceptron model and the learning rule
are based on gradient descent to change weights
depending on the desired outputs.
 1938 Rashevsky initiated studies for representing activation
and propagation in neural networks using differential
equations.
 1943 McCulloch and Pitts invented the first artificial model
for biological neurons.
 1943 Landahl, McCulloch, and Pitts noted that many
arithmetic and logical operations could be implemented
using McCulloch and Pitts neuron models.

14
History of neural networks
 1954 Gabor invented the learning filter using gradient
descent to obtain optimal weights that minimize the mean
squared error.
 1956 Taylor introduced an associative memory network
using Hebb's rule.
 1958 Rosenblatt invented a learning method for the
McCulloch and Pitts neuron model.
 1960 Widrow and Hoff introduced the Adaline as a simple
network trained by a gradient descent rule.
 1961 Rosenblatt proposed the backpropagation scheme for
training multilayer networks.
 1964 Taylor constructed a winner-take-all circuit.

15
History of neural networks
 1969 Minsky and Papert demonstrated the limits of simple
perceptions.
 Combinations of many neurons can be more powerful than
single neurons.
 1962 Dreyfus formulated learning rules to large neural
networks.
 Gradient descent is offently not successful in obtaining a
desired solution to a problem.
 Random, probabilistic, or stochastic methods have been
developed.

16
Content

 Introduction
 History of neural networks
 Biological neurons
 Artificial neuron models

17
Biological neurons
 A typical biological neuron is composed of a cell body, an
axon and dendrites.

 The dendrites surround the body of the neuron.

 The axon of a neuron forms synaptic connections with others.

18
Biological neurons
 The small gap between an end point and a dendrite is called
a synapse.
 The synapses decide that which information is
propagated.
 The number of synapses received by each neuron range
from 100 to 100,000.

19
Content

 Introduction
 History of neural networks
 Biological neurons
 Artificial neuron models

20
Artificial neuron models
 The artificial neuron was created inspired by the
biological neuron.
 Each part of the artifical neuron has an equivalent part in
the biological neuron.

21
Artificial neuron models
 Many different weighted inputs are summed.

 The neuron output is:

22
Artificial neuron models
Step function
 Step function is commonly used in single neuron model.

 Common values for a, b, c are

(a = 0, b = 1, c = 0) or (a = -1, b = 1, c = 0)
 The function is defined as:

23
Artificial neuron models
Ramp function
 Ramp function is commonly used in single neuron model.

 Common values for a, b, c, d are,

(a = 0, b = 1, c = 0, d = 1) or (a = -b, c = -1, d = 1)
 The function is defined as:

24
Artificial neuron models
Sigmoid function
 The most popular node functions used in neural nets are
sigmoid (S-shaped) functions.
 These functions are continuous and differentiable
everywhere.

 Common values for a, b, c are

(a = 0, b = 1, c = 0) or (a = -1, b = 1, c = 0)
 The function is defined as:
25
Artificial neuron models
Piecewise linear function
 Piecewise linear functions are combinations of various
linear functions.
 Piecewise linear functions are easier to compute than
nonlinear function.
 In the figure, the dashed line is the piecewise linear function,
and the straight line is the sigmoid function.

26
Artificial neuron models
Gaussian function
 Bell-shaped curves known as Gaussian or radial basis
functions.
 Gaussian node functions are used in Radial Basis
Networks.

27
Artificial neuron models
Other functions
 There are many different functions that are used as activation
functions.

28
Artificial neuron models
Other functions

29
Homework

 Prepare a report on the use of artificial neural networks in the

medicine (diagnosis and treatment).

30