Biological Neurons and Neural Networks, Artificial Neurons

Neural Computation : Lecture 2

© John A. Bullinaria, 2015

1. Organization of the Nervous System and Brain

2. Brains versus Computers: Some Numbers

3. Biological Neurons and Neural Networks

4. Rate Coding versus Spike Time Coding

5. The McCulloch-Pitts Neuron

6. Some Useful Notation: Vectors, Matrices, Functions

7. The McCulloch-Pitts Neuron Equation

L2-1
 The Nervous System
The human nervous system can be broken down into three stages that may be
represented in block diagram form as:

Neural
Stimulus Receptors Network/ Effectors Response
Brain

The receptors collect information from the environment – e.g. photons on the retina.

The effectors generate interactions with the environment – e.g. activate muscles.

The flow of information/activation is represented by arrows – feedforward and feedback.

Naturally, this module will be primarily concerned with how the neural network in the
middle works, but understanding its inputs and outputs is also important.

L2-2
 Levels of Brain Organization
The brain contains both large scale and small scale anatomical structures and different
functions take place at the higher and lower levels.

There is a hierarchy of interwoven levels of organization:

1. Molecules and Ions

2. Synapses
3. Neuronal microcircuits
4. Dendritic trees
5. Neurons
6. Local circuits
7. Inter-regional circuits
8. Central nervous system

The ANNs studied in this module are mostly approximations of levels 5 and 6.

L2-3
 Brains versus Computers : Some numbers

1. There are approximately 10 billion neurons in the human cortex, compared with
tens of thousands of processors in the most powerful parallel computers.

2. Lack of processing units can be compensated by speed. The typical operating

speeds of biological neurons is measured in milliseconds (10-3 s), while current
silicon chips can usually operate in nanoseconds (10-9 s).

3. Each biological neuron is connected to several thousands of other neurons, similar

to the connectivity in powerful parallel computers.

4. The human brain is extremely energy efficient, using approximately 10-16 joules
per operation per second, whereas the best computers today use around 10-6 joules
per operation per second.

5. Brains have been evolving for tens of millions of years, but computers have only
been evolving for tens of decades, though different mechanisms are involved.

L2-4
 Slice Through a Real Brain

http://library.med.utah.edu/WebPath/HISTHTML/NEURANAT/NEURANCA.html
L2-5
Structure of a Human Brain

L2-6
 Basic Components of Biological Neurons

1. The majority of neurons encode their activations or outputs as a series of brief

electrical pulses (i.e. spikes or action potentials).

2. The neuron’s cell body (soma) processes the incoming activations and converts
them into output activations.

3. The neuron’s nucleus contains the genetic material in the form of DNA. This
exists in most types of cells, not just neurons.

4. Dendrites are fibres which emanate from the cell body and provide the receptive
zones that receive activation from other neurons.

5. Axons are fibres acting as transmission lines that send activation to other neurons.

6. The junctions that allow signal transmission between the axons and dendrites are
called synapses. The process of transmission is by diffusion of chemicals called
neurotransmitters across the synaptic cleft.

L2-7
Schematic Diagram of a Biological Neuron

L2-8
 Neural Signal Processing

The key components of neural signal processing are:

1. Signals from connected neurons are collected by the dendrites.

2. The cells body (soma) sums the incoming signals (spatially and temporally).

3. When sufficient input is received (i.e., a threshold is exceeded), the neuron

generates an action potential or ‘spike’ (i.e., it ‘fires’).

4. That action potential is transmitted along the axon to other neurons, or to

structures outside the nervous systems (e.g., muscles).

5. If sufficient input is not received (i.e. the threshold is not exceeded), the inputs
quickly decay and no action potential is generated.

6. Timing is clearly important – input signals must arrive together, strong inputs will
generate more action potentials per unit time.

L2-9
 Neuron Action Potential

From: Principles of Neurocomputing for Science & Engineering, Ham & Kostanic, McGraw-Hill, 2001.

L2-10
 Excitatory Postsynaptic Potential

From: Principles of Neurocomputing for Science & Engineering, Ham & Kostanic, McGraw-Hill, 2001.

L2-11
 Rate Coding versus Spike Time Coding

In biological neural networks, the individual spike timings are often important. So
“spike time coding” is the most realistic representation for artificial neural networks.

However, averages of spike rates across time or populations of neurons carry a lot of
the useful information, and so “rate coding” is a useful approximation.

Spike coding is more powerful, but the computer models are much more complicated
and more difficult to train.

Rate coding blurs the information coded in individual neurons, but usually leads to
simpler models with differentiable outputs, which we will see later is important for
generating efficient learning algorithms.

Sigmoid shaped activation functions in the rate coding approach follow from the
cumulative effect of Gaussian distributed spikes.

L2-12
 The McCulloch-Pitts Neuron
A simple rate coding model of real neurons is also known as a Threshold Logic Unit :

in1

in2
out
∑

inn

1. A set of synapses (i.e. connections) brings in activations from other neurons.

2. A processing unit sums the inputs, and then applies a non-linear activation function
(which is also often called a threshold or transfer or squashing function).
3. An output line transmits the result to other neurons.

L2-13
 Some Useful Notation
We often need to deal with ordered sets of numbers, which we write as vectors, e.g.
x = (x1, x2, x3, …, xn) , y = (y1, y2, y3, …, ym)

The components xi can be added up to give a scalar (number), e.g.

n
s = x1 + x2 + x3 + … + xn = ∑ xi
i=1

Two vectors of the same length may be added to give another vector, e.g.
z = x + y = (x1 + y1, x2 + y2, …, xn + yn)

Two vectors of the same length may be multiplied to give a scalar, e.g.
n
p = x.y = x1y1 + x2 y2 + …+ xnyn = ∑ xi yi
i=1

To avoid any ambiguity or confusion, we will mostly be using the component notation
(i.e. explicit indices and summation signs) throughout this module.

L2-14
 The Power of the Notation : Matrices
We can use the same vector component notation to represent more complex things with
many dimensions/indices. For two indices we have matrices, e.g. an m × n matrix

 w11 w12 ... w1n 

 
 w21 w22 ... w1n 
w=
: : : 
 
 wm1 wm1 ... wmn 

Matrices of the same size can be added or subtracted component by component.

An m × n matrix a can be multiplied with an n × p matrix b to give an m × p matrix c.

This becomes straightforward if we write it in terms of components:
n
cik = ∑ aij bjk
j =1

An n component vector can be regarded as a 1 × n or n × 1 matrix.

L2-15
 Some Useful Functions
A function y = f(x) describes a relationship (input-output mapping) from x to y.

Example 1 The threshold or step function step(x) is defined as

1.2

1.0

0.8

1 if x≥0

sgn(x)
step( x)=  0.6

0 if x<0 0.4

0.2

0.0
-4 -2 0 2 4
x

Example 2 The logistic sigmoid function Sigmoid(x) is defined as

1.2

1.0

€ Sigmoid(x) =
1 Sigmoid(x)
0.8

0.6

1 + e− x 0.4

0.2

0.0
-8 -4 0 4 8
x

This is a smoothed (differentiable) form of the threshold function.

L2-16
 The McCulloch-Pitts Neuron Equation
Using the above notation, it is possible to write down a simple equation for the output
out of a McCulloch-Pitts neuron as a function of its n inputs ini :
n
out = step(∑ ini − θ )
i=1

where θ is the neuron’s activation threshold. We can easily see that:

n n
out = €
1 if ∑ ink ≥ θ out = 0 if ∑ ink < θ
k =1 k =1

Note that the McCulloch-Pitts neuron is an extremely simplified model of real

biological neurons. Some of its missing features include: non-binary inputs and
outputs, non-linear summation, smooth thresholding, stochasticity, and temporal
information processing.

Nevertheless, McCulloch-Pitts neurons are computationally very powerful. One can

show that networks of such neurons are capable of universal computation.

L2-17
 Overview and Reading

1. Biological neurons, consisting of a cell body, axons, dendrites and

synapses, are able to process and transmit neural activation.
2. The McCulloch-Pitts neuron model (Threshold Logic Unit) is a crude
rate-coding approximation to real neurons, that performs a simple
summation and thresholding function on activation levels.
3. Appropriate mathematical notation facilitates the specification and
programming of artificial neurons and networks of artificial neurons.

Reading

1. Haykin-1999: Sections 1.1, 1.2, 1.3

2. Ham & Kostanic: Sections 1.2, 1.3
3. Beale & Jackson: Sections 1.2, 3.1, 3.2
4. Gurney: Sections 2.1, 2.2.

L2-18