n = -5:0.

1:5;
1
a = hardlims(n);
0.8 plot(n,a)
n = -5:0.1:5;
0.6
a = hardlim(n);
1
plot(n,a) 0.4

0.2 0.5

0
-5 0 5 0

-0.5

• The hard-limiting threshold function -1

-5 0 5
– Corresponds to the biological paradigm
• either fires or not

 1, net  0
f (net ) sgn( net )  bipolar binary
  1, net  0

 1, net  0
f (net ) sgn( net )  unipolar binary
0, net  0
 Activation functions of a neuron
Step function Sign function Sigmoid function Linear function

Y Y Y Y
+1 +1 +1 +1

0 X 0 X 0 X 0 X
-1 -1 -1 -1

1, if X 0  1, if X 0 1
Y step  Y sign  Y sigmoid  Y linear  X
0, if X  0  1, if X  0 1  e X

2
f ( net )   1 bipolar continuous
1  e (  net )
 1, net  0
f ( net )  sgn( net )  bipolar binary
  1, net  0
1
f ( net )  unipolar continuous
1  e (  net )
 1, net  0
f ( net )  sgn( net )  unipolar binary
0 , net  0
 1 a=logsig(n) = 1 / (1 + exp(-n))
0.9

0.8

0.7 n = -5:0.1:5;
0.6

0.5
a = logsig(n);
0.4 plot(n,a)
0.3

0.2

0.1

0
-5 -4 -3 -2 -1 0 1 2 3 4 5

0.9

0.8

a = tansig(n) = 2/(1+exp(-2*n))-1 0.7

0.6

0.5

n = -5:0.1:5; 0.4

a = tansig(n); 0.3

0.2

plot(n,a) 0.1

0
-5 -4 -3 -2 -1 0 1 2 3 4 5
Homework Problems
Design a single output two layer network which classifies the shaded
region in Fig. from the other region.

(1,3)

(3, 2)

(1,1)
 The equations of the decision boundaries are

h1
x1  1 0
h2
0.5 x1  x 2  0.5 0
 0.5 x1  x 2  3.5 0 h3

1
1

So the hidden layer weights are:

For the first neuron, weights are 1 0 -1
For the second neuron, weights are 0.5 -1 0.5
For the third neuron, weights are -0.5 -1 0.5 3.5

Let the outputs of the hidden neurons are h1, h2 and h3. h1=1 means it is ‘on’ denoted by
h1>0. Similarly h1=0 by off state by h1<0. Now take the part numbered 1. here h1=0,
h2=0 and h3=0. Similarly we can write for parts also (see the table). We are interested in
making a neural network for which parts 1, 3 and 5 will produce output (o) 1 and other
parts 0.
 1
(0,4)
*

h1>0

(2,3)
*
6
2
7 h2>0
(0,2) * * *
(2,2) (4,2) 5

4 h3>0

* *
(0,0) (2,0)
   0
w3    0
h1 h2 h3 o
1 0 0 0 1 w 2  w3    0
w1  w 2  w 3    0
2 0 0 1 0
3 0 1 1 1 w1  w 2    0
4 1 1 1 0 w1    0
5 1 1 0 1
6 1 0 0 0
w1  w 3    0
7 1 0 1 0

From the first equation we know that  is negative. From the second equation, w 3 is
more negative than  and so on…I took  =-1, w3=-2, w2=3 and w1=-3.

p=[2;0];
net = newff([0 10;0 10],[3 1],{'hardlim' 'hardlim'});
net.b{1}=[-1;.5;3.5];
net.b{2}=[1];
net.lw{2,1}=[-3 3 -2];
net.iw{1,1}=[1 0;.5 -1;-.5 -1];
y=sim(net,p)
ERROR SURFACE
X= [2 3; 12 7; -3 5];
y = [5 19 2];
w1 = 0:0.1:2;
w2 = 0:0.1:2;
for p1 = 1:length(w1)
for p2 = 1:length(w2)
err(p1,p2) = 0;
for n = 1:3
% compute network output for example n
ynet = w1(p1)*X(n,1) + w2(p2)*X(n,2);
% update total error
err(p1,p2) = err(p1,p2) + (y(n) - ynet)^2;
end;
end;
end;
% plot error function
surf(w1,w2,err);
 Learning by Error Minimization
We like to minimize the squared error (which is a
function of the weights), for each training
pair/pattern:

Square makes error positive and penalizes large

errors
1/2 just makes some of the maths easier
The total error will be the sum of errors across all
patterns
Need to change the weights in order to minimize
the error
– Use principle of gradient descent - Calculate
derivative (gradient) of the Error with respect to
the weights, and then change the weights by a
small increment in the opposite direction to the
The gradient Descent Optimization

Direction of steepest
descent
 1 1 t
E  (d i  oi )  (d i  f ( wi x)) 2
2

2 2
t
E  (d i  oi ) f ' ( wi x) x
The components of the gradient vector are
E t
 (d i  oi ) f ' ( wi x) x j for j 1,2,...., n
wij
w i   E
w i  [ d i  oi ] f ' ( net i ) x
w  r x
t t
r [d i  f ( wi x)] f ' ( wi x)
w  r x

w2 w1  [d i  oi ] f ' (neti ) x d=t

 Example 1.(same as the binary one)

X1=[1 –2 0 –1]1, X2=[0 1.5 -.5 -1]1, X3=[-1 1 .5 -1]1

d1 = -1 d 2 = -1 d3 = 1
2
w1=[1 -1 0 .5]; f (net )  (  net )
1
1 e
Net1=net1=[1 -1 0 .5]*[1 -2 0 -1]'=2.5
2e (  net )
f ' (net ) 
w1 [1  e (  net ) ]2
w2 2e (  net ) 1 2
w3 o (  net ) 2
 (1  o )
[1  e ] 2
w4 net1 2.5 O1 0.848
f ' ( net 1 ) 0.140
Complete this problem
w 2  [ d i  oi ] f ' ( net 1 ) x1  w1
for one epoch

[0.974  .948 0 0.526]'

net 2  1.948