Hybrid intelligent systems

Neuro-fuzzy systems
 Hybrid Systems
 A hybrid intelligent system is one that combines at least two
intelligent technologies.
 For example, combining a neural network with a fuzzy system
results in a hybrid neuro-fuzzy system.

 The combination of:

 probabilistic reasoning,
 fuzzy logic,
 neural networks and
 evolutionary computation forms the core of soft computing,

 Soft Computing is an emerging approach to building hybrid

intelligent systems capable of reasoning and learning in an
uncertain and imprecise environment.
 Neuro-fuzzy systems
 Fuzzy logic and neural networks are natural
complementary tools in building intelligent systems.

 While neural networks are low-level computational

structures that perform well when dealing with raw data,
fuzzy logic deals with reasoning on a higher level, using
linguistic information acquired from domain experts.

 However, fuzzy systems lack the ability to learn and

cannot adjust themselves to a new environment.
 On the other hand, although neural networks can
learn, they are opaque to the user.
Synergy of Neural and Fuzzy

 Integrated neuro-fuzzy systems can combine the

parallel computation and learning abilities of
neural networks with the human-like knowledge
representation and explanation abilities of fuzzy
systems.

 As a result, neural networks become more

transparent, while fuzzy systems become capable
of learning.
 A neuro-fuzzy system is a neural network which is
functionally equivalent to a fuzzy inference model.

 Neuro-Fuzzy System can be trained to develop IF-

THEN fuzzy rules and determine membership functions
for input and output variables of the system.
 The structure of a neuro-fuzzy system is similar to
a multi-layer neural network.

 In general, a neuro-fuzzy system has:

 input and output layers,
 and three hidden layers
 that represent membership functions and
fuzzy rules.
 Neuro-fuzzy system
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5

A1 A1 R1
x1 R1

x1 x1
A2  R2
x1 A2 R2
wR3 C1
C1
A3 A3 R3 R3 wR6
y
wR1 
B1 B1 R4 R4 wR2
x2 C2
wR4
C2
R5
x2 x2 B2 wR5
B2 R5
x2
R6
B3
B3 R6
Each layer in the neuro-fuzzy system is associated
with a particular step in the fuzzy inference process.
Layer 1 is the input layer. Each neuron in this layer
transmits external crisp signals directly to the next
layer. That is,
(1) (1)
yi  xi
Layer 2 is the fuzzification layer.
• Neurons in this layer represent fuzzy sets used in the
antecedents of fuzzy rules.
• A fuzzification neuron receives a crisp input and
determines the degree to which this input belongs to
the neuron’s fuzzy set.
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5

A1 A1 R1
x1 R1

x1 x1
A2  R2
x1 A2 R2
wR3 C1
C1
A3 A3 R3 R3 wR6
y
wR1 
B1 B1 R4 R4 wR2
x2 C2
wR4
C2
R5
x2 x2 B2 wR5
B2 R5
x2
R6
B3
B3 R6

Layer 1 is the input layer. Each neuron in this layer transmits

external crisp signals directly to the next layer. That is,
(1) (1)
yi  xi
Layer 2 is the fuzzification layer.
• Neurons in this layer represent fuzzy sets used in the antecedents of
fuzzy rules.
• A fuzzification neuron receives a crisp input and determines the
degree to which this input belongs to the neuron’s fuzzy set.
1. The activation function of a membership neuron is set
to the function that specifies the neuron’s fuzzy set.
2. We use triangular sets, and therefore, the activation
functions for the neurons in Layer 2 are set to the
triangular membership functions.
3. A triangular membership function can be specified by
two parameters {a, b} as follows:
 ( 2) b
 0, if xi  a 
2

 2 xi( 2)  a
 b b
yi( 2)  1  ( 2)
, if a   xi  a 
 b 2 2
 ( 2) b
 0, if xi  a 
2

 Triangular activation
functions
 
1 1
a = 4, b =6 a = 4.5, b =6 a = 4, b =6
0.8 0.8
a = 4, b =4
0.6 0.6

0.4 0.4

0.2 0.2

0
0 1 2 3 4 5 6 7 8
X 0
0 1 2 3 4 5 6 7 8
X

(a) Effect of parameter a. (b) Effect of parameter b.

Layer 1 Layer 2 Layer 3 Layer 4 Layer 5

A1 A1 R1
x1 R1

x1 x1
A2  R2
x1 A2 R2
wR3 C1
C1
A3 A3 R3 R3 wR6
y
wR1 
B1 B1 R4 R4 wR2
x2 C2
wR4
C2
R5
x2 x2 B2 wR5
B2 R5
x2
R6
B3
B3 R6

Layer 3 is the fuzzy rule layer.

1. Each neuron in this layer corresponds to a single fuzzy rule.

2. A fuzzy rule neuron receives inputs from the fuzzification neurons that represent
fuzzy sets in the rule antecedents.

3. For instance, neuron R1, which corresponds to Rule 1, receives inputs from neurons
A1 and B1.
Layer 3 is the fuzzy rule layer.

1. Each neuron in this layer corresponds to a single

fuzzy rule.

2. A fuzzy rule neuron receives inputs from the

fuzzification neurons that represent fuzzy sets in
the rule antecedents.

3. For instance, neuron R1, which corresponds to

Rule 1, receives inputs from neurons A1 and B1.
 In a neuro-fuzzy system, intersection can be
implemented by the product operator.

Thus, the output of neuron i in Layer 3 is obtained

as:

1   A1   B1   R1
(3)
yi(3)  x1(i3)  x2(3i )    xk(3i ) yR
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5

A1 A1 R1
x1 R1

x1 x1
A2  R2
x1 A2 R2
wR3 C1
C1
A3 A3 R3 R3 wR6
y
wR1 
B1 B1 R4 R4 wR2
x2 C2
wR4
C2
R5
x2 x2 B2 wR5
B2 R5
x2
R6
B3
B3 R6

Layer 4 is the output membership layer. Neurons in this layer represent fuzzy
sets used in the consequent of fuzzy rules.
An output membership neuron combines all its inputs by using the fuzzy
operation union.
This operation can be implemented by the probabilistic OR. That is,
( 4)
yi( 4)  x1 1   R3   R6  C1
( 4) ( 4) ( 4)
i  x2 i    xli yC

The value of C1 represents the integrated firing strength of fuzzy rule
neurons R3 and R6.
Layer 4 is the output membership layer. Neurons
in this layer represent fuzzy sets used in the
consequent of fuzzy rules.
An output membership neuron combines all its
inputs by using the fuzzy operation union.
This operation can be implemented by the
probabilistic OR. That is,
(4)
yi( 4)  x1(i4)  x2( 4i )    xli( 4) yC1   R3   R6  C1

The value of C1 represents the integrated firing

strength of fuzzy rule neurons R3 and R6.
Layer 1 Layer 2 Layer 3 Layer 4 Layer 5

A1 A1 R1
x1 R1

x1 x1
A2  R2
x1 A2 R2
wR3 C1
C1
A3 A3 R3 R3 wR6
y
wR1 
B1 B1 R4 R4 wR2
x2 C2
wR4
C2
R5
x2 x2 B2 wR5
B2 R5
x2
R6
B3
B3 R6

Layer 5 is the defuzzification layer. Each neuron in this layer represents a single
output of the neuro-fuzzy system. It takes the output fuzzy sets clipped by the
respective integrated firing strengths and combines them into a single fuzzy
set.

Neuro-fuzzy systems can apply standard defuzzification methods, including

the centroid technique.

We will use the sum-product composition method.

Layer 5 is the defuzzification layer. Each neuron
in this layer represents a single output of the
neuro-fuzzy system. It takes the output fuzzy sets
clipped by the respective integrated firing
strengths and combines them into a single fuzzy
set.
Neuro-fuzzy systems can apply standard
defuzzification methods, including the centroid
technique.
We will use the sum-product composition
method.
 The sum-product composition calculates the crisp
output as the weighted average of the centroids of
all output membership functions.

For example, the weighted average of the centroids of

the clipped fuzzy sets C1 and C2 is calculated as,

C1  aC1  bC1  C 2  aC 2  bC 2

y
C1  bC1  C 2  bC 2
How does a neuro-fuzzy system learn?
A neuro-fuzzy system is essentially a multi-layer
neural network,

and thus it can apply standard learning algorithms

developed for neural networks,

including the back-propagation algorithm.

 When a training input-output example is presented to the system, the
back-propagation algorithm computes the system output and compares it
with the desired output of the training example.

 The error is propagated backwards through the network from the

output layer to the input layer.

 The neuron activation functions are modified as the error is

propagated.

 To determine the necessary modifications, the back-propagation

algorithm differentiates the activation functions of the neurons.

 Let us demonstrate how a neuro-fuzzy system works on a simple example.

 Training patterns

0
1

0
0
 The data set is used for training the five-rule neuro-
fuzzy system shown below.
Five-rule neuro-fuzzy system
S 1
1 wR1
x2 0.99 wR5
0.8
L 2 0
S y 0.6
wR4

Weight
wR3
3 0.72 
L 0.4
0.61
S 4
x2 0.2 wR2
0.79
L 5 0
0 10 20 30 40 50
Epoch

(a) Five-rule system. (b) Training for 50 epochs.

Good and Bad Rules from Experts in systems

 Suppose that fuzzy IF-THEN rules incorporated into the

system structure are supplied by a domain expert.
 Prior or existing knowledge can dramatically expedite
the system training.

 Besides, if the quality of training data is poor, the expert

knowledge may be the only way to come to a solution at all.
 However, experts do occasionally make mistakes, and
thus some rules used in a neuro-fuzzy system may be
false or redundant.
 Therefore, a neuro-fuzzy system should also be capable
of identifying bad rules.
 Example: A neuro-fuzzy system for XOR

 Given input and output linguistic values, a neuro-fuzzy system

can automatically generate a complete set of fuzzy IF-THEN
rules.

 Let us create the system for the XOR example.

 This system consists of 22  2 = 8 rules.
 Because expert knowledge is not embodied in the system
this time, we set all initial weights between Layer 3 and
Layer 4 to 0.5.

 After training we can eliminate all rules whose certainty

factors are less than some sufficiently small number, say 0.1.
 As a result, we obtain the same set of four fuzzy IF-THEN
rules that represents the XOR operation.
 Eight-rule neuro-fuzzy
system for XOR
0.8
S 1 0 wR2 wR8
x1 2
0.7
0.78
3
0.6 wR3 wR5
L 0.69 S
0.5
4 y
0
0.62
 0.4
5
0.3 wR6 & wR7
S 6 0 L
0 0.2
x2 wR1
7 0.1
0.80 wR4
L 8 0
0 10 20 30 40 50
Epoch

(a) Eight-rule system. (b) Training for 50 epochs.

 Neuro-fuzzy systems: summary
 The combination of fuzzy logic and neural networks constitutes a
powerful means for designing intelligent systems.

 Domain knowledge can be put into a neuro-fuzzy system by

human experts in the form of linguistic variables and fuzzy rules.

 When a representative set of examples is available, a neuro-fuzzy

system can automatically transform it into a robust set of fuzzy
IF-THEN rules, and thereby reduce our dependence on expert
knowledge when building intelligent systems.