INFORMATION SCIENCES 73,273-2&t (1993) 273

Addition as Fuzzy Mutual Entropy

BART KOSKO
Department of Electrical Engineering-Systems, Signal and Image Processing Institute,
Universityof Southern California, Los Angeles, California 90089-2564

ABSTRACT

A sum of real numbers equals the mutual entropy of a fuzzy set and its complement
set. A “fuzzy” or multivalued set is a point in a unit hypercube. Fuzzy mutual (Kullback)
entropy arises from the logarithm of a unique measure of fuzziness. The proof uses the
logistic map, a diffeomorphism that maps extended real space onto the fuzzy cube
embedded in it. The logistic map equates the sum of a vector’s components with the
mutual entropy of two dual fuzzy sets. Diffeomap projection offers a new way to study
the fuzzy structure of real algorithms.

1. THE THEOREM: ADDITION AS FUZZY

MUTUAL ENTROPY

Any sum of real numbers x1,. . . , x, equals the fuzzy mutual entropy of
fuzzy set F in the unit hypercube [O,11’:

k xi =H(F,Fc) -H(FC/F) (1)

i=l

where F” is the fuzzy set complement of F in the unit hypercube I”. If

you add two numbers, F and F’ lie in the unit square 12. If you add three
numbers, they lie in 13, and so on up.
The proof of (1) maps the extended real space R” = [ - M, 4” diffeomor-
phically onto the embedded unit hypercube I”. The proof views the real
numbers x 1,. . . , x, as the components of the real vector x in R” and maps
x to a unique point or fuzzy set F in the fuzzy space I”.
The mutual entropy terms H(F/F’) and H(F’/F) stem from the
logarithm of the fuzziness of F. As discussed below this fuzziness depends
on how much F resembles its complement fuzzy set F’. In this sense we
OElsevier Science Publishing Co., Inc. 1993
655 Avenue of the Americas, New York, NY 10010 0020-0255/93/$6.00
274 B. KOSKO

can replace the two H terms in (1) with an entropy operator 3.? applied to
fuzzy set F:

iq=iqF). (2)
i-l

The operator X replaces each sum with the value of a map from fuzzy sets
to real numbers.
The infinity “corners” of -w and 03 in R” correspond to the 0 - 1
vertices in I”. The origin in R” corresponds to the midpoint of I”, the
unique fuzzy set F such that F = FC = F n F” =F U F’. The next three
sections review the needed fuzzy information theory and develop the new
measure of fuzzy mutual entropy. Section 5 proves (1).

2. FUZZY SETS AS POINTS IN HYPERCUBES: DEGREES

OF SUBSETHOOD

Multivalence or “fuzziness” holds in sets and between sets. Fuzziness in

a set defines elementhood, the degree a, to which element xi belongs to
set A: ai =Degree(x, 64).
A standard or bivalent or “nonfuzzy” set A contains elements all or
none. The membership degree ai is 1 or 0, present or absent, in or out.
A multivalent set A contains elements to some degree. So ui takes
values in the unit interval [O,11. Black [l] called this multivalence “vague-
ness” and introduced vague sets or vague lists. Zadeh [12] called these
vague or multivalued sets “fuzzy” sets and developed their algebra.
Fuzziness between sets defines subset/rood [7-lo], the degree S(A, B)
to which set A belongs to, or is a subset of, set B: S(A, B) = degree(A cB).
The sets A and B need not be fuzzy. If a fuzzy set A contains an element
xi to degree ui, then S({x,), A) = ui. So subsethood subsumes elementhood.
In the past the subsethood operator S has defined I bivalent operator in
both fuzzy and nonfuzzy set theory: S(A, B) = 0 or 1. The multivalued
subsethood operator can also assume the values 0 < S(A, B) < 1.
The subsethood operator arises from the unique IP-extension of the
Pythagorean theorem [8] in n dimensions:

llA-BI)P=(IA-B*IIP+IIB*-BIIP (3)
ADDITION AS FUZZY MUTUAL ENTROPY 275

for p > 1, n-vectors A, B, and B*,and with the norm

IMP = 2 IXjlP.
i=l
(4)

The usual Pythagorean theorem holds if p = 2. For any p there are 2”

vectors or sets B* that satisfy (3). Then bT =ui or bi. For fuzzy sets these
2” choices reflect the 2” choices of picking any vertex of the unit hyper-
cube I” as the origin or empty set. Once picked, bT = min(a,, bi). If A and
B are bit vectors or regular nonfuzzy subsets of finite space X= (x1,. . . , x,),
then this implies that B* equals A intersect B, and the same holds for
fuzzy subsets A and B of X:

B*=AnB. (5)

Suppose set or space X is finite with X= {x1,. . ., x,}. Then the 2”

nonfuzzy subsets of X map to the 2” bit vectors of length n. These map in
turn to the 2” corners of the unit hypercube I”. This equates a set with a
point in the Boolean n-lattice. We can also view fuzzy subsets of X as
n-vectors with components in [O,l]. Then each vector component aj of
fuzzy set A ={a,,..., a,)defines a fuzzy unit or fifit [7] and A defines a fit
vector. Fit value a, measures the degree to which element xi belongs to or
fits in set A. This identifies A with a point on or in the unit hypercube I”
[8]. Fuzzy sets fill in the Boolean n-cube to give the solid hypercube
I”. The midpoint of the unit cube is the fit vector F = (4,. . . , i) where each
element xi belongs to F as much as it belongs to its complement F".
The usual set operations apply to fit vectors as Zadeh [12] proposed for
fuzzy set functions: A n B = (min(u,, b,), . . . , min(u,, b,)), A U B =
~max(a,,b,),..., maxfu,,b,)), AC=(l-ur,...,l-a,). Suppose A=(+ z)
and B=($ 3).Then

AnB=(f +)

AUB=(+ b)

A"=(: $)

AuA"=($ f).
276 B. KOSKO

Note that A L4” # 0 and A UA” #X in this sample and for all fuzzy sets
A. Aristotle’s bivalent “law” of noncontradiction and excluded middle no
longer hold. They hold only to some degree. They hold 100% only for the
bit vectors at cube vertices. They hold 0% at the cube midpoint when
A =A”. For fit vectors between these extremes they hold only to some
degree. The next section shows how the overlap term A nAc and underlap
term A UA” give a unique measure of the fuzziness [5] or entropy of A.
If A and B are not fuzzy sets, then the 100% subsethood relation
A cl3 holds if and only if ai G bi for all i. It still holds if A and B are fuzzy
sets: S(A, B) = 1 iff ai <bi. Then all of B’s 100% subsets define a
hyperrectangle in I” with a long diagonal that runs from the origin to
point B. X4, B) = 1 iff A lies in or on this hyperrectangle, the fuzzy
power set of B, P’(2B>. S( A, B) < 1 iff A lies outside the hyperrectangle.
The closer A lies to the hyperrectangle, the larger the value S(A, B). The
minimum distance lies between A and B*, the 100% subset of B closest
to A in any 1P metric [8]. This distance gives the 1J’“orthogonal” projec-
tion of A onto F(2B) shown in Figure 1 and gives the term IIA -B*I(’ in
the general IP-Pythagorean theorem (3).

{x2) = (0.1) x = (1,l)

\
\

0 = (0,O) Xl
(Xl) = (1.0)
i i
Fig. 1. Pythagorean geometry of the subsethood theorem of fuzzy sets.
ADDITION AS FUZZY AUNT ENTROPY 277

The subsethood theorem follows from this orthogonal projection and

unifies multivalued set theory. To see this first let c(A) denote the count
or cardinality of A:

c(A) =u,+u,+ . . . “t-11, (6)

=[a,--Ol+la,-o/+ . . . i-la,-01 (7)
==Z’(ra,A). (8)

If A =(5 $1, then c(A) = z. The equalities (6)-G?) geometrize the

count c(A) as the I1 or fuzzy Humm@ distance between A and the origin
or empty set 0. The count extends the classical counting measure of
combinatorics to fuzzy sets &I: c(A) equals the counting measure of A on
nonfuzzy sets A. It gives the number of elements in A if A is finite and
gives ~0if A is infinite-if A maps one-to-one to one of its proper subsets.
The subsethood measure S(A, B) depends on the minimal distance
&A, B*). In the fuzzy Hamming metric this means S(A,B) = l-
Z’(A, B*)/f(A). Boundary conditions on the empty set [8], [lo] show that
f(A) =c(A). Since B* =AnB, I’(A, B*) equals the I’ difference [c(A)-
c(AnB)] shown in Figure 1. So sfA,B)=l-[c(A)-c(AnB)]/ctA).
This gives the subsethood theorem:

S( A, I?) = c(c;;;) . (9)

IfA=(~~)andB=f~~),thenS(A,B)=~/~=~andS(B,A)=~/~=
$. So B is more a subset of A than A is of B.
The derived ratio in (9) has the same form as the conditional probability
P(B/A). In general the event probability P(A) is the degree to which the
sample space X is a subset of its own subset or event A, P(A) = SCX, A).
This looks like the identity P(A) =P(A/X). The subsethood theorem (9)
also implies that the whole-in-the-part term S(X, A) gives the relative
frequency n,,/n if A denotes a bit vector with nA Is or successes and with
n -nA OSor failures: S(X, A) =c(A nX)/c(X>=c(A)/c(X> =n,/n.
The subsethood theorem (9) also implies S({xJ, A) = ai since the single-
ton set {xi) maps to the unit bit vector (0 . . . 0 10 . . . 0) with a I in the ith
slot and OS elsewhere and since A =(Ui,. . . , a,>. Then CC{XiI) = 1 and
&xi} nA) =ui. So S((X,}, A) =ui and subsethood formally subsumes
elementhood.
278 B. KOSKO

Maps between unit cubes define fuzzy systems; S: I” -+ IP. Fuzzy systems
associate output fuzzy sets with input fuzzy sets and so generalize if-then
rules. Fuzzy systems are uniformly dense in the space of continuous
functions [91: a fuzzy system can approximate any real continuous (or Bore1
measurable) function on a compact set to any degree of accuracy. The
fuzzy system contains fuzzy rules of the form IF X=A, THEN Y =B that
associate an output fuzzy set B with an input fuzzy set A. The rule defines
a fuzzy Cartesian product A x B or patch in the input-output state space
XX Y. A fuzzy system approximates a function by covering its graph with
patches and averaging patches that overlap. All the rules fire to some
degree as in a neural associative memory [lo]. The approximation theorem
shows that finite discretizations of A and B suffice for the covering. So
the patch or fuzzy Cartesian product A x B reduces to a fuzzy n-by-p
matrix M or relation or point in Z“J’. Then M defines the system mapping
M: I” + IP and the subsethood measure in (9) applies to M. In the same
product space each fuzzy system is a subset to some degree of all other
fuzzy systems. Then (11) below shows that each fuzzy system has a unique
numerical measure of fuzziness [5] or entropy.

3. FUZZINESS AND ENTROPY

How fuzzy is a fuzzy set? A nonfuzzy set lies at a vertex of cube I” and
has 0% fuzziness. The cube midpoint P equals its own opposite, P= PC,
and it alone has 100% fuzziness. In between it varies. The fuzziness of set
F grows as the distance falls between F and FC-as F and F” lie closer to
the midpoint P.
This cube geometry motivates the ratio measure of fuzziness, E(F) =
a/b, where a is the distance I’(F, Fnear)from F to the nearest vertex Fnear
and b is the distance 1’(F, F,,) from F to the farthest vertex F,,,. A long
diagonal connects Fnearto F,,. The fuzzy entropy theorem [7] reduces this
ratio to a ratio of counts:

c(FnFC)
E(F) = (10)
c(FuF’) ’

If F = (3 i), then E(F) = &/g = A. Figure 2 shows the fuzzy entropy

theorem in the unit square.
The fuzzy entropy theorem (10) shows that the fuzziness of fuzzy set F
depends on how much its overlap F n FC and underlap F uF” break
Aristotle’s laws of noncontradiction and excluded middle. Since the under-
lap F U F” always fully contains the overlap F n F”, S(F n F’, F U FC) = 1,
ADDITION AS FUZZY MUTUAL ENTROPY 279

{X21 = (0.1) x = (1,l)

0 = (0,O) {Xl) =Cl.O)

5 t
Fig. 2, Geometry of the fuzzy entropy theorem.

we might expect E(F) to involve subsethood in the converse direction

S(FUF”, FnFc) when the part partially contains the whole. In fact (9)
and (10) reduce fuzziness to subsethood:

E(F) =S(FUF’,Fi-@). (11)

The probabilistic entropy H(P) [3,5] holds for fit vectors on the simplex
in Z”. Then

H(P)= &og;_ (12)

i=l I

and c(P)=p, + . . . +p, = 1. The fuzziness measure E(P) differs from

H(P) for the same P. If no pi > i, then E(P) = l/(n - l>, and so E(P)
falls to zero as the cube dimension n grows to infinity. The uniform set
(’n,. . ., i) belongs to this set of P vectors along with un~untably many
280 B. KOSKO

others. If some pj > t, then E(P) < l/(n - 1). So the uniform set maxi-
mizes E(P) but does not uniquely maximizes it. So E differs from H.
Now consider how E resembles H. Consider the probability element pi
and the motivation for the logarithm measure (12) as the average infor-
mation or entropy of a message or event: “information is inversely related
to the probability of occurrence” [3]. The more improbable the event, the
more informative the event if the event occurs. So information increases
with l/p,. The same intuition holds for monotone-increasing transforms of
l/p,. This includes the logarithmic transform log l/p, and only the
logarithmic transform in the additive case. The weighted average over the
system or alphabet gives the entropy as the expected information (12).
In the one-fit case E(F) reduces to f/(1 -f> if f< 3 and to (1 -f>/f if
f> 3. This ratio grows to 1 as f moves to the midpoint i and falls to 0 as f
moves to 0 or 1. The more vague or fuzzy the event, the more informative
the event if it occurs. The operator E is subadditive on fuzzy sets since in a
fuzzy space all events connect to one another to some degree. Integration
also shows that f/l -f and 1 -f/f define a continuous probability density
on [O,11 if normalized by In 4 - 1. So far we have only reviewed fuzzy
entropy. We now extend it to mutual entropy to set up the proof of the
main theorem.

4. FUZZY MUTUAL ENTROPY

Fuzzy mutual entropy arises from a natural question: Why not take the
logarithm of the unit fuzziness f/<l -f)? Any monotone transform will
preserve its shape. So why not follow the probability example and use a
logarithm? Then we can weight the log terms with the fit values and get a
more proper measure of the entropy of a fuzzy set. The idea is to replace
the intuition chain

(13)

with the new fuzzy chain

fi+ --+lIl~+~fil~~~
fi (14)
l -fi
i
ADDITION AS FUZZY MUTUAL ENTROPY 281

The new fuzzy entropy term in (14) uses the natural logarithm to simplify
the proof of the main theorem. The sum term defines a fuzzy mutual
entropy.
For probability vectors P and Q in the I” simplex, define the mutual
entropy H(P/Q) of P given Q [ll] as

H(P/Q) = ~Piln~. (15)

i
I

The mutual entropy measures distance in the simplex in the rough sense
that H(P/Q> = 0 if P = Q, and H(P/Q) > 0 if P # Q. This follows from
the Gibbs inequality [31. Some stochastic learning automata and neural
networks [4] minimize H(P/Q) as the learning system’s distribution P
tries to estimate the distribution Q of the sampled environment. In the
cube I”, the fuzzy mutual entropy term in (14) is the usual mutual entropy
H(F/FC) defined on fit vectors.
The sum of the fuzzy information units In (A./l -f,> splits into the
mutual entropies of fuzzy sets F and F’:
LEMMA:

c In-&- =H(F/F”) -H(F”/F). (16)

i
1 fi

Proof. Since fi + (1 -fi> = 1,

(17)

fi
= )+A In] - f: (1 -.fiNfi l -fi
(18)

=H(F/F“) -H(F’/F). Q.E.D. (19)

The fuzziness measure in (10) shows that E(F)=E(F”). This reflects

the 2”-fold symmetry of the fuzzy cube I”. But the mutual entropy
operator is asymmetric. H(F/F”) = H(F’/F) if F =Fc-if F and F’ lie
282 B. KOSKO

at the cube midpoint. The mutual entropy summands grow to infinity or

zero as F and F" move to cube vertices.

5. THE PROOF: DIFFEOMAPS BETWEEN REAL SPACES AND

FUZZY CUBES

Fuzzy cubes map smoothly onto extended real spaces of the same
dimension and vice versa. The 2” infinite limits of extended real space
[ - w, ~1” map to the 2” binary corners of the fuzzy cube I”. The real origin
0 maps to the cube midpoint. Each real point x maps to a unique fuzzy set
F as Figure 3 shows.
A diffeomorphism f: R" -+I"is a one-to-one and onto differentiable
map f with a differentiable inverse f-‘. Different diffeomaps reveal
different fuzzy structure of operations in real space. The theorem (1)
follows from one of the simplest diffeomaps, the logistic map used in
neural models [2, 101 to convert an unbounded real input xi to a bounded
signal or fit value fi:

fi= --L-
1 +e-Xl * (20)

In extended real space R” the logistic map applies to each term of vector
x=(x 1,. . . , ,x,,). Note that fi = 0 iff xi = - ~0,fi = 1 iff xi = ~0,and fi = 3 iff
xi =O. Each real x picks out unique dual fuzzy sets F and F" in fuzzy
space.
The proof of (1) follows from the lemma (16) and from the inverse of
the logistic map (20):

fi
xi=f’(fi) =ln-
1 -fi. (21)

Fig. 3. Diffeomap from real space to fuzzy space.

ADDITION AS FUZZY MUTUAL ENTROPY 283

So each real number is a unit of fuzzy information (141, the logarithm of

the scalar measure of fuzziness (10). Sum over all n vector components xi
and apply the lemma (16) to prove (1) and (2):

Cxi =H( F/F”) -H( FC/F) (22)

=qF) (23)
in operator notation. Q.E.D.
The logistic map (20) also allows a direct proof for each term xi:

(24)

= &ln eXl- &lne-“1 (25)

- &hi&(1 +e-Xl) (26)

(27)

since fj = l/l + e-‘i and 1 -f, =e-‘l/l + e-X’.

6. CONCLUSIONS

Addition or counting is the most basic operation in mathematics. It

equals a basic operation in fuzzy space, the entropy map %’ that assigns a
real number to each fuzzy set. This equality may seem odd since we have
just begun to see the unit hypercube as a fuzzy space with its own set
algebra and geometry. Diffeomap projection-or in some cases the weaker
homeomorphic projection-can help show the fuzzy structure of real
operations and algorithms. Future research may classify diffeomaps by how
they preserve basic operations such as addition or how they carve the fuzzy
cube into entropy regions or balls.
The fuzzy cube may also extend operations and algorithms in informa-
tion theory. The fuzzy approximation theorem [9] converts continuous or
284 B. KOSKO

measurable systems into a finite number of fuzzy patches or points in large

fuzzy cubes. The cube contains both the probability simplex that describes
channel transmissions and the Boolean cube that describes all binary
messages of a fixed length. An algorithm can dig through the cube from
binary vertex to distant binary vertex rather than hop as a gray code from
vertex to local vertex in cubes of high dimension. We can also view
messages as balls of entropy or fuzziness in a cube. Ball diameter falls as
the ball center moves from the vague midpoint to the clear comers.
Diffeomaps can map real messages or systems into signal or noise balls
that overlap in the fuzzy cube.

REFERENCES

1. M. Black, Vagueness: An exercise in logical analysis, Philosophy ofScience 4:427-455

(19371.
2. S. Grossberg, Studies of Mind and Brain, Reidel, 1982.
3. R. W. Hamming, Coding and Information Theory, 2nd edition, Prentice Hall, 1986.
4. G. E. Hinton and T. J. Sejnowski, Learning and relearning in Boltzmann machines,
in Parallel Distributed Processing, Vol. I, D. E. Rumelhart and J. L. McClelland, Eds.,
M.I.T. Press, 1986, pp. 282-317.
5. G. J. Klir and T. A. Folger, Fuzzy Sets, Uncertainty, and Information, Prentice Hall,
1988.
6. B. Kosko, Counting with fuzzy sets, IEEE Trans. Pattern Anal. Machine Intell.
PAMI-8:556-557 (July 1986).
7. B. Kosko, Fuzzy entropy and conditioning, Inform. Sci. 40:165-174 (1986).
8. B. Kosko, Fuzziness vs. probability, ht. J. General Syst. 17(2):211-240 (1990).
9. B. Kosko, Fuzzy systems as universal approximators, in Proc. IEEE Znt. Conf: on
Fuzzy Syst. 1992 (Fuz.z’92), Mar. 1992, pp. 1153-1162.
10. B. Kosko, Neural Networks and Fuzzy Systems, Prentice Hall, 1992.
11. S. Kullback, Information Theoy and Statistics, Wiley, 1959.
12. L. A. Zadeh, Fuzzy sets, Inform. Contr. 8:338-353 (1965).

Received S September 1992; revised 19 November 1992