Self-Organizing Maps

“Kohonen Nets”
Feature Maps
(a form of competitive learning
that also requires cooperation)
Teuovo Kohonen

Dr. Eng., Professor of the Academy of Finland;

Head, Neural Networks Research Centre,
Helsinki University of Technology, Finland
 Several Different Kohonen Nets

• Linear associative memory (discussed

previously; discovered at same time as those
by James Anderson)

• Self-Organizing Maps (SOMs)

• Similar to LVQ (which Kohonen also discovered)
• Fundamentally unsupervised
• Supervision can be added
 What is a “Map”?

A map is a 2-layer network in which

certain topological relationships, such
as proximity of vectors in the first layer,

emerge in the form of spatial proximity

among the neurons in the second layer.
 Example of a Map

• Consider a 2-dimensional visual field, which

is the first layer of neurons.

• Stimuli occur at various points in the field,

each point corresponding to a specific
neuron.

• If two stimuli are in close proximity in the first

layer, then the neurons that are activated in
the second layer are in similar close
proximity.
 Two Map Models
(Haykin, ch. 9)

weighted
signal flow connections

signal flow
weighted
connections
 Uses of Maps

• The dimensions of the stimulus vectors in the

input layer can be higher dimension than
the dimension of the second layer geometry.

• The attendant dimension reduction in the

second layer allows us to visualize proximities
in data that might have not been apparent in
the higher dimensional space.
7-dimensional animal space example
13-dimensional animal space
Animal Similarity in 2 dimensions
 The Map
7-dimensional vector “cloud” 7-dimensional visualization
of similarities
 Maps in Neurobiology

• Related neural functions map onto

identifiable regions of the brain
• retinotopic map: vision, superior colliculus

• phonotopic map: hearing, auditory cortex

• somatotopic map: touch, somatosensory

cortex
 Maps in the Human Brain
Parietal Lobe
somatosensory
Frontal Lobe perception ntegration
thinking, planning, and of visual &
central executive functions; somatospatial information
motor execution

Occipital Lobe
visual
perception and
processing
Temporal Lobe
language function and
auditory perception involved
in long term memory and emotion
 Desired Properties of Maps
• Approximation of input space: generally a
many-one mapping of input space into weight
space

• Topology-preserving: points close together

in input vector space should map to points
close together in weight space.

• Density-preserving: regions of similar

density should map to regions of proportional
density.
 Somatotopic Map Illustrations:
The “Homunculus”

Relationship between body surfaces

and the regions of the brain that control them

(somewhat different from the original “little person inside” meaning).

Homunculus Location & Real Estate
Use-Dependence in Homunculi

http://cogsci.bme.hu/~ikovacs/SandP/prepI_4_2_files/fig6.jpg
 Maps Can Reorganize through Training
Reorganization induced by training a monkey
in a behavioral task engaging a small skin locus
(Recanzone, Merzenich et al.).
The cortical representation of the trained
skin portion (black circle) showed a dramatic
increase; the number of RFs increased;
RF overlap increased. Acuity also improved.

http://cogsci.bme.hu/~ikovacs/SandP/prepI_4_2_files/fig7.jpg
 Project: Map your own homunculus density
http://www.woodrow.org/teachers/biology/institutes/1991/homunculus.html

Idea: Neural density ∝ (discriminatory threshold)-1

sample (truncated):
 Derivation of SOM requires
Cooperation as well as Competition

• Each neuron in input field has an on-center, off-

surround type of behavior.

• When a stimulus is presented, neurons in the field

“compete” by mutual lateral inhibition.

• Cooperative aspect: The winner and neurons in

the neighborhood have their weights strengthened
in the direction (in weight space) of the stimulus.

• Neighboring neurons tend to learn to respond to sets

of similar stimuli.
Hubel and Wiesel Experiments
(1958)
 Hubel and Wiesel Findings (1958)

• Simple cells in the visual cortex respond to

specific angular orientations of light bars.

• Complex cells respond to specific orientation

and direction of motion.

• End-stopped cells respond to specific

orientation, motion, and length.
 Complex Cell Response
(orientation and motion)

Hubel’s book on-line:

http://neuro.med.harvard.edu/site/dh/bcontex.htm
Selectivity of Individual Cells
Receptor Network Modeling in
Kohonen’s Original Paper (1982)
 Self-Organizing Map Building
Competitive: select a winning neuron (e.g. by lateral inhibition)
Cooperative: update weight vectors in a neighborhood of the winning neuron.
Kohonen Rule: iw (q ) = iw ( q – 1 ) + ! ( p ( q ) – iw( q – 1 ))
i " Ni! ( d )
iw (q ) = ( 1 – ! )iw ( q – 1 ) + !p ( q )

N i( d ) = { j , d i, j ! d }

N 13( 1 ) = { 8 , 12, 13, 14, 18} Two possible

neighborhoods
N 13( 2 ) = { 3, 7, 8, 9, 11, 12, 13, 14, 15, 17, 18, 19, 23 }
of neuron 13,
the first
narrower

Optional
post-normalization
step not shown
 Annealing Aspect

• As training progresses:
• the learning rate is gradually reduced
• the size of the neighborhood is gradually
reduced

• Neighborhood can be based on

• Euclidean proximity, or
• Superimposed structure
 Kohonen Learning Algorithm

• Repeatedly present input vectors:

• At each input vector p presentation do the
following:
• Find the winning node k, whose weight vector
is closest to p.
• Train node k and all nodes in the current
neighborhood of k.
• Decrease the learning rate slightly.
• After every M cycles, decrease the size of the
neighborhood.
 “Superimposed Dimensionality”
of SOM

• In a general SOM, an n-dimensional grid, can

be overlaid with the neurons as nodes.

• The edges connecting the neurons (rather

than using Euclidean distance) constrain the
neighborhood for updating.

• Only nodes within a certain diameter on the

grid are in the neighborhood.
Uses of “Imposed Dimensionality” in
Data Analysis

• Data points may have an unknown, but often large,

underlying dimensionality, e.g. the underlying
phenomenon or process that has several dimensions
of attributes (such as color dimensions, various size
or rate dimensions, etc.)

• By training a network with an imposed grid with a

given dimensionality, the relationships among the
data may become visualizable in that number of
dimensions, especially if the imposed number of
dimensions is small.
Uses of “Imposed Dimensionality” in
Data Analysis

• In other words, training the map “organizes”

the data according to similarity in each
dimension.
 Demo Applets

• /cs/cs152/kohonen/ demo1, demo2, demo3

• Legend:
• black = input data point (random over a 2-D
region; 2-dimensional data)
• red = neuron in winner neighborhood; learns by
Kohonen rule
• blue = other neuron
• Learning rate and neighborhood both
decrease with time; demo speeds up over
time.
 Competition Code (Java)
used in Demo
// compete finds the neuron with weights closest to the input.
// It sets winpoint to the indices of that neuron.

public void compete(Input input)

{
// initialize min with an distance to an arbitrary neuron
winpoint = 0;
double min = neuron[winpoint].distance(input);

// find the min distance among all neurons

for( int point = 0; point < points; point++ )

{
double dist = neuron[point].distance(input);
if( min > dist )
{
min = dist; // update the min distance
winpoint = point;
}
}
}
 Competition Code
// learn updates all neurons within current neighborhood
// by applying the Kohonen learning rule against the current
// input.

public void learn(Input input, double learningRate)

{
for( int point = 0; point < points; point++ )
{
if( inWinnersNeighborhood(point) )
{
neuron[point].learn(input, learningRate);
}
}
}
 Optimized Computation for
Fixed Datasets: Batch Mapping

http://www.scholarpedia.org/article/Kohonen_network
 Text algorithm uses normalization
input pattern = [1 0 0 0]ʼ, winner is 2nd row, learning rate = 1,
using post-normalization of weights
 Text Example:
Learning Basis Vectors
Text Example: 3x3 competitive, no cooperation
(3D input, 3D output) (nHood = 0)

Whiter is higher value

unit 1
prefers patterns 4-6
unit 3
prefers patterns 1-3
Text Example: 11x3 map, no cooperation
(11D input, 3D output) (nHood = 0)
Results for 11x3 and 11x7 maps
No cooperation (nHood = 0)
Requirements for Map Formation

• Non-Orthogonality: Input patterns exhibit

some overlap.

• Clustering: Input patterns are exhibit

selective mutual affinities.

• Neighborhood Structuring: Output units

have some kind of topological relationship
 Map Formation

Pairs of input patterns with affinities will

tend to activate output units that are
close topologically.
Example: Retinotopic Map

http://cogsci.bme.hu/~ikovacs/SandP/prepI_4_2_files/fig2.jpg
 Modeling Topology

• The simplest non-trivial topology is a

linear organization of output units.

• The text uses unit indexing for this

purpose.
% KohonenSOM.m
% this script implements a self-organizing map using the
% basic Kohonen algorithm;
%
Competition Code (text book)
pattern array InPat must already be available in the workspace

nOut=7; % set the number of output units

a=1; % set the learning rate
dec=1; % set the learning rate decrement
nHood=1; % set the neighborhood size
nIts=100; % set the number of iterations
[nPat,nIn]=size(InPat); % find number of patterns and of input units
V=rand(nOut,nIn); % set initially random connectivity matrix
for c=1:nIts, % for each learning iteration
pIndx=ceil(rand*nPat); % choose an input pattern at random
x=InPat(pIndx,:)'; % set input vector x to chosen pattern
y=V*x; % compute output to chosen input pattern
[winVal winIndx]=max(y); % find the index of the winning output
fn=winIndx-nHood; % find first neighbor in training neighborhood
ln=winIndx+nHood; % find last neighbor in training neighborhood
if fn < 1, fn=1; end, % keep first neighbor in bounds
if ln > nOut, ln=nOut; end, % keep last neighbor in bounds
for h=fn:ln, % for all units in training neighborhood
hld=V(h,:)+a*x'; % apply Kohonen update
V(h,:)=hld/norm(hld); % normalize new weight vector
a=a*(dec); % decrement the learning rate
end, % end neighbor training loop
end; % end learning loop
Out=(V*InPat')'; % find the output for all input patterns
Effect of Adding Topology: 11x7 Network
(nHood = 1)
 Alternate Result Possibilities
due to random selection in map formation
“Fractured” Map
Modeling Tonotopic Maps in
the Auditory System
Linear Map
 Tonotopic Map in
Monkey Auditory Cortex

http://www.owlnet.rice.edu/~psyc351/Images/TonotopicMapCortex.jpg
 Cochlear Tonotopic Map
(Culler, et al., 1943)

http://www.owlnet.rice.edu/~psyc351/Images/CochlearTonotopicMap.jpg
 Tonotopic Map Columnar Organization

http://www.d.umn.edu/~jfitzake/Lectures/UndergradPharmacy/SensoryPhysiology/Audition/TonotopicMaps.html
Possible Broad Tuning Curves
for Auditory Neurons
Modeled Responses of Input Units to
Two Different Frequencies
Unit Responses to Different Frequencies
for Trained Map
Modeling Orientation Selectivity
in the Visual Cortex

Hubel’s book on-line:

http://neuro.med.harvard.edu/site/dh/bcontex.htm
Modeling Angles of an Elongated Stimulus:
Code 2D Orientations as a Bit Vector
(after van der Malsburg)
Simplified Orientation Codings
 One-Dimensional Version of
the 4 Orientation Encodings

Patterns
Map Trained from Orientation Encodings
25x7 network

Note interpolatory responses

provided by the map.
Map of Trained Responses to Angular Stimuli
(van der Malsburg)

Spiral
patterns
show
related
orientations
Columnar Organization Across Neural Layers,
Maps Within Layers

Established by
Hubel & Wiesel, 1974
Possible Multisensory Map
in the Superior Colliculus
Multisensory Selectivity Distribution & Map
(V = Visual, A = Auditory, S = Somatosensory)

“pinwheel” of relationships
in the map
More Science and Engineering
Applications of SOMs
 Phonetic Typewriter:
Kohonen’s Original Application

• Objective: Spoken words →phonemes

• Languages: Finnish, Japanese
• Claimed > 92% accuracy in 10 mins. of
training.
Phonetic typewriter setup
15 channels
microphone filter & normalize

12-bit
256 pt
each 10 ms
Both Pre- and Post-Processing
are Common
Resulting Phonotopic Map
 Spectra Corresponding to Quasi-Phonemes

“quasi” is because true phonemes The symbol # stands for the

do not have uniform length, as these do. stop consonants /k,p,t/.

Spectrum Phoneme

http://www.scholarpedia.org/article/Kohonen_network
An Output Sequence on
Phonotopic Map
 Matlab demos
nnd14fm1, nnd14fm2
Same input spaces, different imposed dimensions
1D Organization 2D Organization
 The grid adjusts to represent
the data distribution

• Non-rectangular data distribution

http://www.patol.com/java/fill/index.html
 3D data to 2D Grid

http://rfhs8012.fh-regensburg.de/~saj39122/jfroehl/diplom/e-index.html
 Heuristic Solutions to the TSP
• Finding a heuristic (not necessarily optimal) solution to the
Euclidean Traveling Salesperson Problem using Kohonen net
• “Elastic net” approach

red points = neurons,

red lines = overlay
circles = cities
neurons travel toward cities

http://www.patol.com/java/TSP/index.html
This approach to the TSP is neither optimal nor
particularly fast, just “interesting”

http://eple.hib.no/~hib00ssl/TSPKohonen.html
 Multi-Competitive Learning Demo
including “Growing Neural Gas”
http://www.sund.de/netze/applets/gng/full/GNG_0.html
• The following competitive algorithms are available in one applet:
• LBG (Linde, Buzo, Gray) [extension of “k-means clustering”]
• LBG-U (Fritzke)
• Hard Competitive Learning (standard algorithm)
• Neural Gas (Martinetz and Schulten)
• Competitive Hebbian Learning (Martinetz and Schulten)
• Neural Gas with Competitive Hebbian Learning (Martinetz and Schulten)
• Growing Neural Gas (Fritzke)
• Growing Neural Gas with Utility (GNG-U, Fritzke)
• Self-Organizing Map (Kohonen)
• Growing Grid (Fritzke)
• Thirteen choices of input distribution
 Growing Neural Gas
Shaded areas show data distribution.
Circles are neurons, which can be added and
deleted during adaptation.
Growing Neural Gas
Growing Neural Gas
Unsupervised Applications of SOMs

• Discovering similarities in data (clustering)

• Data-mining

• KDD (Knowledge Discovery in Databases)

 Color Quantization Problem
(discussed earlier)

Heckbert's median cut algorithm "Color Quantization by Dynamic Programming

and Principal Analysis" by Xiaolin Wu in ACM Transactions on Graphics,
Vol. 11, No. 4, October 1992, 348-372.
 WEBSOM: Classifying news messages
by terms occurring within (“Semantotopic map”)
(see http://websom.hut.fi/websom/comp.ai.neural-nets-new/html/root.html)

Automatically generated labels and examples of titles in

which the labels have occurred
ai - SubSymbolic AI :-O
aisb - AISB and ECCS 1997 - Call for Registrations
alamos - Graduate Research Assistantship in Los Alamos National Laboratory
atree - Atree 3.0 EK mirror sites
backpropagator's - Backpropagator's review, by Donald R. Tveter
bayes - nn and stat decisions for med: Bayes vs neorotic nets
benchmark - Benchmark data for signal processing
bootstrap - Bootstrapping vs. Bayesian
brain - Brain usage Was:Re: Function of sleep
conjugate - Moller's Scaled conjugate gradienconnect - Direct Connect Problems
consciousness - electromagnetics, brain waves and consciousness
costa - New paper available "The Application of Fuzzy Logic in
curves - Predicting Learning Curves (Ph.D. thesis, online)
decay - weight decay
elman - Elman (or Edelman?) nn, references please.
encoding - Question on Encoding Input Data
engines - IC ENGINES & NEURAL NETS
extrapolation - Generalization (Interpolation & Extrapolation)
fortran - Neural Net Fortran Code
ga - GA for learning in ANN
genesis - GENESIS 2.0
hidden - Hidden layer heuristics
intelligence - Commercial Intelligence?
interpreted - Lisp is not an interpreted language
java - NN in Java?
Unexpected Semantic Encounter
(O’Reilly books Safari)
 Example: Classifying World Poverty
from a 39-dimension indicator
(http://www.cis.hut.fi/research/som-research/worldmap.html)

The colors represent neurons/centers and

were automatically assigned after
2-D clustering using a Kohonen map.
Colors (cluster representatives)
transferred to a world map
 Poverty Map Interpretation
The same ordered display can be used for illustrating the
clustering density in different regions of the data space.
The density of the reference vectors of an organized map
will reflect the density of the input samples.

In clustered areas the reference vectors will be close to

each other, and in the empty space between the clusters
they will be more sparse. Thus, the cluster structure in the
data set can be brought visible by displaying the distances
between reference vectors of neighboring units.
 Outliers
In measurement data there may exist outliers, data items lying very far from
the main body of the data. The outliers may result, for instance, from
measurement errors or typing errors made while inserting the statistics into a
data base. In such cases it would be desirable that the outliers would not affect
the result of the analysis. This is indeed the case for map displays generated
by the SOM algorithm.

Each outlier affects only one map unit and its neighborhood, while the
rest of the display may still be used for inspecting the rest of the data.

Furthermore, the outliers can be easily detected based on the clustering

display: the input space is, by definition, very sparsely populated near
the outliers.

If desired, the outliers can then be discarded and the analysis can be
continued with the rest of the data set. It is also possible that the outliers are
not erroneous but that some data items really are strikingly different from the
rest. In any case the map display reveals the outliers, whereby they can either
be discarded or paid special attention to.
Congressional Voting Patterns
http://en.wikipedia.org/wiki/Self-organizing_map

A self-organizing map
showing U.S. Congress
voting patterns
visualized in Synapse.

The first two boxes show

clustering and distances
while the remaining ones
show the component
planes.

Red means a yes vote

while blue means a no
vote in the component
planes (except the party
component where red
is Republican and blue
is Democratic).
 Tree Data-Structure Experiment
• Encode a 2-dimensional tree using 5-
dimensional vectors as shown:
ABCDE
F
G
H K L M N O P Q R
I S W
J T X 1 2 3 4 5 6
U Y
V Z
 Tree Data Structure Experiment

• Use a 2D imposed structure, with 10 x 7

neurons

• Results:
Branches of tree tend to self-align in
rows or columns of the array.

5D → 2D mapping
Tree Data Structure Experiment

Compare Original 5DTree

ABCDE
F
G
H K L M N O P Q R
I S W
J T X 1 2 3 4 5 6
U Y
V Z
 Tree Data Structure
Using Hexagonal Array

Compare Original 5DTree

ABCDE
F
G
H K L M N O P Q R
I S W
J T X 1 2 3 4 5 6
U Y
V Z
 Supervised Applications of
Kohonen Nets

• Most applications that can be treated

with feed-forward networks can also be
treated in some way by variants of
Kohonen nets.

• Example: Learning a function f:D→R is

done by treating the sample space as a
set of (d, r) pairs.
 Function Approximation
Example

• Assume 1-dimensional domain for

simplicity (not required in general).

• Represent pairs y = f(x) as (x, y).

• Learn pairs by a 2-D Kohonen net.

A Possible Aberration
from “Fractured” Map
 Master-Slave Technique

• A technique known as “master-slave” can

reduce the likelihood of an aberration, as
shown on a previous diagram.
• Select competition winners based only on x, not
on the pair (x, y).

• The weight components corresponding to x are

called the master weights, and others are called
slave weights.

• Adapt both master and slave weights together.

No Aberration with Master-Slave

not using Master-Slave

 2-D Example
Josef Göppert & Wolfgang Rosenstiel, 1996
 Example:
Robot Hand-Eye Coordination

• Want a robot to coordinate its hand with

what it sees on the monitor.

• Robot arm should be able to place an object

in a given position identified in view.

• The problem is how to set the armʼs angles

(called “inverse kinematics”).
 Robot Hand-Eye
Robot
Arm
(Angles)
Video View
(x, y Coordinates)
 Use of Master-Slave

• The view is the master.

• The hand is the slave.
• Winner is determined w.r.t. the view.
• Both view and hand are adapted together
• Example:
• 20x20 neuron map, 2D overlay
• 4 weights per neuron
(2 for master/view and 2 for slave/angles)
 Robot
Arm
Video View (Angles)
(x, y Coordinates)
 Problems Unsuitable for
Kohonen Nets

• Kohonen nets work by clustering: Nearby

data points are expected to behave similarly,
e.g. have similar outputs (topology-preserving
property).

• Parity-like problems such as the XOR do not

have this property. These would tend to be
unstable for solution by a Kohonen net.
 Examples of
Diverse Applications Papers
Supervised Applications
Supervised Applications
Structured Data Classification
 Growing SOM (GSOM)
http://en.wikipedia.org/wiki/Growing_self-organizing_map

Growth Rule:
When total error
of node > growth
threshold): Grow
nodes if i is a
boundary node.

Distribute weights
to neighbors if is
a non-boundary
node.

Initialize the new

node weight
vectors to match
the neighboring
node weights.
Growing Hierarchical SOM (GHSOM)
http://www.ifs.tuwien.ac.at/~andi/ghsom/
 Hybrid SOM
(SOM + Backpropagation)
 Free Book, 35 Chapters

http://www.intechopen.com/books/self-organizing-maps-applications-and-novel-algorithm-design