DATA MINING

Introductory and Advanced Topics

Part II

Margaret H. Dunham
Department of Computer Science and Engineering
Southern Methodist University

Companion slides for the text by Dr. M.H.Dunham, Data Mining,

Introductory and Advanced Topics, Prentice Hall, 2002.
© Prentice Hall 1
 Data Mining Outline
 PART I
– Introduction
– Related Concepts
– Data Mining Techniques
 PART II
– Classification
– Clustering
– Association Rules
 PART III
– Web Mining
– Spatial Mining
– Temporal Mining

© Prentice Hall 2
 Classification Outline
Goal: Provide an overview of the classification
problem and introduce some of the basic
algorithms
 Classification Problem Overview
 Classification Techniques
– Regression
– Distance
– Decision Trees
– Rules
– Neural Networks

© Prentice Hall 3
 Classification Problem
 Given a database D={t1,t2,…,tn} and a set
of classes C={C1,…,Cm}, the
Classification Problem is to define a
mapping f:DC where each ti is assigned
to one class.
 Actually divides D into equivalence
classes.
 Prediction is similar, but may be viewed
as having infinite number of classes.

© Prentice Hall 4
 Classification Examples
 Teachers classify students’ grades as
A, B, C, D, or F.
 Identify mushrooms as poisonous or
edible.
 Predict when a river will flood.
 Identify individuals with credit risks.
 Speech recognition
 Pattern recognition

© Prentice Hall 5
 Classification Ex: Grading
x
 If x >= 90 then grade
=A. <90 >=90
 If 80<=x<90 then x A
grade =B.
<80 >=80
 If 70<=x<80 then x B
grade =C.
 If 60<=x<70 then <70 >=70
grade =D. x C
 If x<50 then grade =F. <50 >=60

© Prentice Hall
F D 6
 Classification Ex: Letter
Recognition
View letters as constructed from 5 components:

Letter A Letter B

Letter C Letter D

Letter E Letter F

© Prentice Hall 7
 Classification Techniques
 Approach:
1. Create specific model by evaluating
training data (or using domain
experts’ knowledge).
2. Apply model developed to new data.
 Classes must be predefined
 Most common techniques use DTs,
NNs, or are based on distances or
statistical methods.
© Prentice Hall 8
 Defining Classes

Distance Based

Partitioning Based

© Prentice Hall 9
 Issues in Classification
 Missing Data
– Ignore
– Replace with assumed value
 Measuring Performance
– Classification accuracy on test data
– Confusion matrix
– OC Curve

© Prentice Hall 10
 Height Example Data
Name Gender Height Output1 Output2
Kristina F 1.6m Short Medium
Jim M 2m Tall Medium
Maggie F 1.9m Medium Tall
Martha F 1.88m Medium Tall
Stephanie F 1.7m Short Medium
Bob M 1.85m Medium Medium
Kathy F 1.6m Short Medium
Dave M 1.7m Short Medium
Worth M 2.2m Tall Tall
Steven M 2.1m Tall Tall
Debbie F 1.8m Medium Medium
Todd M 1.95m Medium Medium
Kim F 1.9m Medium Tall
Amy F 1.8m Medium Medium
Wynette F 1.75m Medium Medium
© Prentice Hall 11
Classification Performance

True Positive False Negative

False Positive True Negative

© Prentice Hall 12
 Confusion Matrix Example

Using height data example with Output1

correct and Output2 actual assignment

Actual Assignment
Membership Short Medium Tall
Short 0 4 0
Medium 0 5 3
Tall 0 1 2

© Prentice Hall 13
Operating Characteristic Curve

© Prentice Hall 14
 Regression
 Assume data fits a predefined function
 Determine best values for regression
coefficients c0,c1,…,cn.
 Assume an error: y = c0+c1x1+…+cnxn+
 Estimate error using mean squared error for
training set:

© Prentice Hall 15
Linear Regression Poor Fit

© Prentice Hall 16
Classification Using Regression
 Division: Use regression function to
divide area into regions.
 Prediction: Use regression function to
predict a class membership function.
Input includes desired class.

© Prentice Hall 17
Division

© Prentice Hall 18
Prediction

© Prentice Hall 19
 Classification Using Distance
 Place items in class to which they are
“closest”.
 Must determine distance between an item
and a class.
 Classes represented by
– Centroid: Central value.
– Medoid: Representative point.
– Individual points
 Algorithm: KNN
© Prentice Hall 20
 K Nearest Neighbor (KNN):
 Training set includes classes.
 Examine K items near item to be
classified.
 New item placed in class with the most
number of close items.
 O(q) for each tuple to be classified.
(Here q is the size of the training set.)

© Prentice Hall 21
KNN

© Prentice Hall 22
KNN Algorithm

© Prentice Hall 23
Classification Using Decision
Trees
 Partitioning based: Divide search
space into rectangular regions.
 Tuple placed into class based on the
region within which it falls.
 DT approaches differ in how the tree is
built: DT Induction
 Internal nodes associated with attribute
and arcs with values for that attribute.
 Algorithms: ID3, C4.5, CART
© Prentice Hall 24
 Decision Tree
Given:
– D = {t1, …, tn} where ti=<ti1, …, tih>
– Database schema contains {A1, A2, …, Ah}
– Classes C={C1, …., Cm}
Decision or Classification Tree is a tree associated
with D such that
– Each internal node is labeled with attribute, Ai
– Each arc is labeled with predicate which can be
applied to attribute at parent
– Each leaf node is labeled with a class, Cj

© Prentice Hall 25
DT Induction

© Prentice Hall 26
 DT Splits Area

M
Gender
F

Height

© Prentice Hall 27
 Comparing DTs

Balanced
Deep
© Prentice Hall 28
 DT Issues
 Choosing Splitting Attributes
 Ordering of Splitting Attributes
 Splits
 Tree Structure
 Stopping Criteria
 Training Data
 Pruning

© Prentice Hall 29
Decision Tree Induction is often based on
Information Theory

So

© Prentice Hall 30
Information

© Prentice Hall 31
 DT Induction
 When all the marbles in the bowl are
mixed up, little information is given.
 When the marbles in the bowl are all
from one class and those in the other
two classes are on either side, more
information is given.

Use this approach with DT Induction !

© Prentice Hall 32
 Information/Entropy
 Given probabilitites p1, p2, .., ps whose sum is
1, Entropy is defined as:

 Entropy measures the amount of randomness

or surprise or uncertainty.
 Goal in classification
– no surprise
– entropy = 0

© Prentice Hall 33
 Entropy

log (1/p) H(p,1-p)

© Prentice Hall 34
 ID3
 Creates tree using information theory
concepts and tries to reduce expected
number of comparison..
 ID3 chooses split attribute with the highest
information gain:

© Prentice Hall 35
 ID3 Example (Output1)
 Starting state entropy:
4/15 log(15/4) + 8/15 log(15/8) + 3/15 log(15/3) = 0.4384
 Gain using gender:
– Female: 3/9 log(9/3)+6/9 log(9/6)=0.2764
– Male: 1/6 (log 6/1) + 2/6 log(6/2) + 3/6 log(6/3) =
0.4392
– Weighted sum: (9/15)(0.2764) + (6/15)(0.4392) =
0.34152
– Gain: 0.4384 – 0.34152 = 0.09688
 Gain using height:
0.4384 – (2/15)(0.301) = 0.3983
 Choose height as first splitting attribute

© Prentice Hall 36
 C4.5
 ID3 favors attributes with large number of
divisions
 Improved version of ID3:
– Missing Data
– Continuous Data
– Pruning
– Rules
– GainRatio:

© Prentice Hall 37
 CART
 Create Binary Tree
 Uses entropy
 Formula to choose split point, s, for node t:

 PL,PR probability that a tuple in the training set

will be on the left or right side of the tree.

© Prentice Hall 38
 CART Example
 At the start, there are six choices for
split point (right branch on equality):
– P(Gender)=2(6/15)(9/15)(2/15 + 4/15 + 3/15)=0.224
– P(1.6) = 0
– P(1.7) = 2(2/15)(13/15)(0 + 8/15 + 3/15) = 0.169
– P(1.8) = 2(5/15)(10/15)(4/15 + 6/15 + 3/15) = 0.385
– P(1.9) = 2(9/15)(6/15)(4/15 + 2/15 + 3/15) = 0.256
– P(2.0) = 2(12/15)(3/15)(4/15 + 8/15 + 3/15) = 0.32
 Split at 1.8

© Prentice Hall 39
 Classification Using Neural
Networks
 Typical NN structure for classification:
– One output node per class
– Output value is class membership function value
 Supervised learning
 For each tuple in training set, propagate it
through NN. Adjust weights on edges to
improve future classification.
 Algorithms: Propagation, Backpropagation,
Gradient Descent
© Prentice Hall 40
 NN Issues
 Number of source nodes
 Number of hidden layers
 Training data
 Number of sinks
 Interconnections
 Weights
 Activation Functions
 Learning Technique
 When to stop learning
© Prentice Hall 41
Decision Tree vs. Neural
Network

© Prentice Hall 42
 Propagation

Tuple Input

Output

© Prentice Hall 43
NN Propagation Algorithm

© Prentice Hall 44
Example Propagation

© Prentie Hall

© Prentice Hall 45
 NN Learning
 Adjust weights to perform better with the
associated test data.
 Supervised: Use feedback from
knowledge of correct classification.
 Unsupervised: No knowledge of
correct classification needed.

© Prentice Hall 46
NN Supervised Learning

© Prentice Hall 47
 Supervised Learning
 Possible error values assuming output from
node i is yi but should be di:

 Change weights on arcs based on estimated

error
© Prentice Hall 48
 NN Backpropagation
 Propagate changes to weights
backward from output layer to input
layer.
 Delta Rule:  wij= c xij (dj – yj)
 Gradient Descent: technique to modify
the weights in the graph.

© Prentice Hall 49
Backpropagation

Error

© Prentice Hall 50
Backpropagation Algorithm

© Prentice Hall 51
Gradient Descent

© Prentice Hall 52
Gradient Descent Algorithm

© Prentice Hall 53
Output Layer Learning

© Prentice Hall 54
Hidden Layer Learning

© Prentice Hall 55
 Types of NNs
 Different NN structures used for
different problems.
 Perceptron
 Self Organizing Feature Map
 Radial Basis Function Network

© Prentice Hall 56
 Perceptron

 Perceptron is one of the simplest NNs.

 No hidden layers.

© Prentice Hall 57
 Perceptron Example
 Suppose:
– Summation: S=3x1+2x2-6
– Activation: if S>0 then 1 else 0

© Prentice Hall 58
 Self Organizing Feature Map
(SOFM)
 Competitive Unsupervised Learning
 Observe how neurons work in brain:
– Firing impacts firing of those near
– Neurons far apart inhibit each other
– Neurons have specific nonoverlapping
tasks
 Ex: Kohonen Network

© Prentice Hall 59
Kohonen Network

© Prentice Hall 60
 Kohonen Network
 Competitive Layer – viewed as 2D grid
 Similarity between competitive nodes and
input nodes:
– Input: X = <x1, …, xh>
– Weights: <w1i, … , whi>
– Similarity defined based on dot product
 Competitive node most similar to input “wins”
 Winning node weights (as well as surrounding
node weights) increased.
© Prentice Hall 61
Radial Basis Function Network
 RBF function has Gaussian shape
 RBF Networks
– Three Layers
– Hidden layer – Gaussian activation
function
– Output layer – Linear activation function

© Prentice Hall 62
Radial Basis Function Network

© Prentice Hall 63
 Classification Using Rules
 Perform classification using If-Then
rules
 Classification Rule: r = <a,c>
Antecedent, Consequent
 May generate from from other
techniques (DT, NN) or generate
directly.
 Algorithms: Gen, RX, 1R, PRISM

© Prentice Hall 64
Generating Rules from DTs

© Prentice Hall 65
Generating Rules Example

© Prentice Hall 66
Generating Rules from NNs

© Prentice Hall 67
1R Algorithm

© Prentice Hall 68
1R Example

© Prentice Hall 69
PRISM Algorithm

© Prentice Hall 70
PRISM Example

© Prentice Hall 71
 Decision Tree vs. Rules
 Tree has implied  Rules have no
order in which ordering of
splitting is predicates.
performed.
 Tree created based  Only need to look at
on looking at all one class to
classes. generate its rules.

© Prentice Hall 72
 Clustering Outline
Goal: Provide an overview of the clustering
problem and introduce some of the basic
algorithms

 Clustering Problem Overview

 Clustering Techniques
– Hierarchical Algorithms
– Partitional Algorithms
– Genetic Algorithm
– Clustering Large Databases

© Prentice Hall 73
 Clustering Examples
 Segment customer database based on
similar buying patterns.
 Group houses in a town into
neighborhoods based on similar
features.
 Identify new plant species
 Identify similar Web usage patterns

© Prentice Hall 74
Clustering Example

© Prentice Hall 75
Clustering Houses

Geographic
SizeDistance
Based Based

© Prentice Hall 76
 Clustering vs. Classification
 No prior knowledge
– Number of clusters
– Meaning of clusters
 Unsupervised learning

© Prentice Hall 77
 Clustering Issues
 Outlier handling
 Dynamic data
 Interpreting results
 Evaluating results
 Number of clusters
 Data to be used
 Scalability

© Prentice Hall 78
Impact of Outliers on
Clustering

© Prentice Hall 79
 Clustering Problem
 Given a database D={t1,t2,…,tn} of tuples
and an integer value k, the Clustering
Problem is to define a mapping
f:D{1,..,k} where each ti is assigned to
one cluster Kj, 1<=j<=k.
 A Cluster, Kj, contains precisely those
tuples mapped to it.
 Unlike classification problem, clusters
are not known a priori.
© Prentice Hall 80
 Types of Clustering
 Hierarchical – Nested set of clusters
created.
 Partitional – One set of clusters
created.
 Incremental – Each element handled
one at a time.
 Simultaneous – All elements handled
together.
 Overlapping/Non-overlapping
© Prentice Hall 81
 Clustering Approaches

Clustering

Hierarchical Partitional Categorical Large DB

Agglomerative Divisive Sampling Compression

© Prentice Hall 82
Cluster Parameters

© Prentice Hall 83
 Distance Between Clusters
 Single Link: smallest distance between points
 Complete Link: largest distance between points
 Average Link: average distance between points
 Centroid: distance between centroids

© Prentice Hall 84
 Hierarchical Clustering
 Clusters are created in levels actually
creating sets of clusters at each level.
 Agglomerative
– Initially each item in its own cluster
– Iteratively clusters are merged together
– Bottom Up
 Divisive
– Initially all items in one cluster
– Large clusters are successively divided
– Top Down

© Prentice Hall 85
 Hierarchical Algorithms
 Single Link
 MST Single Link
 Complete Link
 Average Link

© Prentice Hall 86
 Dendrogram
 Dendrogram: a tree data
structure which illustrates
hierarchical clustering
techniques.
 Each level shows clusters
for that level.
– Leaf – individual clusters
– Root – one cluster
 A cluster at level i is the
union of its children clusters
at level i+1.

© Prentice Hall 87
Levels of Clustering

© Prentice Hall 88
 Agglomerative Example
A B C D E A B
A 0 1 2 2 3
B 1 0 2 4 3
C 2 2 0 1 5 E C
D 2 4 1 0 3
E 3 3 5 3 0
D

Threshold of
1 2 34 5

A B C D E
© Prentice Hall 89
 MST Example

A B
A B C D E
A 0 1 2 2 3
B 1 0 2 4 3 E C
C 2 2 0 1 5
D 2 4 1 0 3
E 3 3 5 3 0 D

© Prentice Hall 90
Agglomerative Algorithm

© Prentice Hall 91
 Single Link
 View all items with links (distances)
between them.
 Finds maximal connected components
in this graph.
 Two clusters are merged if there is at
least one edge which connects them.
 Uses threshold distances at each level.
 Could be agglomerative or divisive.

© Prentice Hall 92
MST Single Link Algorithm

© Prentice Hall 93
Single Link Clustering

© Prentice Hall 94
 Partitional Clustering
 Nonhierarchical
 Creates clusters in one step as opposed
to several steps.
 Since only one set of clusters is output,
the user normally has to input the
desired number of clusters, k.
 Usually deals with static sets.

© Prentice Hall 95
 Partitional Algorithms
 MST
 Squared Error
 K-Means
 Nearest Neighbor
 PAM
 BEA
 GA

© Prentice Hall 96
MST Algorithm

© Prentice Hall 97
 Squared Error
 Minimized squared error

© Prentice Hall 98
Squared Error Algorithm

© Prentice Hall 99
 K-Means
 Initial set of clusters randomly chosen.
 Iteratively, items are moved among sets
of clusters until the desired set is
reached.
 High degree of similarity among
elements in a cluster is obtained.
 Given a cluster Ki={ti1,ti2,…,tim}, the
cluster mean is mi = (1/m)(ti1 + … + tim)

© Prentice Hall 100

 K-Means Example
 Given: {2,4,10,12,3,20,30,11,25}, k=2
 Randomly assign means: m1=3,m2=4
 K1={2,3}, K2={4,10,12,20,30,11,25},
m1=2.5,m2=16
 K1={2,3,4},K2={10,12,20,30,11,25}, m1=3,m2=18
 K1={2,3,4,10},K2={12,20,30,11,25},
m1=4.75,m2=19.6
 K1={2,3,4,10,11,12},K2={20,30,25}, m1=7,m2=25
 Stop as the clusters with these means are the
same.

© Prentice Hall 101

K-Means Algorithm

© Prentice Hall 102

 Nearest Neighbor
 Items are iteratively merged into the
existing clusters that are closest.
 Incremental
 Threshold, t, used to determine if items
are added to existing clusters or a new
cluster is created.

© Prentice Hall 103

Nearest Neighbor Algorithm

© Prentice Hall 104

 PAM
 Partitioning Around Medoids (PAM)
(K-Medoids)
 Handles outliers well.
 Ordering of input does not impact results.
 Does not scale well.
 Each cluster represented by one item,
called the medoid.
 Initial set of k medoids randomly chosen.

© Prentice Hall 105

PAM

© Prentice Hall 106

 PAM Cost Calculation
 At each step in algorithm, medoids are
changed if the overall cost is improved.
 Cjih – cost change for an item tj associated
with swapping medoid ti with non-medoid th.

© Prentice Hall 107

PAM Algorithm

© Prentice Hall 108

 BEA
 Bond Energy Algorithm
 Database design (physical and logical)
 Vertical fragmentation
 Determine affinity (bond) between attributes
based on common usage.
 Algorithm outline:
1. Create affinity matrix
2. Convert to BOND matrix
3. Create regions of close bonding

© Prentice Hall 109

 BEA

Modified from [OV99]

© Prentice Hall 110

 Genetic Algorithm Example
 {A,B,C,D,E,F,G,H}
 Randomly choose initial solution:
{A,C,E} {B,F} {D,G,H} or
10101000, 01000100, 00010011
 Suppose crossover at point four and
choose 1st and 3rd individuals:
10100011, 01000100, 00011000
 What should termination criteria be?

© Prentice Hall 111

GA Algorithm

© Prentice Hall 112

 Clustering Large Databases
 Most clustering algorithms assume a large
data structure which is memory resident.
 Clustering may be performed first on a
sample of the database then applied to the
entire database.
 Algorithms
– BIRCH
– DBSCAN
– CURE

© Prentice Hall 113

 Desired Features for Large
Databases
 One scan (or less) of DB
 Online
 Suspendable, stoppable, resumable
 Incremental
 Work with limited main memory
 Different techniques to scan (e.g.
sampling)
 Process each tuple once

© Prentice Hall 114

 BIRCH
 Balanced Iterative Reducing and
Clustering using Hierarchies
 Incremental, hierarchical, one scan
 Save clustering information in a tree
 Each entry in the tree contains
information about one cluster
 New nodes inserted in closest entry in
tree
© Prentice Hall 115
 Clustering Feature
 CT Triple: (N,LS,SS)
– N: Number of points in cluster
– LS: Sum of points in the cluster
– SS: Sum of squares of points in the cluster
 CF Tree
– Balanced search tree
– Node has CF triple for each child
– Leaf node represents cluster and has CF value
for each subcluster in it.
– Subcluster has maximum diameter

© Prentice Hall 116

BIRCH Algorithm

© Prentice Hall 117

Improve Clusters

© Prentice Hall 118

 DBSCAN
 Density Based Spatial Clustering of
Applications with Noise
 Outliers will not effect creation of cluster.
 Input
– MinPts – minimum number of points in
cluster
– Eps – for each point in cluster there must
be another point in it less than this distance
away.

© Prentice Hall 119

 DBSCAN Density Concepts
 Eps-neighborhood: Points within Eps distance
of a point.
 Core point: Eps-neighborhood dense enough
(MinPts)
 Directly density-reachable: A point p is directly
density-reachable from a point q if the distance
is small (Eps) and q is a core point.
 Density-reachable: A point si density-
reachable form another point if there is a path
from one to the other consisting of only core
points.
© Prentice Hall 120
Density Concepts

© Prentice Hall 121

DBSCAN Algorithm

© Prentice Hall 122

 CURE
 Clustering Using Representatives
 Use many points to represent a cluster
instead of only one
 Points will be well scattered

© Prentice Hall 123

CURE Approach

© Prentice Hall 124

CURE Algorithm

© Prentice Hall 125

CURE for Large Databases

© Prentice Hall 126

Comparison of Clustering
Techniques

© Prentice Hall 127

 Association Rules Outline
Goal: Provide an overview of basic Association
Rule mining techniques
 Association Rules Problem Overview
– Large itemsets
 Association Rules Algorithms
– Apriori
– Sampling
– Partitioning
– Parallel Algorithms
 Comparing Techniques
 Incremental Algorithms
 Advanced AR Techniques

© Prentice Hall 128

Example: Market Basket Data
 Items frequently purchased together:
Bread PeanutButter
 Uses:
– Placement
– Advertising
– Sales
– Coupons
 Objective: increase sales and reduce
costs
© Prentice Hall 129
 Association Rule Definitions
 Set of items: I={I1,I2,…,Im}
 Transactions: D={t1,t2, …, tn}, tj I

 Itemset: {Ii1,Ii2, …, Iik}  I

 Support of an itemset: Percentage of
transactions which contain that itemset.
 Large (Frequent) itemset: Itemset
whose number of occurrences is above
a threshold.
© Prentice Hall 130
 Association Rules Example

I = { Beer, Bread, Jelly, Milk, PeanutButter}

Support of {Bread,PeanutButter} is 60%
© Prentice Hall 131
 Association Rule Definitions
 Association Rule (AR): implication X
 Y where X,Y  I and X  Y = ;
 Support of AR (s) X  Y:
Percentage of transactions that
contain X Y
 Confidence of AR () X  Y: Ratio of
number of transactions that contain X
 Y to the number that contain X

© Prentice Hall 132

Association Rules Ex (cont’d)

© Prentice Hall 133

 Association Rule Problem
 Given a set of items I={I1,I2,…,Im} and a
database of transactions D={t1,t2, …, tn}
where ti={Ii1,Ii2, …, Iik} and Iij  I, the
Association Rule Problem is to
identify all association rules X  Y with
a minimum support and confidence.
 Link Analysis
 NOTE: Support of X  Y is same as
support of X  Y.

© Prentice Hall 134

 Association Rule Techniques
1. Find Large Itemsets.
2. Generate rules from frequent itemsets.

© Prentice Hall 135

Algorithm to Generate ARs

© Prentice Hall 136

 Apriori
 Large Itemset Property:
Any subset of a large itemset is large.
 Contrapositive:

If an itemset is not large,

none of its supersets are large.

© Prentice Hall 137

Large Itemset Property

© Prentice Hall 138

Apriori Ex (cont’d)

s=30%  = 50%
© Prentice Hall 139
 Apriori Algorithm
1. C1 = Itemsets of size one in I;
2. Determine all large itemsets of size 1, L1;
3. i = 1;
4. Repeat
5. i = i + 1;
6. Ci = Apriori-Gen(Li-1);
7. Count Ci to determine Li;
8. until no more large itemsets found;

© Prentice Hall 140

 Apriori-Gen
 Generate candidates of size i+1 from
large itemsets of size i.
 Approach used: join large itemsets of
size i if they agree on i-1
 May also prune candidates who have
subsets that are not large.

© Prentice Hall 141

Apriori-Gen Example

© Prentice Hall 142

Apriori-Gen Example (cont’d)

© Prentice Hall 143

 Apriori Adv/Disadv
 Advantages:
– Uses large itemset property.
– Easily parallelized
– Easy to implement.
 Disadvantages:
– Assumes transaction database is memory
resident.
– Requires up to m database scans.

© Prentice Hall 144

 Sampling
 Large databases
 Sample the database and apply Apriori to the
sample.
 Potentially Large Itemsets (PL): Large
itemsets from sample
 Negative Border (BD - ):
– Generalization of Apriori-Gen applied to
itemsets of varying sizes.
– Minimal set of itemsets which are not in PL,
but whose subsets are all in PL.

© Prentice Hall 145

Negative Border Example

PL PL BD-(PL)
© Prentice Hall 146
 Sampling Algorithm
1. Ds = sample of Database D;
2. PL = Large itemsets in Ds using smalls;
3. C = PL  BD-(PL);
4. Count C in Database using s;
5. ML = large itemsets in BD-(PL);
6. If ML =  then done
7. else C = repeated application of BD-;
8. Count C in Database;
© Prentice Hall 147
 Sampling Example
 Find AR assuming s = 20%
 Ds = { t1,t2}
 Smalls = 10%
 PL = {{Bread}, {Jelly}, {PeanutButter},
{Bread,Jelly}, {Bread,PeanutButter}, {Jelly,
PeanutButter}, {Bread,Jelly,PeanutButter}}
 BD-(PL)={{Beer},{Milk}}
 ML = {{Beer}, {Milk}}
 Repeated application of BD- generates all
remaining itemsets

© Prentice Hall 148

 Sampling Adv/Disadv
 Advantages:
– Reduces number of database scans to one
in the best case and two in worst.
– Scales better.
 Disadvantages:
– Potentially large number of candidates in
second pass

© Prentice Hall 149

 Partitioning
 Divide database into partitions D1,D2,
…,Dp
 Apply Apriori to each partition
 Any large itemset must be large in at
least one partition.

© Prentice Hall 150

 Partitioning Algorithm
1. Divide D into partitions D1,D2,…,Dp;
2. For I = 1 to p do
3. Li = Apriori(Di);
4. C = L1  …  Lp;
5. Count C on D to generate L;

© Prentice Hall 151

 Partitioning Example
L1 ={{Bread}, {Jelly},
{PeanutButter},
{Bread,Jelly},
{Bread,PeanutButter},
D1 {Jelly, PeanutButter},
{Bread,Jelly,PeanutButter}}

L2 ={{Bread}, {Milk},
D2 {PeanutButter}, {Bread,Milk},
{Bread,PeanutButter}, {Milk,
PeanutButter},
{Bread,Milk,PeanutButter},
S=10% {Beer}, {Beer,Bread},
{Beer,Milk}}
© Prentice Hall 152
 Partitioning Adv/Disadv
 Advantages:
– Adapts to available main memory
– Easily parallelized
– Maximum number of database scans is
two.
 Disadvantages:
– May have many candidates during second
scan.

© Prentice Hall 153

 Parallelizing AR Algorithms
 Based on Apriori
 Techniques differ:
– What is counted at each site
– How data (transactions) are distributed
 Data Parallelism
– Data partitioned
– Count Distribution Algorithm
 Task Parallelism
– Data and candidates partitioned
– Data Distribution Algorithm

© Prentice Hall 154

Count Distribution Algorithm(CDA)
1. Place data partition at each site.
2. In Parallel at each site do
3. C1 = Itemsets of size one in I;
4. Count C1;
5. Broadcast counts to all sites;
6. Determine global large itemsets of size 1, L1;
7. i = 1;
8. Repeat
9. i = i + 1;
10. Ci = Apriori-Gen(Li-1);
11. Count Ci;
12. Broadcast counts to all sites;
13. Determine global large itemsets of size i, Li;
14. until no more large itemsets found;

© Prentice Hall 155

CDA Example

© Prentice Hall 156

Data Distribution Algorithm(DDA)
1. Place data partition at each site.
2. In Parallel at each site do
3. Determine local candidates of size 1 to count;
4. Broadcast local transactions to other sites;
5. Count local candidates of size 1 on all data;
6. Determine large itemsets of size 1 for local
candidates;
7. Broadcast large itemsets to all sites;
8. Determine L1;
9. i = 1;
10. Repeat
11. i = i + 1;
12. Ci = Apriori-Gen(Li-1);
13. Determine local candidates of size i to count;
14. Count, broadcast, and find Li;
15. until no more large itemsets found;

© Prentice Hall 157

DDA Example

© Prentice Hall 158

 Comparing AR Techniques
 Target
 Type
 Data Type
 Data Source
 Technique
 Itemset Strategy and Data Structure
 Transaction Strategy and Data Structure
 Optimization
 Architecture
 Parallelism Strategy

© Prentice Hall 159

Comparison of AR Techniques

© Prentice Hall 160

Hash Tree

© Prentice Hall 161

Incremental Association Rules
 Generate ARs in a dynamic database.
 Problem: algorithms assume static
database
 Objective:
– Know large itemsets for D
– Find large itemsets for D  { D}
 Must be large in either D or  D
 Save Li and counts

© Prentice Hall 162

 Note on ARs
 Many applications outside market
basket data analysis
– Prediction (telecom switch failure)
– Web usage mining
 Many different types of association rules
– Temporal
– Spatial
– Causal

© Prentice Hall 163

 Advanced AR Techniques
 Generalized Association Rules
 Multiple-Level Association Rules
 Quantitative Association Rules
 Using multiple minimum supports
 Correlation Rules

© Prentice Hall 164

 Measuring Quality of Rules
 Support
 Confidence
 Interest
 Conviction
 Chi Squared Test

© Prentice Hall 165