See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/259235118

Random Forests and Decision Trees

Article · September 2012

CITATIONS READS
868 50,672

4 authors, including:

Rehanullah Khan Nasir Ahmad

Qassim University University of Engineering and Technology, Peshawar
115 PUBLICATIONS 2,365 CITATIONS 105 PUBLICATIONS 1,688 CITATIONS

SEE PROFILE SEE PROFILE

All content following this page was uploaded by Nasir Ahmad on 12 December 2013.

The user has requested enhancement of the downloaded file.

IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 5, No 3, September 2012
ISSN (Online): 1694-0814
www.IJCSI.org 272

Random Forests and Decision Trees

Jehad Ali1, Rehanullah Khan2, Nasir Ahmad3, Imran Maqsood4
1
Computer Systems Engineering, UET Peshawar, Pakistan
2
Sarhad University of Science and Information Technology, Peshawar, Pakistan
3
Computer Systems Engineering, UET Peshawar, Pakistan
4
Computer Software Engineering, UET Mardan, Pakistan

Abstract especially if Decision Trees are set on data the size of

which is large i.e. it has more number of instances.
In this paper, we have compared the classification results of two In 2001, Breiman [4] presented the idea of Random
models i.e. Random Forest and the J48 for classifying twenty Forests which perform well as compared with other
versatile datasets. We took 20 data sets available from UCI classifiers including Support Vector Machines, Neural
repository [1] containing instances varying from 148 to 20000.
Networks and Discriminant Analysis, and overcomes the
We compared the classification results obtained from methods i.e.
Random Forest and Decision Tree (J48). The classification
over fitting problem.
parameters consist of correctly classified instances, incorrectly
classified instances, F-Measure, Precision, Accuracy and Recall. Those methods such as Bagging or Random subspaces [5,6]
We discussed the pros and cons of using these models for large which are made from ensemble of various classifiers and
and small data sets. The classification results show that Random those which use randomization for producing diversity
Forest gives better results for the same number of attributes and have proven to be very efficient. In order to introduce
large data sets i.e. with greater number of instances, while J48 is diversity and to build classifiers different from each other,
handy with small data sets (less number of instances). The results they use randomization in the induction process. Random
from breast cancer data set depicts that when the number of
Forests have gained a substantial interest in machine
instances increased from 286 to 699, the percentage of correctly
classified instances increased from 69.23% to 96.13% for learning because of its efficient discriminative
Random Forest i.e. for dataset with same number of attributes but classification [7, 8].
having more instances, the Random Forest accuracy increased.
In computer vision community, Random Forests were
Keywords: Random Forests, Decision Trees, J48. introduced by Lepetit et. al. [9, 10]. His work in this field
provided a foundation for papers such as class recognition
[11, 12], bi-layer video segmentation [13], image
1. Introduction classification [14] and person identification [15], which
use Random Forests. A wide range of visual cues are also
The application of the Decision Tree algorithm [2] can be enabled naturally by the Random Forest including color,
observed in various fields. Text classification and text shape, texture and depth. Random Forests are considered
extraction, comparing data statistically etc. are the fields general purpose vision tools and considered as efficient.
where they are used. Besides this in libraries books can be
classified into different categories on the basis of its type Random Forest as defined in [4] is a generic principle of
with the implementation of Decision Tree algorithm. In classifier combination that uses L tree-structured base
hospitals it can be used for diagnosis of diseases i.e. brain classifiers {h(X,Ѳn), N=1,2,3,…L}, where X denotes the
tumor, Cancer, heart problems, Hepatitis etc. Companies, input data and {Ѳn} is a family of identical and dependent
hospitals, Schools, colleges and universities use it for distributed random vectors. Every Decision Tree is made
maintaining their records. Similarly, In Stock market, it by randomly selecting the data from the available data. For
can be used for statistics. example a Random Forest for each Decision Tree (as in
Random Subspaces) can be built by randomly sampling a
Decision Tree algorithms are effective [3] in that they feature subset, and/or by the random sampling of a training
provide human-readable rules of classification. Beside this data subset for each Decision Tree (the concept of
it has some drawbacks, one of which is the sorting of all Bagging).
numerical attributes when the tree decides to split a node. In a Random Forest, the features are randomly selected in
Such split on sorting all numerical attributes becomes each decision split. The correlation between trees is
costly i.e. efficiency or running time and memory size, reduces by randomly selecting the features which improves

Copyright (c) 2012 International Journal of Computer Science Issues. All Rights Reserved.
IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 5, No 3, September 2012
ISSN (Online): 1694-0814
www.IJCSI.org 273

the prediction power and results in higher efficiency. As 2. Classification Methods

such the advantages of Random Forest are [16]:
 Overcoming the problem of over fitting
 In training data, they are less sensitive to outlier 2.1 Decision Trees
data
 Parameters can be set easily and therefore, Decision Trees embody a supervised classification
eliminates the need for pruning the trees approach [20]. The idea came from the ordinary tree
 variable importance and accuracy is generated structure which is made-up of a root and nodes (the
automatically positions where places branches divides), branches and
Random Forest not only keeps the benefits achieved by the leaves. In a similar manner, a Decision Tree is constructed
Decision Trees but through the use of bagging on samples, from nodes which represent circles and the branches are
its voting scheme [17] through which decision is made and represented by the segments that connect the nodes. A
a random subsets of variables, it most of the time achieves Decision Tree starts from the root, moves downward and
better results than Decision Trees. generally are drawn from left to right. The node from
where the tree starts is called a root node. The node where
The Random Forest is appropriate for high dimensional the chain ends is known as the “leaf” node. Two or more
data modeling because it can handle missing values and branches can be extended from each internal node i.e. a
can handle continuous, categorical and binary data. The node that is not leaf node. A node represents a certain
bootstrapping and ensemble scheme makes Random Forest characteristic while the branches represent a range of
strong enough to overcome the problems of over fitting values. These ranges of values act as a partition points for
and hence there is no need to prune the trees. Besides high the set of values of the given characteristic. Figure 1
prediction accuracy, Random Forest is efficient, describes the structure of a tree.
interpretable and non-parametric for various types of
datasets [18]. The model interpretability and prediction
accuracy provided by Random Forest is very unique
among popular machine learning methods. Accurate
predictions and better generalizations are achieved due to
utilization of ensemble strategies and random sampling.

Bagging scheme provides generalization property which

improves with the decrease of variance and improves the
over-all generalization error. As such, the decrease in bias
[19] is achieved by the boosting method. Random Forest
three main features that gained focus [17] are:
 Accurate predictions results for a variety of
applications
 Through model training, the importance of each
feature can be measured
 Trained model can measure the pair-wise
proximity between the samples
Figure 1: Tree Structure
In this article, we concentrate on the classification
performance of the Decision Tree (J48) and the Random The grouping of data in the Decision Tree is based on the
Forest for large and small datasets. The objective of this values of attributes of the given data. A Decision Tree is
comparison is creating a base-line, which will be useful for made from the pre-classified data. The division into classes
the classification scenarios. It will also help in the selection is decided upon the features that best divides the data. The
of appropriate model. data items are split according to the values of these
features. This process is applied to each split subset of the
The rest of the paper is organized as follows: Section 2 data items recursively. The process terminates as for as all
describes Decision Tree related classification algorithms the data items in current subset belong to the same class.
including the Random Forest. Experimental setup and the
datasets used are described in Section 3. Section 4 presents We use the J48 implementation of the Decision Trees of
results and conclusion. WEKA (open source software). In WEKA, we can analyze

Copyright (c) 2012 International Journal of Computer Science Issues. All Rights Reserved.
IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 5, No 3, September 2012
ISSN (Online): 1694-0814
www.IJCSI.org 274

data and besides this, it also includes implementation for node, m variables are selected at random out of
regression, data pre-processing, clustering, classification the M and the best split on these m is used for
and visualization through various algorithms. More than splitting the node. During the forest growing, the
sixty algorithms are available in WEKA. Following is an value of m is held constant.
overview of few of the Decision Tree based algorithms.  Each tree is grown to the largest possible extent.
No pruning is used.
2.1.1 REPTree
Random Forest generally exhibits a significant
In REPTree, decision/regression tree is constructed with performance improvement as compared to single tree
information gain as the splitting criterion and reduced error classifier such as C4.5. The generalization error rate
pruning is used to prune it. It sorts values only for numeric that it yields compares favorably to Adaboost,
attributes once. The method of fractional instances is used however it is more robust to noise.
to handle missing values with C4.5. REP Tree is a fast
Decision Tree learner.
3. Experimental Analysis
2.1.2 Random Tree
A random tree is a tree constructed randomly from a set of In this section, we concentrate on the classification
possible trees having K random features at each node. “At performance of the Decision Tree (J48) and the Random
random” in this context means that in the set of trees each Forest for large and small datasets. The objective of this
tree has an equal chance of being sampled. Or we can say comparison is creating a base-line, which will be useful for
that trees have a “uniform” distribution. Random trees can the classification scenarios. It will also help in the selection
be generated efficiently and the combination of large sets of appropriate model.
of random trees generally leads to accurate models. There
has been an extensive research in the recent years over 3.1 Data Sets
Random trees in the field of machine Learning.
For classification problems, we took these datasets
2.1.3 J.48
from the UCI Machine Learning repository [1]. In
Ross Quinlan [21] developed C4.5 algorithm which is used breast cancer data, some attributes are linear and few
to generate a Decision Tree. Decision Trees are produced are nominal. The detailed description, attributes,
from the J48 i.e. Open Source Java implementation of source of each dataset can be found from UCI
C4.5 release in WEKA data mining tool [22]. This is a repository. Table 1 shows the names of the dataset,
standard Decision Tree algorithm. One of the classification
the number of instances and number of attributes for
algorithms in data mining is Decision Tree Induction. The
Classification algorithm [23] is inductively learned to the twenty datasets we used for our analysis and
construct a model from the pre-classified data set. Each comparison. As an visual information, the Figures 2,
data item is defined by values of the characteristics or 3, 4 shows the distribution of data variables in the
features. Classification may be viewed as mapping from a corresponding three sampled data sets. Figure 2
set of features to a particular class. shows the Dataset Lymphography. Its total number
of instances are 148, the total attributes are 19 and
2.2 Random Forests having four classes. Figure 3 shows the Dataset
Sonar with 208 instances and 61 attributes and bi-
Random Forest developed by Leo Breiman [4] is a group
of un-pruned classification or regression trees made from classes data. Figure 4 shows the Dataset Heart-h. It
the random selection of samples of the training data. has 14 attributes and 294 instances and binary class
Random features are selected in the induction process. data.
Prediction is made by aggregating (majority vote for
classification or averaging for regression) the predictions
of the ensemble. Each tree is grown as described in [24]:
 By Sampling N randomly, If the number of cases
in the training set is N but with replacement, from
the original data. This sample will be used as the
training set for growing the tree.
 For M number of input variables, the variable m
is selected such that m<<M is specified at each

Copyright (c) 2012 International Journal of Computer Science Issues. All Rights Reserved.
IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 5, No 3, September 2012
ISSN (Online): 1694-0814
www.IJCSI.org 275

Table 1: Datasets used and their details

Serial Number Dataset name number of instances number of attributes
1 Lymph 148 19
2 Autos 205 26
3 Sonar 208 61
4 Heart-h 270 14
5 Breast cancer 286 10
6 Heart-c 303 14
7 Ionosphere 351 35
8 colic 368 23
9 Colic.org 368 28
10 Primary tumor 399 18
11 Balance Scale 625 25
12 Soyben 683 36
13 Credit a 690 16
14 Breast W 699 10
15 Vehicle 846 19
16 vowel 990 14
17 Credit g 1000 21
18 Segment 2310 20
19 Waveform 5000 41
20 Letter 20,000 17
Figure 4: Dataset Heart-h

Figure 5 and Figure 6 shows the different parameter

settings and the variables used for the J48 and the Random
Forest. Binary splits: show the use of binary splits when
building the trees. Confidence factor: shows the pruning
of the trees smaller values show more pruning. Debug: if
this is set to true additional information are displayed on
the console. Seed: used for randomizing the data when
reduced error pruning is used. Unprunned: shows whether
pruning is used or not. MinNumObj: Shows the minimum
number of instances per leaf. Save Instance Data: whether
to save data for visualization. numFolds: shows the
Figure 2: Dataset Lymphography amount of data used for pruning. Reduced error pruning:
whether reduced error pruning is used or not instead of
C.4.5. Sub-tree Raising: Used for Sub-tree rising when
we pruning is used. Use Laplace: whether counts at leafs
are smoothed based on Laplace. MaxDepth: shows the
maximum depth of the trees, 0 is used for unlimited.
numFeatures: The number of attributes used while
random selection. numTrees: the number of trees to be
generated. Seed: The random number that will be used as
seed value.

Figure 3: Dataset Sonar

Copyright (c) 2012 International Journal of Computer Science Issues. All Rights Reserved.
IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 5, No 3, September 2012
ISSN (Online): 1694-0814
www.IJCSI.org 276

Figure 5: Parameters settings for the J48 Figure 6: Parameters settings for the Random Forest

4. Results and Discussion

We compared the classification results of the J48 and the remaining for testing purpose and repeats the process 10
Random Forest. To avoid over fitting problem, we times.
obtained the accuracy using 10-fold cross validation which
uses 9/10 of data as for training the algorithm and the

Table 2: Comparison of the Random Forest and the J48 classification results for the 20 datasets.
Serial NO Data Set No. of instances No. of attributes Random Forest J-48 Results
Correctly classified Incorrectly classified Correctly classified Incorrectly classified
instances instances instances instances
1 Lymph 148 19 81.08% 18.91% 77.02% 22.97%
2 Autos 205 26 83.41% 16.58% 80.95% 18.04%
3 Sonar 208 61 80.77% 19.23% 71.15% 28.84%
4 Heart-h 270 14 77.89% 22.10% 80.95% 19.04%
5 Breast cancer 286 10 69.23% 30.76% 75.52% 24.47%
6 Heart-c 303 14 81.51% 18.48% 77.56% 22.44%
7 Ionosphere 351 35 92.88% 7.12% 91.45% 8.54%
8 colic 368 23 86.14% 13.85% 85.32% 14.67%
9 Colic.org 368 28 68.47% 31.52% 66.30% 33.69%
10 Primary tumor 399 18 42.48% 57.52% 39.82% 60.17%
11 Balance Scale 625 25 80.48% 19.52% 76.64% 23.36%
12 Soyben 683 36 91.65% 8.34% 91.50% 8.49%
13 Credit a 690 16 85.07% 14.92% 86.09% 13.91%
14 Breast W 699 10 96.13% 3.68% 94.56% 5.43%
15 Vehicle 846 19 77.06% 22.93% 72.45% 27.54%
16 vowel 990 14 96.06% 3.03% 81.51% 18.48%
17 Credit g 1000 21 72.50% 27.50% 70.50% 29.50%
18 Segment 2310 20 97.66% 2.33% 96.92% 3.07%
19 Waveform 5000 41 81.94% 18.06% 75.30% 24.70%
20 Letter 20,000 17 94.71% 5.29% 87.98% 12.02%

Copyright (c) 2012 International Journal of Computer Science Issues. All Rights Reserved.
IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 5, No 3, September 2012
ISSN (Online): 1694-0814
www.IJCSI.org 277

Table 2 shows the correctly classified instances and

incorrectly classified instances for the Random Forest and The results from breast cancer data set depicts that when
J48 classifiers, the name of the corresponding dataset, the number of instances increased from 286 to 699, the
number of instances and number of attributes are shown in correctly classified instances increased from 69.23% to
columns 2, 3 and 4 respectively. 96.13% for the Random Forest.

The classification results show that the Random Forest

gives better results for the same number of attributes and
large datasets i.e. with greater number of instances while
J48 is handy with small datasets i.e. less number of
instances.

Table 3: Comparison of Random Forest and the J48 in terms of Precision, Recall and F-measure

Dataset No. of No. of Random forest J-48

Instances attribute Precision Recall F- Precision Recall F-measure
s measure
Breast cancer 286 10 0.667 0.692 0.674 0.752 0.755 0.713
Breast W 699 10 0.962 0.961 0.961 0.946 0.946 0.946
Credit a 690 16 0.851 0.851 0.851 0.861 0.861 0.861
Credit g 1000 21 0.705 0.725 0.707 0.687 0.705 0.692
colic 368 23 0.854 0.853 0.85 0.86 0.861 0.861
Colic.org 368 28 0.662 0.685 0.63 0.44 0.663 0.529
Heart-h 270 14 0.775 0.779 0.774 0.807 0.81 0.806
Heart-c 303 14 0.819 0.815 0.813 0.776 0.776 0.774
vowel 990 14 0.961 0.961 0.961 0.816 0.815 0.815
Ionosphere 351 35 0.929 0.929 0.929 0.915 0.915 0.913
Soyben 683 36 0.926 0.917 0.918 0.917 0.915 0.913
Vehicle 846 19 0.764 0.771 0.767 0.722 0.725 0.722
Sonar 208 61 0.813 0.808 0.808 0.713 0.712 0.712
Autos 205 26 0.836 0.834 0.834 0.833 0.822 0.82
Balance Scale 625 25 0.817 0.805 0.81 0.732 0.766 0.749
Lymph 148 19 0.804 0.811 0.8 0.776 0.77 0.772
Segment 2310 20 0.977 0.977 0.977 0.969 0.969 0.969
Primary tumor 399 18 0.394 0.425 0.406 0.333 0.398 0.704
Waveform 5000 41 0.82 0.819 0.82 0.753 0.753 0.753
Letter 20,000 17 0.948 0.947 0.947 0.881 0.88 0.88
0.961 for the Random Forest classifier. Similarly for the
J48, the precision increased from 0.752 to 0.946, F-
measure from 0.713 to 0.946 and recall from 0.755 to
Table 3 shows the Precision, Recall and the F-measure for 0.946. The highest precision value obtained is for the
the Random Forest and J48 for the 20 datasets. From the Random Forest i.e. 0.977 for the Segment Dataset which
table, it can be seen that for the same data set with greater shows the highly accurate model of the Random Forest
number of instances, i.e. when the number of instances ensemble method.
increased from 286 to 699 while keeping the attributes
constant the precision increases from 0.667 to 0.962, F- From the results, it can be concluded that the Random
measure from 0.674 to 0.961 and Recall from 0.692 to Forest achieves increased classification performance and

Copyright (c) 2012 International Journal of Computer Science Issues. All Rights Reserved.
IJCSI International Journal of Computer Science Issues, Vol. 9, Issue 5, No 3, September 2012
ISSN (Online): 1694-0814
www.IJCSI.org 278

yields results that are accurate and precise in the cases of [21] Quinlan, J. R. C4.5: Programs for Machine Learning.
large number of instances. These scenarios also cover the Morgan Kaufmann Publishers, 1993.
missing values problem in the datasets and thus besides [22] http://en.wikipedia.org/wiki/C4.5_algorithm
[23] Report from Pike research,
accuracy, it also overcomes the over-fitting problem
http://www.pikeresearch.com/research/smartgrid- data-
generated due to missing values in the datasets. Therefore, analytics
for the classification problems, if one has to choose a [24]http://www.stat.berkeley.edu/~breiman/RandomForests/cc_h
classifier among the tree based classifiers set, we ome.htm#prox Symposium, volume 1, July, 2005.
recommend to use the Random Forest with confidence for
variety of classification problems.

References
[1] A. Asuncion and D. Newman, "UCI machine learning
repository," 2007. [Online]. Available:
http://archive.ics.uci.edu/ml/ Jehad Ali is pursuing his M.Sc Computer Systems Engineering
[2] T.M. Mitchell, Machine Learning. McGraw-Hill, 1997. from University of Engineering and Technology, Peshawar,
[3] Yael Ben-Haim, “A Streaming Parallel Decision Tree Pakistan. He did his B.Sc. Computer Systems Engineering from
Algorithm” , Elad Tom-Tov , 2010 the same university. He is working as a Computer Engineer in
Ghulam Ishaq Khan Institute (GIKI) of Engineering Sciences and
[4] Breiman, L., Random Forests, Machine Learning 45(1), 5-32, Technology, Topi, Pakistan. His research interest’s areas are
2001. image processing, computer vision, machine learning, Computer
[5] "Bagging predictors," Machine Learning, vol. 24, no. 2, pp. Networks and pattern recognition.
123-140, 1996.
[6] T. Ho, "The random subspace method for constructing
decision forests," IEEE Transactions on Pattern Analysis and Rehanullah Khan graduated from the University of
Machine Intelligence, vol. 20, no. 8, pp. 832-844, 1998. Engineering and Technology Peshawar, with a B.Sc degree
[7] Amit, Y., Geman, D.: Shape quantization and recognition (Computer Engineering) in 2004 and M.Sc (Information Systems)
in 2006. He obtained PhD degree (Computer Engineering) in 2011
with randomized trees. Neural Computation 9(7), 1545–1588
from Vienna University of Technology, Austria. He is currently an
(1997) Associate Professor at the Sarhad University of Science and
[8] Breiman, L.: Random Forests. ML Journal 45(1), 5–32 Technology, Peshawar. His research interests include color
(2001) interpretation, segmentation and object recognition.
[9] Lepetit, V., Fua, P.: Keypoint recognition using randomized
trees. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1465– Nasir Ahmad graduated from University of Engineering
1479 (2006) and Technology Peshawar with a B.Sc Electrical Engineering
[10] Ozuysal, M., Fua, P., Lepetit, V.: Fast keypoint recognition degree. He obtained his PhD degree from UK in 2011. He is a
in ten lines of code. In: IEEE CVPR (2007) faculty member of Department of Computer Systems Engineering,
University of Engineering and Technology Peshawar, Pakistan. His
[11] Winn, J., Criminisi, A.: Object class recognition at a glance.
Research Areas include Pattern Recognition, Computer vision and
In: IEEE CVPR, video track (2006) Digital Signal Processing.
[12] Shotton, J., Johnson, M., Cipolla, R.: Semantic texton
forests for image categorization and segmentation. In: IEEE Imran Maqsood graduated from the University of Engineering
CVPR, Anchorage (2008) and Technology Peshawar, with a B.Sc degree (Computer
[13] Yin, P., Criminisi, A., Winn, J.M., Essa, I.A.: Tree-based Engineering) in 2004 and M.Sc in 2006. He is pursuing his PhD
classifiers for bilayer video segmentation. In: CVPR (2007) degree. He is currently an Assistant Professor at the Department
[14] Bosh, A., Zisserman, A., Munoz, X.: Image classification of Computer Software Engineering, UET Mardan Campus,
using Random Forests and ferns. In: IEEE ICCV (2007) Peshawar, Pakistan.
[15] Apostolof, N., Zisserman, A.: Who are you? - real-time
person identification. In: BMVC (2007).
[16] Introduction to Decision Trees and Random Forests, Ned
Horning; American Museum of Natural History’s
[17] Breiman, L.: Random Forests. Machine. Learning. 45, 5–32
(2001). DOI 10.1023/A:1010933404324
[18] Yanjun Qi., “Random Forest for Bioinformatics”.
www.cs.cmu.edu/~qyj/papersA08/11-rfbook.pdf
[19] Yang, P., Hwa Yang, Y., Zhou, B., Zomaya, Y., et al.: “A
review of ensemble methods in bioinformatics”. Current
Bioinformatics 5(4), 296–308 (2010)
[20] “Comparison of Decision Tree methods for finding active
objects” Yongheng Zhao and Yanxia Zhang, National
Astronomical Observatories, CAS, 20A Datun Road,
Chaoyang District, Bejing 100012 China

Copyright (c) 2012 International Journal of Computer Science Issues. All Rights Reserved.

View publication stats