Abstract
In decision tree learning attribute selection is usually based on greedy local splitting criterion. More extensive search quickly leads to intolerable time consumption. Moreover, it has been observed that lookahead cannot benefit prediction accuracy as much as one would hope. It has even been claimed that lookahead would be mostly harmful in decision tree learning.
We present a computationally efficient splitting algorithm for numerical domains, which, in many cases, leads to more accurate trees. The scheme is based on information gain and an efficient variant of lookahead. We consider the performance of the algorithm, on one hand, in view of the greediness of typical splitting criteria and, on the other hand, the possible pathology caused by oversearching in the hypothesis space. In empirical tests, our algorithm performs in a promising manner.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Blake, C.L., Merz, C.J.: UCI repository of machine learning databases, University of California, Department of Information and Computer Science, Irvine, CA (1998)
Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J.: Classification and Regression Trees. Wadsworth, Pacific Grove (1984)
Devroye, L.: Branching processes and their applications in the analysis of tree structures and tree algorithms. In: Habib, M., McDiarmid, C., Ramirez-Alfonsin, J., Reed, B. (eds.) Probabilistic Methods for Algorithmic Discrete Mathematics. Algorithms and Combinatorics, vol. 16, pp. 249–314. Springer, New York (1998)
Dong, M., Kothari, R.: Look-ahead based fuzzy decision tree induction. IEEE Trans. Fuzzy Syst. 9, 461–468 (2001)
Elomaa, T., Rousu, J.: Generalizing boundary points. In: Proc. Seventeenth National Conference on Artificial Intelligence, pp. 570–576. AAAI Press, Menlo Park (2000)
Esposito, F., Malerba, D., Semeraro, G.: A comparative analysis of methods for pruning decision trees. IEEE Trans. Pattern Anal. Mach. Intell. 19, 476–491 (1997)
Fulton, T., Kasif, S., Salzberg, S.: Efficient algorithms for finding multi-way splits for decision trees. In: Prieditis, A., Russell, S. (eds.) Proc. Twelfth International Conference on Machine Learning, pp. 244–251. Morgan Kaufmann, San Francisco (1995)
Karp, R.M., Pearl, J.: Searching for an optimal path in a tree with random costs. Artif. Intell. 21, 99–117 (1983)
McDiarmid, C., Provan, G.M.A.: An expected-cost analysis of backtracking and non-backtracking algorithms. In: Proc. Twelfth International Joint Conference on Artificial Intelligence, pp. 172–177. Morgan Kaufmann, San Mateo (1991)
Mingers, J.: An empirical comparison of pruning methods for decision tree induction. Mach. Learn. 4, 227–243 (1989)
Murphy, P., Pazzani, M.: Exploring the decision forest: An empirical investigation of Occam’s Razor in decision tree induction. J. Artif. Intell. Res. 1, 257–275 (1994)
Murthy, S., Salzberg, S.: Lookahead and pathology in decision tree induction. In: Proc. Fourteenth International Joint Conference on Artificial Intelligence, pp. 1025–1031. Morgan Kaufmann, San Francisco (1995)
Nau, D.S.: Decision quality as a function of search depth on game trees. J. ACM 30, 687–708 (1983)
Pearl, J.: Game tree pathology. Artif. Intell. 20, 427–453 (1983)
Quinlan, J.R.: Induction of decision trees. Mach. Learn. 1, 81–106 (1986)
Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo (1993)
Quinlan, J.R., Cameron-Jones, M.: Oversearching and layered search in empirical learning. In: Proc. Fourteenth International Joint Conference on Artificial Intelligence, pp. 1019–1024. Morgan Kaufmann, San Francisco (1995)
Ragavan, H., Rendell, L.: Lookahead feature construction for learning hard concepts. In: Proc. Tenth International Conference on Machine Learning, pp. 252–259. Morgan Kaufmann, San Francisco (1993)
Sarkar, U.K., Chakrabarti, P.P., Ghose, S., DeSarkar, S.C.: Improving greedy algorithms by lookahead-search. J. Alg. 16, 1–23 (1994)
Shepherd, B., Piper, J., Rutovitz, D.: Comparison of ACLS and classical linear methods in a biological application. In: Hayes, J.E., Michie, D., Richards, J. (eds.) Machine Intelligence. Logic and the Acquisition of Knowledge, vol. 11, pp. 423–434. Oxford University Press, Oxford (1988)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Elomaa, T., Malinen, T. (2003). On Lookahead Heuristics in Decision Tree Learning. In: Zhong, N., Raś, Z.W., Tsumoto, S., Suzuki, E. (eds) Foundations of Intelligent Systems. ISMIS 2003. Lecture Notes in Computer Science(), vol 2871. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-39592-8_63
Download citation
DOI: https://doi.org/10.1007/978-3-540-39592-8_63
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-20256-1
Online ISBN: 978-3-540-39592-8
eBook Packages: Springer Book Archive