Abstract
Experimental comparisons between statistical and machine learning methods appear with increasing frequency in the literature. However, there does not seem to be a consensus on how such a comparison is performed in a methodologically sound way. Especially the effect of testing multiple hypotheses on the probability of producing a ”false alarm” is often ignored.
We transfer multiple comparison procedures from the statistical literature to the type of study discussed in this paper. These testing procedures take the number of tests performed into account, thereby controlling the probability of generating ”false alarms”. The multiple comparison procedures selected are illustrated on well-know regression and classification data sets.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
M.J. Crowder and D.J. Hand. Analysis of repeated measures. Chapman 0000 Hall, London, 1990.
P. Cheeseman and R.W. Oldford, editors. Selecting models from data: AI and statistics IV. Lecture notes in statistics nr. 89. Springer-Verlag, New York, 1994.
F. Diebold and R. Mariano. Comparing predictive accuracy. Technical re- port, University of Pennsylvania, 1994.
Olive Jean Dunn. Multiple comparisons among means. Journal of the Amer- ican Statistical Association, 56: 52–64, 1961.
R. A. Fisher. The design of experiments. Olivier & Boyd, Edinburgh, 1935.
D. Fletcher and E. Goss. Forecasting with neural networks: an application using bankruptcy data. Information éf Management, 24: 159–167, 1993.
D. J. Hand, editor. Artificial intelligence frontiers in statistics: AI and statis- tics III. Chapman Si Hall, London, 1993.
W. Hays. Statistics. Holt, Rinehart and Winston, Inc, Fort Worth, 1988.
Y. Hochberg and A. Tamhane. Multiple comparison procedures. Wiley 0000 Sons, New York, 1987.
J. Kim, H. Weistroffer, and R. Redmond. Expert systems for bond rating: a comparative analysis of statistical, rule-based and neural network systems. Expert Systems, 10: 167–171, 1993.
L. Marascuilo and M. McSweeney. Nonparametric and distribution free methods for the social sciences. Brooks/Cole Publishing Company, Monterey, CA, 1977.
D. Michie, D.J. Spiegelhalter, and C.C. Taylor, editors. Machine learning, neural and statistical classification. Ellis Horwood, New York, 1994.
Lutz Prechelt. A quantitative study of neural network learning algorithm evaluation practices. In Proc Intl. Conf. on Artificial Neural Networks, Cambridge, UK, June 26–28, 1995.
A. Refenes, M. Azema-Barac, L. Chen, and S. Karoussos. Currency exchange rate prediction and neural network design strategies. Neural Computing & Applications, 1: 46–58, 1993.
B.D. Ripley. Flexible non-linear approaches to classification. In V. Cherkassky, J.H. Friedman, and H. C, editors, From Statistics to Neural Networks: Theory and Pattern Recognition Applications. Springer-Verlag, 1993.
Z. Tang, C. de Almeida, and P. Fishwick. Time series forecasing using neural networks vs. box-jenkins methodology. Simulation, 57: 303–310, 1991.
Kar Yan Tam and Melody Y. Kiang. Managerial applications of neural net- works: the case of bank failure predictions. Management science,38(7):926–947, 1992.
B. Winer, D. Brown, and K. Michels. Statistical principles in experimental design. McGraw-Hill, New York, 1991.
A. Weigend and N. Gershenfield. Time series prediction: forecasting the future and understanding the past. Addison-Wesley, Reading, 1994.
WHR90] A. Weigend, A. Huberman, and D. Rumelhart. Predicting the future: a connectionist approach. International Journal of Neural Systems,1(3):193209, 1990.
P.H. Westfall and S.S. Young. Resampling-Based Multiple Testing. John Wiley & Sons, New York, 1993.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1996 Springer-Verlag New York, Inc.
About this chapter
Cite this chapter
Feelders, A., Verkooijen, W. (1996). On the Statistical Comparison of Inductive Learning Methods. In: Fisher, D., Lenz, HJ. (eds) Learning from Data. Lecture Notes in Statistics, vol 112. Springer, New York, NY. https://doi.org/10.1007/978-1-4612-2404-4_26
Download citation
DOI: https://doi.org/10.1007/978-1-4612-2404-4_26
Publisher Name: Springer, New York, NY
Print ISBN: 978-0-387-94736-5
Online ISBN: 978-1-4612-2404-4
eBook Packages: Springer Book Archive