[go: up one dir, main page]

Skip to main content

On the Statistical Comparison of Inductive Learning Methods

  • Chapter
Learning from Data

Part of the book series: Lecture Notes in Statistics ((LNS,volume 112))

Abstract

Experimental comparisons between statistical and machine learning methods appear with increasing frequency in the literature. However, there does not seem to be a consensus on how such a comparison is performed in a methodologically sound way. Especially the effect of testing multiple hypotheses on the probability of producing a ”false alarm” is often ignored.

We transfer multiple comparison procedures from the statistical literature to the type of study discussed in this paper. These testing procedures take the number of tests performed into account, thereby controlling the probability of generating ”false alarms”. The multiple comparison procedures selected are illustrated on well-know regression and classification data sets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. M.J. Crowder and D.J. Hand. Analysis of repeated measures. Chapman 0000 Hall, London, 1990.

    MATH  Google Scholar 

  2. P. Cheeseman and R.W. Oldford, editors. Selecting models from data: AI and statistics IV. Lecture notes in statistics nr. 89. Springer-Verlag, New York, 1994.

    Google Scholar 

  3. F. Diebold and R. Mariano. Comparing predictive accuracy. Technical re- port, University of Pennsylvania, 1994.

    Google Scholar 

  4. Olive Jean Dunn. Multiple comparisons among means. Journal of the Amer- ican Statistical Association, 56: 52–64, 1961.

    Article  MATH  Google Scholar 

  5. R. A. Fisher. The design of experiments. Olivier & Boyd, Edinburgh, 1935.

    Google Scholar 

  6. D. Fletcher and E. Goss. Forecasting with neural networks: an application using bankruptcy data. Information éf Management, 24: 159–167, 1993.

    Article  Google Scholar 

  7. D. J. Hand, editor. Artificial intelligence frontiers in statistics: AI and statis- tics III. Chapman Si Hall, London, 1993.

    Google Scholar 

  8. W. Hays. Statistics. Holt, Rinehart and Winston, Inc, Fort Worth, 1988.

    Google Scholar 

  9. Y. Hochberg and A. Tamhane. Multiple comparison procedures. Wiley 0000 Sons, New York, 1987.

    Book  MATH  Google Scholar 

  10. J. Kim, H. Weistroffer, and R. Redmond. Expert systems for bond rating: a comparative analysis of statistical, rule-based and neural network systems. Expert Systems, 10: 167–171, 1993.

    Article  Google Scholar 

  11. L. Marascuilo and M. McSweeney. Nonparametric and distribution free methods for the social sciences. Brooks/Cole Publishing Company, Monterey, CA, 1977.

    Google Scholar 

  12. D. Michie, D.J. Spiegelhalter, and C.C. Taylor, editors. Machine learning, neural and statistical classification. Ellis Horwood, New York, 1994.

    MATH  Google Scholar 

  13. Lutz Prechelt. A quantitative study of neural network learning algorithm evaluation practices. In Proc Intl. Conf. on Artificial Neural Networks, Cambridge, UK, June 26–28, 1995.

    Google Scholar 

  14. A. Refenes, M. Azema-Barac, L. Chen, and S. Karoussos. Currency exchange rate prediction and neural network design strategies. Neural Computing & Applications, 1: 46–58, 1993.

    Article  Google Scholar 

  15. B.D. Ripley. Flexible non-linear approaches to classification. In V. Cherkassky, J.H. Friedman, and H. C, editors, From Statistics to Neural Networks: Theory and Pattern Recognition Applications. Springer-Verlag, 1993.

    Google Scholar 

  16. Z. Tang, C. de Almeida, and P. Fishwick. Time series forecasing using neural networks vs. box-jenkins methodology. Simulation, 57: 303–310, 1991.

    Article  Google Scholar 

  17. Kar Yan Tam and Melody Y. Kiang. Managerial applications of neural net- works: the case of bank failure predictions. Management science,38(7):926–947, 1992.

    Google Scholar 

  18. B. Winer, D. Brown, and K. Michels. Statistical principles in experimental design. McGraw-Hill, New York, 1991.

    Google Scholar 

  19. A. Weigend and N. Gershenfield. Time series prediction: forecasting the future and understanding the past. Addison-Wesley, Reading, 1994.

    Google Scholar 

  20. WHR90] A. Weigend, A. Huberman, and D. Rumelhart. Predicting the future: a connectionist approach. International Journal of Neural Systems,1(3):193209, 1990.

    Google Scholar 

  21. P.H. Westfall and S.S. Young. Resampling-Based Multiple Testing. John Wiley & Sons, New York, 1993.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 1996 Springer-Verlag New York, Inc.

About this chapter

Cite this chapter

Feelders, A., Verkooijen, W. (1996). On the Statistical Comparison of Inductive Learning Methods. In: Fisher, D., Lenz, HJ. (eds) Learning from Data. Lecture Notes in Statistics, vol 112. Springer, New York, NY. https://doi.org/10.1007/978-1-4612-2404-4_26

Download citation

  • DOI: https://doi.org/10.1007/978-1-4612-2404-4_26

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-0-387-94736-5

  • Online ISBN: 978-1-4612-2404-4

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics