Abstract
This work analyzes the problem of selecting an adequate neural network architecture for a given function, comparing existing approaches and introducing a new one based on the use of the complexity of the function under analysis. Numerical simulations using a large set of Boolean functions are carried out and a comparative analysis of the results is done according to the architectures that the different techniques suggest and based on the generalization ability obtained in each case. The results show that a procedure that utilizes the complexity of the function can help to achieve almost optimal results despite the fact that some variability exists for the generalization ability of similar complexity classes of functions.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Arai M (1993) Bounds on the number of hidden units in binary-valued three-layer neural networks. Neural Netw 6(6): 855–860
Barron AR (1993) Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans Inform Theory 39(3): 930–945
Barron AR (1994) Approximation and estimation bounds for artificial neural networks. Mach Learn 14(1): 115–133
Baum EB, Haussler D (1990) What size net gives valid generalization?. Neural Comput 1(1): 151–160
Blumer A, Ehrenfeucht A, Haussler D, Warmuth MK (1989) Learnability and the Vapnik-Chervonenkis dimension. J ACM 36(4): 929–965
Camargo LS, Yoneyama T (2001) Specification of training sets and the number of hidden neurons for multilayer perceptrons. Neural Comput 13(12): 2673–2680
Demuth H , Beale M (1994) MATLAB neural networks toolbox—user’s guide version 4. The Math Works, USA
Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7: 1–30
Franco L (2006) Generalization ability of Boolean functions implemented in feedforward neural networks. Neurocomputing 70: 351–361
Franco L, Anthony M (2004) On a generalization complexity measure for Boolean functions. In: Proceedings of the 2004 IEEE international joint conference on neural networks. pp 973–978
Franco L, Anthony M (2006) The influence of oppositely classified examples on the generalization complexity of Boolean functions. IEEE Trans Neural Netw 17(3): 578–590
Franco L, Cannas SA (2000) Generalization and selection of examples in feedforward neural networks. Neural Comput 12(10): 2405–2426
Franco L, Cannas SA (2001) Generalization properties of modular networks, implementing the parity function. IEEE Trans Neural Netw 12: 1306–1313
Frean M (1990) The upstart algorithm, a method for constructing and training feedforward neural networks. Neural Comput 2(2): 198–209
Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32: 675–701
Hájek J, Šidák Z, Sen PK (1999) Theory of rank tests, 2nd ed. Academic Press, Orlando
Haykin S (1994) Neural networks (a comprehensive foundation). Prentice Hall, USA
Lawrence S, Giles CL, and Tsoi A (1996) What size neural network gives optimal generalization? Convergence properties of backpropagation. Technical Report UMIACS-TR-96-22 and CS-TR-3617, University of Maryland
LeCun Y, Denker J, Solla S, Howard RE, Jackel LD (1990) Optimal brain damage. In: Touretzky DS (eds) Advances in neural information processing systems II. Morgan Kaufman, San Mateo
Liang X (2006) Removal of hidden neurons in multilayer perceptrons by orthogonal projection and weight crosswise propagation. Neural Comput Appl 16(1): 57–68
Linial N, Mansour Y, Nisan N (1993) Constant depth circuits, Fourier transform and learnability. J ACM 40: 607–620
Liu Y, Starzyk JA, Zhu Z (2007) Optimizing number of hidden neurons in neural networks. In: AIAP’07 proceedings of the 25th conference on proceedings of the 25th IASTED international multi-conference. Anaheim, CA, USA, pp 121–126, ACTA Press
Masters T (1993) Practical neural network recipes in C++. Academic Press Professional, Inc., San Diego
Mayoraz E (1991) On the Power of Networks of Majority Functions’. In: IWANN ’91 Proceedings of the international workshop on artificial neural networks. Springer-Verlag, London, UK, pp 78–85
Mezard M, Nadal J-P (1989) Learning in feedforward layered networks (the tiling algorithm. J Phys A Math Gen 22(12): 2191–2203
Mirchandani G, Cao W (1989) On hidden nodes for neural nets. IEEE Trans Circuit Systems 36(5): 661–664
Moller MF (1993) Scaled conjugate gradient algorithm for fast supervised learning. Neural Netw 6(4): 525–533
Neuralware I (2001) The reference guide. http://www.neuralware.com
Piramuthu S, Shaw M, Gentry J (1994) A classification approach using multi-layered neural networks. Decis Support Syst 11: 509–525
Scarselli F, Tsoi AC (1998) Universal approximation using feedforward neural networks a survey of some existing methods, and some new results. Neural Netw 11(1): 15–37
Siu K-Y, Roychowdhury VP (1994) On optimal depth threshold circuits for multiplication and related problems. SIAM J Discret Math 7(2): 284–292
Vázquez EG, Yanez A, Galindo P, Pizarro J (2001) Repeated measures multiple comparison procedures applied to model selection in neural networks. In: IWANN ’01 proceedings of the 6th international work-conference on artificial and natural neural networks. Springer–Verlag, London, UK, pp 88–95
Šmieja FJ (1993) Neural network constructive algorithms (trading generalization for learning efficiency?. Circuits Syst Signal Process 12(2): 331–374
Wang J, Yi Z, Zurada JM, Lu B-L, Yin H (eds) (2006) Advances in neural networks—ISNN 2006, Third international symposium on neural networks, Chengdu, China, May 28–June 1, 2006, Proceedings, Part I, Vol. 3971 of Lecture notes in computer science. Springer
Weigend AS, Rumelhart DE, Huberman BA (1990) Generalization by weight-elimination with application to forecasting. In: NIPS. pp 875–882
Wilcoxon F (1945) Individual comparisons by ranking methods. Biometrics 1: 80–83
Witten IH, Frank E (2005) Data mining, practical machine learning tools and techniques. Morgan Kaufmann, San Mateo
Yuan HC, Xiong FL, Huai XY (2003) A method for estimating the number of hidden neurons in feed-forward neural networks based on information entropy. Comput Electr Agric 40: 57–64
Zhang Z, Ma X, Yang Y (2003) Bounds on the number of hidden neurons in three-layer binary neural networks. Neural Netw 16(7): 995–1002
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gómez, I., Franco, L. & Jerez, J.M. Neural Network Architecture Selection: Can Function Complexity Help?. Neural Process Lett 30, 71–87 (2009). https://doi.org/10.1007/s11063-009-9108-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-009-9108-2