An attempt of finding an appropriate number of convolutional layers in cnns based on benchmarks of heterogeneous datasets

Vadim V. Romanuke

doi:10.2478/ecce-2018-0006

An attempt of finding an appropriate number of convolutional layers in cnns based on benchmarks of heterogeneous datasets

Authors

Vadim V. Romanuke Professor, Polish Naval Academy

DOI:

https://doi.org/10.2478/ecce-2018-0006

Keywords:

convolutional neural networks, convolutional layers, error rate, hyperparameters, performance

Abstract

An attempt of finding an appropriate number of convolutional layers in convolutional neural networks is made. The benchmark datasets are CIFAR-10, NORB and EEACL26, whose diversity and heterogeneousness must serve for a general applicability of a rule presumed to yield that number. The rule is drawn from the best performances of convolutional neural networks built with 2 to 12 convolutional layers. It is not an exact best number of convolutional layers but the result of a short process of trying a few versions of such numbers. For small images (like those in CIFAR-10), the initial number is 4. For datasets that have a few tens of image categories and more, initially setting five to eight convolutional layers is recommended depending on the complexity of the dataset. The fuzziness in the rule is not removable because of the required diversity and heterogeneousness

References

H. H. Aghdam and E. J. Heravi, Guide to Convolutional Neural Networks: A Practical Application to Traffic-Sign Detection and Classification. Cham, Switzerland: Springer, 2017. https://doi.org/10.1007/978-3-319-57550-6

A. Gibson and J. Patterson, Deep Learning: A Practitioner’s Approach. O’Reilly Media, 2017.

S. Srinivas, R. K. Sarvadevabhatla, K. R. Mopuri, N. Prabhu, S. S. S. Kruthiventi, and R. V. Babu, “Chapter 2 - An Introduction to Deep Convolutional Neural Nets for Computer Vision,” in Deep Learning for Medical Image Analysis, S. K. Zhou, H. Greenspan, and D. Shen, Eds. Academic Press, 2017, pp. 25-52. https://doi.org/10.1016/b978-0-12-810408-8.00003-1

V. Andrearczyk and P. F. Whelan, “Using Filter Banks in Convolutional Neural Networks for Texture Classification,” Pattern Recognition Letters, vol. 84, pp. 63-69, Dec. 2016. https://doi.org/10.1016/j.patrec.2016.08.016

Z. Liao and G. Carneiro, “A Deep Convolutional Neural Network Module that Promotes Competition of Multiple-Size Filters,” Pattern Recognition, vol. 71, pp. 94-105, 2017. https://doi.org/10.1016/j.patcog.2017.05.024

D. Ciresan, U. Meier, J. Masci, L. M. Gambardella, and J. Schmidhuber, “Flexible, High Performance Convolutional Neural Networks for Image Classification,” in Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, vol. 2, pp. 1237- 1242, 2011.

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification With Deep Convolutional Neural Networks,” Communications of the ACM, vol. 60, iss. 6, pp. 84-90, 2017. https://doi.org/10.1145/3065386

J. Mutch and D. G. Lowe, “Object Class Recognition and Localization Using Sparse Features With Limited Receptive Fields,” International Journal of Computer Vision, vol. 80, iss. 1, pp. 45-57, 2008. https://doi.org/10.1007/s11263-007-0118-0

V. V. Romanuke, “Appropriate Number and Allocation of ReLUs in Convolutional Neural Networks,” Research Bulletin of the National Technical University of Ukraine “Kyiv Polytechnic Institute”, no. 1, pp. 69-78, 2017. https://doi.org/10.20535/1810-0546.2017.1.88156

P. Date, J. A. Hendler, and C. D. Carothers, “Design Index for Deep Neural Networks,” Procedia Computer Science, vol. 88, pp. 131-138, 2016. https://doi.org/10.1016/j.procs.2016.07.416

K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” Computer Vision and Pattern Recognition, 2015.

V. V. Romanuke, “Boosting Ensembles of Heavy Two-Layer Perceptrons for Increasing Classification Accuracy in Recognizing Shifted-Turned- Scaled Flat Images With Binary Features,” Journal of Information and Organizational Sciences, vol. 39, no. 1, pp. 75-84, 2015.

V. V. Romanuke, “Two-Layer Perceptron for Classifying Flat Scaled- Turned-Shifted Objects by Additional Feature Distortions in Training,” Journal of Uncertain Systems, vol. 9, no. 4, pp. 286-305, 2015.

P. K. Rhee, E. Erdenee, S. D. Kyun, M. U. Ahmed, and S. Jin, “Active and Semi-Supervised Learning for Object Detection With Imperfect Data,” Cognitive Systems Research, vol. 45, pp. 109-123, 2017. https://doi.org/10.1016/j.cogsys.2017.05.006

P. Tang, H. Wang, and S. Kwong, “G-MS2F: GoogLeNet Based Multi- Stage Feature Fusion of Deep CNN for Scene Recognition,” Neurocomputing, vol. 225, pp. 188-197, 2017. https://doi.org/10.1016/j.neucom.2016.11.023

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going Deeper With Convolutions,” Computer Vision and Pattern Recognition, 2014.

V. V. Romanuke, “Classifying Scaled-Turned-Shifted Objects With Optimal Pixel-to-Scale-Turn-Shift Standard Deviations Ratio in Training 2-Layer Perceptron on Scaled-Turned-Shifted 4800-Featured Objects Under Normally Distributed Feature Distortion,” Electrical, Control and Communication Engineering, vol. 13, iss. 1, pp. 45-54, 2017. https://doi.org/10.1515/ecce-2017-0007

V. V. Romanuke, “Classification Error Percentage Decrement of Two- Layer Perceptron for Classifying Scaled Objects on the Pattern of Monochrome 60-by-80-Images of 26 Alphabet Letters by Training With Pixel-Distorted Scaled Images,” Scientific bulletin of Chernivtsi National University of Yuriy Fedkovych. Series: Computer systems and components, vol. 4, iss. 3, pp. 53-64, 2013.

M. Sun, Z. Song, X. Jiang, J. Pan, and Y. Pang, “Learning Pooling for Convolutional Neural Network,” Neurocomputing, vol. 224, pp. 96-104, 2017. https://doi.org/10.1016/j.neucom.2016.10.049

D. Scherer, A. Müller, and S. Behnke, “Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition,” in International Conference on Artificial Neural Networks (ICANN 2010), pp. 92-101, 2010. https://doi.org/10.1007/978-3-642-15825-4_10

S. Lai, L. Jin, and W. Yang, “Toward High-Performance Online HCCR: A CNN Approach With DropDistortion, Path Signature and Spatial Stochastic Max-Pooling,” Pattern Recognition Letters, vol. 89, pp. 60-66, 2017. https://doi.org/10.1016/j.patrec.2017.02.011

N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Dropout: A Simple Way to Prevent Neural Networks From Overfitting,” Journal of Machine Learning Research, vol. 15, pp. 1929-1958, 2014.

L. P. F. Garcia, A. C. P. L. F. de Carvalho, and A. C. Lorena, “Effect of Label Noise in the Complexity of Classification Problems,” Neurocomputing, vol. 160, pp. 108-119, 2015. https://doi.org/10.1016/j.neucom.2014.10.085.