Flexible, High Performance Convolutional Neural Networks for Image Classification

Citations: 1,124 | Paper Published: 16 Jul 2011 | Institutions: Dalle Molle Institute for Artificial Intelligence Research | Authors: Dan Ciresan, Ueli Meier, Jonathan Masci, Luca Maria Gambardella, Jürgen Schmidhuber

Abstract in 2 Lines

The paper presents a fast implementation of Convolutional Neural Network variants on GPUs, achieving state-of-the-art results in object classification and handwritten digit recognition benchmarks. The feature extractors are learned in a supervised way without being pre-wired or carefully designed.

Contributions

The contribution of this paper is the presentation of a fast implementation of Convolutional Neural Network variants on GPUs, which allows for training large CNNs within days instead of months. This enables investigating the influence of various structural parameters by exploring large parameter spaces and performing error analysis on repeated experiments to systematically test their impact on classification performance. The deep hierarchical architectures achieve state-of-the-art results in object classification (NORB, CIFAR10) and handwritten digit recognition (MNIST), with surprisingly rapid learning rates using simple back-propagation algorithms.

Practical Implications

• The fast implementation of Convolutional Neural Network variants on GPUs can lead to significant improvements in object classification and handwritten digit recognition benchmarks.
• The learned feature extractors without being pre-wired or carefully designed provide great flexibility for adapting the networks to any particular application.
• All structural CNN parameters such as input image size, number of hidden layers, number of maps per layer, kernel sizes, skipping factors and connection tables are adaptable which makes it easier for researchers/practitioners to use these models.

Methods

Methodology

• The authors present a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants.
• They use supervised learning to train the feature extractors without being pre-wired or carefully designed.
• Deep hierarchical architectures with simple back-propagation training achieve state-of-the-art results on object classification and handwritten digit recognition benchmarks (NORB, CIFAR10, MNIST).
• All structural CNN parameters such as input image size, number of hidden layers, number of maps per layer etc. can be adapted for different applications.

Data

• NORB: A 3D object recognition dataset with six classes and ten instances per class, where each instance is a pair of stereo images.
• CIFAR10: A natural image classification dataset consisting of 60,000 color images in 10 classes (6k/images).
• MNIST: A handwritten digit recognition dataset containing grayscale images representing digits from zero to nine.

All these datasets are widely used benchmarks for evaluating deep learning models' performance on various tasks such as object detection/recognition or handwriting recognition/classification problems.

Results

Performance Analysis

The paper presents a fast, fully parameterizable GPU implementation of Convolutional Neural Network variants that achieve state-of-the-art results on benchmark datasets for object classification (NORB, CIFAR10) and handwritten digit recognition (MNIST). The best network achieved test error rates of 0.35%, 2.53% and 19.51% on MNIST, NORB and CIFAR10 respectively using simple back-propagation algorithms with surprisingly rapid learning rates within just a few epochs in some cases

Limitations

• The authors do not provide a detailed analysis or comparison with other state-of-the-art methods, which makes it difficult to assess how much improvement their proposed method provides over existing techniques.
• They achieve good results on benchmark datasets, these may not be representative of real-world scenarios. Therefore, further evaluation is needed in more complex and diverse environments.