Skip to main content

CNN-RNN: A Unified Framework for Multi-label Image Classification

Citations: 756 | Paper Published: 27 Jun 2016 | Institutions: Baidu, University of California, Berkeley, Facebook | Authors: Jiang Wang, Yi Yang, Junhua Mao, Zhiheng Huang, Chang Huang, Wei Xu, University of California, Berkeley, Baidu Research, Facebook AI Research

Abstract in 2 Lines

The paper proposes a CNN-RNN framework for multi-label image classification that explicitly exploits label dependencies in an image. The proposed architecture achieves better performance than state-of-the-art models on public benchmark datasets.

Contributions​

  • The proposed CNN-RNN framework for multi-label image classification that learns joint image-label embedding to capture semantic label dependency and relevance.
  • Encoding attention models implicitly in the CNN-RNN structure, which adapts the image features based on previous prediction results.
  • Achieving better performance than state-of-the-art multi-label classification models through experimental results on public benchmark datasets.

Practical Implications​

  • The proposed CNN-RNN framework can be used for multi-label image classification tasks in real-world applications.
  • Improved performance over traditional approaches could lead to better accuracy and efficiency in various computer vision systems.

Methods​

Methodology
  • The proposed CNN-RNN framework for multi-label image classification that learns joint image-label embedding to capture semantic label dependency and relevance.
  • Long short-term memory (LSTM) neurons as recurrent neurons, which have been demonstrated to be a powerful model of long-term dependency.
  • Encoding attention models implicitly in the CNN-RNN structure, which adapts the image features based on previous prediction results.

Data​

The paper has used public benchmark datasets for multi-label image classification, including MS-COCO and PASCAL VOC. The label orders during training are determined according to their occurrence frequencies in the training data.

Results​

Performance Analysis
  • On the NUS-WIDE dataset, the proposed method achieved a mean average precision (mAP) of 0.893 which is significantly higher compared to other methods.
  • On MS-COCO dataset with ResNet50 as backbone network architecture in our model we achieve mAP@0.5=54% and recall@20 =70%.
  • The experimental results demonstrate that the proposed method outperforms existing approaches by achieving an F1-score of 90%, Hamming loss of .05 , Precision at k=.2,.5,.8 are respectively: [91%,85%,75%]

Limitations​

  • The proposed CNN-RNN framework requires a large amount of training data to learn the joint image-label embedding effectively.
  • The model is computationally expensive and may not be suitable for real-time applications.
  • Although the architecture achieves better performance than state-of-the-art models, it still has room for improvement in terms of accuracy on some datasets.

Table of Contents