多標籤影像分類是一項具挑戰性的工作，目標是同時找出不同大小的物件並且辨識正確的標籤。然而，常見的做法是使用整張影像抽取特徵，較小物體的資訊可能會因此被稀釋，或是成為雜訊，造成辨識困難。在先前的研究裡顯示，使用關注機制和標籤關係能各自增進特徵擷取和共生關係，以取得更強健的資訊，幫助多標籤分類任務。
在本工作中，我們使用Transformer之架構，將視覺區域特徵關注至全域特徵，同時考慮標籤之間的共生關係，最後將加權後之新特徵產生出一動態的語義分類器，在語義空間內分類得出預測標籤。在實驗中，顯示我們的模型可達到很好的成效。

Multi-label classification is a challenge task since we must identify many kinds of objects in different scales. While using global features of an image may discard small object information, many researches have shown that an attention mechanism improves feature extraction and that label relations reveal label co-occurrence, both of which benefit a multi-label classification task.
In this work, we extract attended features from one image by Transformer and simultaneously consider labels’ co-occurrence. Then, we use the attended features to generate a classifier applied on the semantic space to predict the labels. Experiments validate the proposed method.

[1]M. Yeh, Y. Li, “Multilabel Deep Visual-Semantic Embedding”, In TPAMI, 2019
[2]Z. M. Chen, X. S. Wei, P. Wang and Y. Guo, “Multi-label image recognition with graph convolutional networks”, In CVPR, 2019.
[3]V. O. Yazici, A. Gonzalez-Garcia, A. Ramisa, B. Twardowski, and J. V. D. Weijer, “Orderless recurrent models for multi-label classification”, In CVPR, 2020.
[4]J. Zhang, Q. Wu, C. Shen, J. Zhang, and J. Lu, “Multilabel image classification with regional latent semantic dependencies”, In IEEE Transactions on Multimedia 20(10): 2801-2813, 2018.
[5]F. Zhu, H. Li, W. Ouyang, N. Yu, and X. Wang, “Learning spatial regularization with image-level supervisions for multi-label image classification”, In CVPR, 2017.
[6]T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks”, In arXiv:1609.02907, 2016.
[7]K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition”, In CVPR, 2016.
[8]M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. “The Pascal Visual Object Classes (VOC) Challenge”. In IJCV, pp. 303–338, 2010.
[9]T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and C. L. Zitnick. “Microsoft COCO: Common objects in context”. In ECCV. 2014.
[10]T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, and Y. Zheng, “Nuswide: A real-world web image database from national university of singapore,” In Proceedings of the ACM international conference on image and video retrieval, 2009.
[11]C.-J. Wang, T.-H. Wang, H.-W. Yang, B.-S. Chang, and M.-F. Tsai, “ICE: Item concept embedding via textual information,” In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 85–94, 2017.
[12]Y. Gong, Y. Jia, T. Leung, A. Toshev, and S. Ioffe, “Deep Convolutional Ranking for Multilabel Image Annotation”, In arXiv:1312.4894, 2013.
[13]F. Liu, T. Xiang, T. Hospedales, T. M. Yang, and C. Sun, “Semantic Regularisation for Recurrent Image Annotation”, In CVPR, 2017.
[14]R. You, Z. Guo, L. Cui, X. Long, Y. Bao, and S. Wen, “Cross-modality attention with semantic graph embedding for multi-label classification”, In AAAI, 2020.
[15]A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean,T. Mikolov, “DeViSE: A deep visual-semantic embedding model”, In NIPS, 2013.
[16]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, I. Polosukhin, et al, “Attention is all you need”, In NeurIPS, 2017.
[17]A. SharifRazavian, H. Azizpour, J. Sullivan and S. Carlsson, “CNN features off-the-shelf: an astounding baseline for recognition”, In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 806-813, 2014.
[18]J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. “Imagenet: A large-scale hierarchical image database”, In CVPR, 2009.
[19]K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition”, In ICLR, 2015.
[20]J. Wang, Y. Yang, J. Mao, Z. Huang, C. Huang, and W. Xu, “CNN-RNN: A unified framework for multi-label image classification”, In CVPR, 2016.
[21]D. Huynh and E. Elhamifar, “Fine-grained generalized zero-shot learning via dense attribute-based attention”, In CVPR, 2020.
[22]Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, “Transductive multi-view zero-shot learning”, In TPAMI, 2015.
[23]T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space”, In ICLR, 2013.
[24]A. Vaswani, P. Ramachandran, A. Srinivas, N. Parmar, B. Hechtman and J. Shlens, “Scaling Local Self-Attention for Parameter Efficient Visual Backbones”, In CVPR, 2021.
[25]K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. R. Salakhutdinov, R. S. Zemel, and Y. Bengio, “Show, attend and tell: Neural image caption generation with visual attention”, In ICML, 2015.
[26]S. Narayan, A. Gupta, S. Khan, F. S. Khan, L. Shao, and M. Shah, “Discriminative Region-based Multi-Label Zero-Shot Learning”, arXiv preprint arXiv:2108.09301, 2021.
[27]S. Liu, L. Zhang, X. Yang, H. Su, and J. Zhu, “Query2Label: A Simple Transformer Way to Multi-Label Classification”, arXiv preprint arXiv:2107.10834, 2021.
[28]N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers”, In ECCV, 2021.
[29]A. Srinivas, T. Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani, “Bottleneck transformers for visual recognition”, In CVPR, 2021.
[30]A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, and J. Uszkoreit, “An image is worth 16x16 words: Transformers for image recognition at scale”, In ICLR, 2021.
[31]T.Chen, M. Xu, X. Hui, H. Wu, and L. Lin, “Learning semantic-specific graph representation for multi-label image recognition”, In ICCV, 2019.