注意力机制作为一种解决资源分配的手段,用来解决资源过载问题。最开始仅应用在机器翻译、文本处理等领域,最近十年也开始被应用于图像处理领域中,考虑将其与医学领域相结合,用注意力解决医学相关问题,有利于提高我国医疗服务行业的发展。文章介绍了注意力机制并总结了注意力机制在医学上的主要应用:医学图像检测与分割、医学图像分类、疾病预测、行为识别。在医学领域引入注意力机制,有利于提高医生的诊断效率,缩短就诊时间,也为虚拟现实技术的进一步发展带来了可能。
As a means of resource allocation, attention mechanism is used to solve the problem of resource overload. At first, it was only used in machine translation, text processing and other fields. In the last decade, it has also been used in the field of image processing. It is considered to combine it with the medical field to solve medical related problems with attention, which is conducive to improving the development of China’s medical service industry. This paper introduces the attention mechanism and summarizes its main applications in medicine: medical image detection and segmentation, medical image classification, disease prediction, and behavior recognition. The introduction of attention mechanism in the medical field is conducive to improving the diagnostic efficiency of doctors, shortening the time for medical treatment, and also brings possibilities for the further development of virtual reality technology.
As a means of resource allocation, attention mechanism is used to solve the problem of resource overload. At first, it was only used in machine translation, text processing and other fields. In the last decade, it has also been used in the field of image processing. It is considered to combine it with the medical field to solve medical related problems with attention, which is conducive to improving the development of China’s medical service industry. This paper introduces the attention mechanism and summarizes its main applications in medicine: medical image detection and segmentation, medical image classification, disease prediction, and behavior recognition. The introduction of attention mechanism in the medical field is conducive to improving the diagnostic efficiency of doctors, shortening the time for medical treatment, and also brings possibilities for the further development of virtual reality technology.
Keywords:Attention Mechanism, Image Detection, Image Segmentation, Behavior Recognition, Medical Science
盛诗梦,丁 皓,徐欣茹,丁思吉,夏冬阳. 注意力机制在医学上的应用综述A Review of the Application of Attention Mechanism in Medicine[J]. 软件工程与应用, 2022, 11(06): 1223-1232. https://doi.org/10.12677/SEA.2022.116124
参考文献References毛健, 赵红东, 姚婧婧. 人工神经网络的发展及应用[J]. 电子设计工程, 2011, 19(24): 62-65.王海星, 田雪晴, 游茂, 等. 人工智能在医疗领域应用现状、问题及建议[J]. 卫生软科学, 2018, 32(5): 3-5+9.葛雪松. 影响注意力的十大因素[J]. 教学研究, 2006, 29(6): 555-557.Mnih, V., Heess, N. and Graves, A. (2014) Recurrent Models of Visual Attention. Proceedings of the 27th International Conference on Neural Information Processing Systems, Volume 2, 2204-2212.Rensink, R.A. (2000) The Dynamic Representation of Scenes. Visual Cognition, 7, 17-42.
<br>https://doi.org/10.1080/135062800394667Liu, G. and Guo, J. (2019) Bidirectional LSTM with Attention Mechanism and Convolutional Layer for Text Classification. Neurocomputing, 337, 325-338. <br>https://doi.org/10.1016/j.neucom.2019.01.078Li, Y., Zeng, J., Shan, S., et al. (2019) Occlusion Aware Facial Expression Recognition Using CNN with Attention Mechanism. IEEE Transactions on Image Processing, 28, 2439-2450. <br>https://doi.org/10.1109/TIP.2018.2886767Chen, Y., Liu, L., Phonevilay, V., et al. (2021) Image Super-Resolution Reconstruction Based on Feature Map Attention Mechanism. Applied Intelligence, 51, 4367-4380. <br>https://doi.org/10.1007/s10489-020-02116-1Niu, Z., Zhong, G. and Yu, H. (2021) A Review on the Attention Mechanism of Deep Learning. Neurocomputing, 452, 48-62. <br>https://doi.org/10.1016/j.neucom.2021.03.091Xu, K., Ba, J., Kiros, R., et al. (2015) Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. Proceedings of the International Conference on Machine Learning, Lille, 7-9 July 2015, 2048-2057.Bahdanau, D., Cho, K. and Bengio, Y. (2014) Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations 2015, San Diego, 7-9 May 2015, 1-15.Luong, M.-T., Pham, H. and Manning, C.D. (2015) Effective Approaches to Attention-Based Neural Machine Translation. <br>https://doi.org/10.18653/v1/D15-1166Chu, Y., Yue, X., Yu, L., et al. (2020) Automatic Image Captioning Based on ResNet50 and LSTM with Soft Attention. Wireless Communications and Mobile Computing, 2020, Article ID: 8909458.
<br>https://doi.org/10.1155/2020/8909458Wang, F., Jiang, M., Qian, C., et al. (2017) Residual Attention Network for Image Classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, 21-26 July 2017, 6450-6458.
<br>https://doi.org/10.1109/CVPR.2017.683Shen, T., Zhou, T., Long, G., et al. (2018) Reinforced Self-Attention Network: A Hybrid of Hard and Soft Attention for Sequence Modeling. Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, 13-19 July 2018, 4345-4352. <br>https://doi.org/10.24963/ijcai.2018/604Lu, J., Yang, J., Batra, D., et al. (2016) Hierarchical Question-Image Co-Attention for Visual Question Answering. Proceedings of the 30th International Conference on Neural Information Processing Systems, Barcelona, 5-10 December 2016, 289-297.Li, X., Song, K., Feng, S., et al. (2018) A Co-Attention Neural Network Model for Emotion Cause Analysis with Emotional Context Awareness. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, October-November 2018, 4752-4757. <br>https://doi.org/10.18653/v1/D18-1506Gao, J., Wang, X., Wang, Y., et al. (2019) CAMP: Co-Attention Memory Networks for Diagnosis Prediction in Healthcare. Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM), Beijing, 8-11 November 2019, 1036-1041. <br>https://doi.org/10.1109/ICDM.2019.00120Gong, X., Liu, X., Li, Y., et al. (2020) A Novel Co-Attention Computation Block for Deep Learning Based Image Co-Segmentation. Image and Vision Computing, 101, Article ID: 103973.
<br>https://doi.org/10.1016/j.imavis.2020.103973Yu, Z., Yu, J., Cui, Y., et al. (2019) Deep Modular Co-Attention Networks for Visual Question Answering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, 15-20 June 2019, 6274-6283. <br>https://doi.org/10.1109/CVPR.2019.00644Liu, Y., Zhang, X., Zhang, Q., et al. (2021) Dual Self-Attention with Co-Attention Networks for Visual Question Answering. Pattern Recognition, 117, Article ID: 107956. <br>https://doi.org/10.1016/j.patcog.2021.107956Zhang, S., Chen, M., Chen, J., et al. (2021) Multimodal Feature-Wise Co-Attention Method for Visual Question Answering. Information Fusion, 73, 1-10. <br>https://doi.org/10.1016/j.inffus.2021.02.022Vaswani, A., Shazeer, N., Parmar, N., et al. (2017) Attention Is All You Need. 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, 4-9 December 2017, 1-11.吴军, 欧阳艾嘉, 张琳. 基于多头注意力机制的磷酸化位点预测模型[J]. 广西师范大学学报(自然科学版), 2022, 40(3): 161-171.赵丹丹, 黄德根, 孟佳娜, 等. 多头注意力与字词融合的中文命名实体识别[J]. 计算机工程与应用, 2022, 58(7): 142-149.石磊, 王毅, 成颖, 等. 自然语言处理中的注意力机制研究综述[J]. 数据分析与知识发现, 2020, 4(5): 1-14.Yang, Z., Yang, D., Dyer, C., et al. (2016) Hierarchical Attention Networks for Document Classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego, June 2016, 1480-1489. <br>https://doi.org/10.18653/v1/N16-1174Cheng, J., Zhao, S., Zhang, J., et al. (2017) Aspect-Level Sentiment Classification with Heat (Hierarchical Attention) Network. Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, Singapore, 6-10 November 2017, 97-106. <br>https://doi.org/10.1145/3132847.3133037Ying, H., Zhuang, F., Zhang, F., et al. (2018) Sequential Recommender System Based on Hierarchical Attention Network. Proceedings of the 27th International Joint Conference on Artificial Intelligence, Stockholm, 13-19 July 2018, 3926-3932.Miculicich, L., Ram, D., Pappas, N., et al. (2018) Document-Level Neural Machine Translation with Hierarchical Attention Networks. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, October-November 2018, 2947-2954. <br>https://doi.org/10.18653/v1/D18-1325Wang, W., Yan, M. and Wu, C. (2018) Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Volume 1, 1705-1714. <br>https://doi.org/10.18653/v1/P18-1158Wang, Y., Wang, S., Tang, J., et al. (2016) Hierarchical Attention Network for Action Recognition in Videos. The International Conference on Computer Vision (ICCV), 61-70. <br>https://arxiv.org/abs/1607.06416Wang, B., Liu, K. and Zhao, J. (2016) Inner Attention Based Recurrent Neural Networks for Answer Selection. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Volume 1, 1288-1297.
<br>https://doi.org/10.18653/v1/P16-1122Ji, J., Luo, Y., Sun, X., et al. (2021) Improving Image Captioning by Leveraging Intra- and Inter-Layer Global Representation in Transformer Network. Proceedings of the AAAI Conference on Artificial Intelligence, 35, 1655-1663.
<br>https://doi.org/10.1609/aaai.v35i2.16258Mirsamadi, S., Barsoum, E. and Zhang, C. (2017) Automatic Speech Emotion Recognition Using Recurrent Neural Networks with Local Attention. Proceedings of the 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), New Orleans, 5-9 March 2017, 2227-2231.
<br>https://doi.org/10.1109/ICASSP.2017.7952552Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2010) An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.金亚荣. 基于深度学习的CT图像病灶检测和识别[D]: [硕士学位论文]. 北京: 北京工业大学, 2020.周涛, 董雅丽, 刘珊, 等. 用于肺部肿瘤图像分割的跨模态多编码混合注意力U-Net[J]. 光子学报, 2022, 51(4): 368-384.梁美彦, 张倩楠, 任竹云, 等. 基于注意力机制的结肠癌病理学图像识别研究[J]. 测试技术学报, 2022, 36(2): 93-100.郝如茜, 王祥舟, 张静, 等. 采用注意力机制的显微图像智能检测方法[J]. 光电工程, 2022, 49(3): 44-54.侯海燕, 符志鹏. 视觉注意力焦点显著性模型及其医学应用[J]. 计算机工程与应用, 2017, 53(10): 213-217.张文秀, 朱振才, 张永合, 等. 基于残差块和注意力机制的细胞图像分割方法[J]. 光学学报, 2020, 40(17): 70-77.吴倩倩, 周蕾蕾, 陆小妍, 等. 基于多头自注意力机制与U-Net的增强CT图像肾脏小肿瘤自动分割研究[J]. 中国医学装备, 2022, 19(2): 27-31.闫超, 孙占全, 田恩刚, 等. 结合多分支结构与自注意力机制的医学图像分割模型及其应用[J]. 小型微型计算机系统, 2021, 42(11): 2375-2381.张建鹏. 基于深度学习的医学图像分类技术研究[D]: [硕士学位论文]. 西安: 西北工业大学, 2019.Zou, Y., Zhang, J., Huang, S., et al. (2022) Breast Cancer Histopathological Image Classification Using Attention High-Order Deep Network. International Journal of Imaging Systems and Technology, 32, 266-279.
<br>https://doi.org/10.1002/ima.22628Zhao, Y., Wang, S., Ren, Y., et al. (2022) CRANet: A Comprehensive Residual Attention Network for Intracranial Aneurysm Image Classification. BMC Bioinformatics, 23, Article No. 322.
<br>https://doi.org/10.1186/s12859-022-04872-y李宇. 基于胶囊网络的脑肿瘤图像分类研究与实现[D]: [硕士学位论文]. 成都: 电子科技大学, 2021.Ma, C., Wang, H. and Hoi, S.C. (2019) Multi-Label Thoracic Disease Image Classification with Cross-Attention Networks. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Shenzhen, 13-17 October 2019, 730-738. <br>https://doi.org/10.1007/978-3-030-32226-7_81邢素霞, 鞠子涵, 刘子娇, 等. 视觉Transformer预训练模型的胸腔X线影像多标签分类[J/OL]. 中国图象图形学报, 2022. <br>http://www.cjig.cn/jig/ch/reader/view_abstract.aspx?flag=2&file_no=202203170000002&journal_id=jigSadafi, A., Makhro, A., Bogdanova, A., et al. (2020) Attention Based Multiple Instance Learning for Classification of Blood Cell Disorders. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, Lima, 4-8 October 2020, 246-256. <br>https://doi.org/10.1007/978-3-030-59722-1_24Pan, L., Wang, H., Wang, L., et al. (2022) Noise-Reducing Attention Cross Fusion Learning Transformer for Histological Image Classification of Osteosarcoma. Biomedical Signal Processing and Control, 77, Article ID: 103824.
<br>https://doi.org/10.1016/j.bspc.2022.103824Hajian, S., Bonchi, F. and Castillo, C. (2016) Algorithmic Bias. Proceedings of the 22nd ACM SIGKDD International Conference on K now ledge Discovery and Data Mining, San Francisco, 13-17 August 2016.葛伟. 基于注意力机制的疾病预测关键技术研究[D]: [硕士学位论文]. 济南: 山东大学, 2020.王江峰, 刘利军, 黄青松, 等. 基于胸部X射线影像的常见疾病预测方法研究[J]. 激光与光电子学进展, 2022, 59(18): 387-394.吴石远. 基于卷积和循环神经网络的房颤预测模型的研究[D]: [硕士学位论文]. 武汉: 武汉科技大学, 2021.王天罡. 基于深度学习的疾病预警及诊断[D]: [硕士学位论文]. 西安: 西安工程大学, 2021.Meng, L. and Li, R. (2021) An Attention-Enhanced Multi-Scale and Dual Sign Language Recognition Network Based on a Graph Convolution Network. Sensors, 21, 1120. <br>https://doi.org/10.3390/s21041120Saunders, B., Camgoz, N.C. and Bowden, R. (2021) Skeletal Graph Self-Attention: Embedding a Skeleton Inductive Bias into Sign Language Production.Huang, J., Zhou, W., Li, H., et al. (2019) Attention-Based 3D-CNNs for Large-Vocabulary Sign Language Recognition. IEEE Transactions on Circuits and Systems for Video Technology, 29, 2822-2832.
<br>https://doi.org/10.1109/TCSVT.2018.2870740Zuo, R. and Mak, B. (2022) C2SLR: Consistency-Enhanced Continuous Sign Language Recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, 18-24 June 2022, 5121-5130.
<br>https://doi.org/10.1109/CVPR52688.2022.00507李广鹏, 刘波, 李坤, 等. 一种基于机器学习的人脸情绪识别方法研究[J]. 计算机技术与发展, 2019, 29(5): 27-31.李小伟. 脑电, 眼动信息与学习注意力及抑郁的中文相关性研究[D]: [博士学位论文]. 兰州: 兰州大学, 2015.马伟娜, 朱蓓蓓, 谢宇. 孤独症儿童面部表情识别能力的眼动研究[J]. 应用心理学, 2015, 21(1): 76-88.李学聪, 战荫伟, 杨卓, 等. 基于标签分布学习的眼部情绪识别[J]. 计算机科学与应用, 2022, 12(4): 1213-1225.
<br>https://doi.org/10.12677/CSA.2022.124123刘天宝, 张凌涛, 于文涛, 等. 基于嵌入注意力机制层级LSTM的音视频情感识别[J]. 激光与光电子学进展, 2021, 58(2): 175-182.姜特, 陈志刚, 万永菁. 基于注意力机制的多任务3D CNN-BLSTM情感语音识别[J]. 华东理工大学学报(自然科学版), 2022, 48(4): 534-542.Wu, Z., Pan, S., Chen, F., et al. (2020) A Comprehensive Survey on Graph Neural Networks. IEEE Transactions on Neural Networks and Learning Systems, 32, 4-24. <br>https://doi.org/10.1109/TNNLS.2020.2978386李景聪, 潘伟健, 林镇远, 等. 采用多路图注意力网络的情绪脑电信号识别方法[J]. 智能系统学报, 2022, 17(3): 531-539.Wan, S., Pan, S., Yang, J., et al. (2021) Contrastive and Generative Graph Convolutional Networks for Graph-Based Semi-Supervised Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 2-9 February 2021, 10049-10057. <br>https://doi.org/10.1609/aaai.v35i11.17206.