像素分割联合深度双分支模型辅助诊断新冠CT图像

杜臻宇; 帕力旦·吐尔逊; 范迎迎; 许春陶; 钱育蓉

doi:10.19304/J.ISSN1000-7180.2022.0374

像素分割联合深度双分支模型辅助诊断新冠CT图像

doi: 10.19304/J.ISSN1000-7180.2022.0374

杜臻宇^{1, 2, 3},
帕力旦·吐尔逊^4, ,,
范迎迎^{1, 2, 3},
许春陶^{1, 2, 3},
钱育蓉^{1, 2, 3}

1.
新疆大学软件学院, 新疆乌鲁木齐 830000
2.
新疆维吾尔自治区信号检测与处理重点实验室, 新疆乌鲁木齐 830046
3.
新疆大学软件工程重点实验室, 新疆乌鲁木齐 830000
4.
新疆师范大学, 新疆乌鲁木齐 830000

基金项目: 国家自然科学基金资助项目（61966035）；自治区科技厅国际合作项目（2020E01023）；国家自然科学基金联合基金——重点项目（U1803261）

详细信息

作者简介:
杜臻宇：男,（1999-）,硕士研究生.研究方向为计算机视觉、医学图像处理

通讯作者:
女（维吾尔族）,（1970-）,博士,副教授. 研究方向为图像处理. E-mail：pldtrs@xjnu.edu.cn

中图分类号: TP183
计量
- 文章访问数: 19
- HTML全文浏览量: 15
- PDF下载量: 0
- 被引次数: 0
出版历程
- 收稿日期: 2022-06-23
- 修回日期: 2022-10-08

Pixel segmentation combined with a depth dual-branch model assists in the diagnosis of COVID-19 CT images

DU Zhenyu^{1, 2, 3},
Palladium Turson^{4
, ,},
FAN Yingying^{1, 2, 3},
XU Chuntao^{1, 2, 3},
QIAN Yurong^{1, 2, 3}

1.
School of Software, Xinjiang University, Urumqi 830000, Xinjiang, China
2.
Key Laboratory of Signal Detection and Processing, Xinjiang Uygur Autonomous Region, Urumqi 830046, Xinjiang, China
3.
Key Laboratory of Software Engineering, Xinjiang University, Urumqi 830000, Xinjiang, China
4.
Xinjiang Normal University, Urumqi 830000, Xinjiang, China

摘要

摘要:
基于医学肺部影像开发智能诊断新冠肺炎的深度学习方法能够减轻大量医护人员的工作,且能够提供可靠的准确性,然而深度学习方法的高准确性通常依赖于数据样本的质量. 在自然界存在的医学图像数据的来源和处理过程并不单一,数据样本差异性较大和质量不佳会增大深度学习模型提取关键特征的难度,有效的数据预处理和合适的模型设计十分关键. 基于肺部CT图像,本论文提出一种像素分割联合双分支模型ReSWNet辅助诊断新冠肺炎感染. 该方法首先训练像素分割模型进行分割预处理,实现肺部CT图像无关背景的剔除,然后通过结合了卷积神经网络和自注意力模型优缺点的双分支模型进行肺炎诊断. 通过在COVID-CT数据集上对该方法进行验证表明,在诊断准确率、召回率和F1分数等性能指标方面,该方法较基线模型分别提高了8.6%、16.05以及7.71%,最后采用可视化结果热力图为诊断提供了可解释性.
- 新冠 /
- 像素分割 /
- 医学图像分类 /
- 胸部CT
Abstract:
Developing deep learning methods for intelligently diagnosing COVID-19 based on medical lung imaging can ease the work of a large number of healthcare workers and provide reliable accuracy, but the high accuracy of deep learning methods often depends on the quality of data samples. The source and processing process of medical image data that exists in nature is not single, and the large difference and poor quality of data samples will increase the difficulty of deep learning models to extract key features, and effective data pre-processing and appropriate model design are critical. Based on lung CT images, this paper proposes a pixel-segmentation combined dual-branching model ReSWNet to assist in the diagnosis of COVID-19 infection. This method first trains the pixel segmentation model for segmentation preprocessing to achieve the rejection of the irrelevant background of lung CT images, and then performs pneumonia diagnosis by combining the advantages and disadvantages of convolutional neural network and self-attention model. The method was validated on the COVID-CT dataset and showed that the method improved by 8.6%, 16.05 and 7.71% compared with the baseline model in terms of diagnostic accuracy, recall rate and F1 score, respectively, and finally the visualization of the results heat map provided interpretability for diagnosis.
- COVID /
- Pixel segmentation /
- Medical image classification /
- CT chest

HTML全文

图 1 2019冠状病毒疾病CT扫描片示例

Figure 1. Examples of CT scans for COVID-19

下载: 全尺寸图片幻灯片

图 2 总体结构图

Figure 2. The overall structure of model

下载: 全尺寸图片幻灯片

图 3 通过Resnet50网络的CT图像的热力图示例

Figure 3. Example of heatmap of CT images through Resnet50 network

下载: 全尺寸图片幻灯片

图 4 CT图像分割

Figure 4. CT image segmentation

下载: 全尺寸图片幻灯片

图 5 残差连接

Figure 5. Residual connection

下载: 全尺寸图片幻灯片

图 6 Swim Transformer块

Figure 6. Swim Transformer block

下载: 全尺寸图片幻灯片

图 7 滑动窗口

Figure 7. Shifted Windows

下载: 全尺寸图片幻灯片

图 8 COVID-19 CT scans数据集的分割样本示例

Figure 8. Example of segmented samples of covid-19 CT scans dataset

下载: 全尺寸图片幻灯片

图 9 部分图像使用Resnet50和ReSWNet模型得到的热力图

Figure 9. Partial image resulting heat map

下载: 全尺寸图片幻灯片

表 1 COVID-CT数据集划分

Table 1. COVID-CT Dataset split

Dataset	Train	Test	Label
COVID-CT	251	98	Covid
COVID-CT	292	105	Non-Covid

下载: 导出CSV

表 2 模型改进以及数据处理对实验结果的影响

Table 2. Model improvements and the impact of data processing on experimental results

Model	Original Image	Segmented Image	Acc	Pre	Recall	F1-score
Resnet50	√		84.73	88.22	79.18	83.32
Resnet50		√	91.12	91.84	90.25	91.01
Swim Transformer	√		79.47	81.86	83.91	82.10
ReSWNet(ours)	√		85.32	86.32	79.39	83.92
ReSWNet(ours)	√	√	93.30	93.81	92.25	93.01

下载: 导出CSV

表 3 交换ReSWNet模型两分支的输入图像对模型性能的影响

Table 3. The effect of exchanging input images from two branches of the ReSWNet model on model performance

Model	Original Image	Segmented Image	Acc	Pre	Recall	F1-score
Resnet50	√		84.23	86.67	79.59	82.98
Swim Transformer		√	84.23	86.67	79.59	82.98
Resnet50	√		93.30	93.81	92.25	93.01
Swim Transformer		√	93.30	93.81	92.25	93.01
Resnet50		√	92.12	92.64	91.55	91.84
Swim Transformer		√	92.12	92.64	91.55	91.84

下载: 导出CSV

表 4 不同深度模型方法在COVID-CT数据集上的对比

Table 4. Comparison of different deep model approaches on COVID-CT datasets

Model	Acc	Pre	Recall	F1-Score
VGG16	83.94	86.65	78.98	82.59
VGG19	81.68	83.31	77.55	80.35
ResNet50	84.73	88.22	79.18	83.32
ResNet101	84.24	87.13	79.18	82.9
DenseNet121	84.63	85.7	81.84	83.72
DenseNet169	84.34	88.82	77.35	82.62
VIT	82.26	89.74	71.42	79.54
Swim Transformer	79.47	81.86	83.91	82.10
ReSWNet (ours)	93.30	93.81	92.25	93.01

下载: 导出CSV

表 5 与其它研究的对比结果

Table 5. Comparison of results with other studies

Model	Acc	Pre	Recall	F1-Score
BaseLine^[10]	84.7	97.0	76.2	85.3
Contrastive Learning ^[5]	78.6	78.0	79.7	78.8
DRE-Net^[23]	86	79	96	87
Decision function^[24]	88.3	-	-	86.7
DenseNet121+SVM^[25]	85.9	-	84.9	-
ReSWNet(ours)	93.30	93.81	92.25	93.01

下载: 导出CSV

参考文献(25)

[1]	WU F, ZHAO S, YU B, et al. A new coronavirus associated with human respiratory disease in China[J]. Nature,2020,579(7798):265-269. DOI: 10.1038/s41586-020-2008-3.
[2]	HUANG C L, WANG Y M, LI X W, et al. Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China[J]. The Lancet,2020,395(10223):497-506. DOI: 10.1016/S0140-6736(20)30183-5.
[3]	DHAMA K, KHAN S, TIWARI R, et al. Coronavirus disease 2019-COVID-19[J]. Clinical Microbiology Reviews,2020,33(4):e00028-20. DOI: 10.1128/CMR.00028-20.
[4]	KANNE J P, LITTLE B P, CHUNG J H, et al. Essentials for radiologists on COVID-19: an update—Radiology scientific expert panel[J]. Radiology,2020,296(2):E113-E114. DOI: 10.1148/radiol.2020200527.
[5]	WANG Z, LIU Q D, DOU Q. Contrastive cross-site learning with redesigned net for COVID-19 CT classification[J]. IEEE Journal of Biomedical and Health Informatics,2020,24(10):2806-2813. DOI: 10.1109/JBHI.2020.3023246.
[6]	XIE X Z, ZHONG Z, ZHAO W, et al. Chest CT for typical coronavirus disease 2019 (COVID-19) pneumonia: relationship to negative RT-PCR testing[J]. Radiology,2020,296(2):E41-E45. DOI: 10.1148/radiol.2020200343.
[7]	LEE E Y P, NG M Y, KHONG P L. COVID-19 pneumonia: what has CT taught us?[J]. The Lancet Infectious Diseases,2020,20(4):384-385. DOI: 10.1016/S1473-3099(20)30134-1.
[8]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 9992-10002.
[9]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[10]	YANG X Y, HE X H, ZHAO J Y, et al. COVID-CT-dataset: a CT scan dataset about COVID-19[J]. arXiv preprint arXiv: 2003.13865, 2020.
[11]	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 618-626.
[12]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[J]. arXiv, 2020. DOI: 10.48550/arXiv.2010.11929.
[13]	SOARES E, ANGELOV P, BIASO S, et al. SARS-CoV-2 CT-scan dataset: a large dataset of real patients CT scans for SARS-CoV-2 identification[J]. medRxiv, 2020.
[14]	GUNRAJ H, WANG L D, WONG A. COVIDNet-CT: a tailored deep convolutional neural network design for detection of COVID-19 cases from chest CT images[J]. Frontiers in Medicine,2020,7:608525. DOI: 10.3389/fmed.2020.608525.
[15]	SETHY P K , SANTI K , BEHERA, et al. Detection of coronavirus Disease (COVID-19) based on Deep Features and Support Vector Machine[J]. International Journal of Mathematical, Engineering and Management Sciences, 2020: 643-651.
[16]	OZTURK T, TALO M, YILDIRIM E A, et al. Automated detection of COVID-19 cases using deep neural networks with X-ray images[J]. Computers in Biology and Medicine,2020,121:103792. DOI: 10.1016/j.compbiomed.2020.103792.
[17]	LI L, QIN L X, XU Z G, et al. Artificial intelligence distinguishes COVID-19 from community acquired pneumonia on chest CT[J]. Radiology,2020,296(2):200905. DOI: 10.1148/radiol.2020200905.
[18]	WANG B, JIN S, YAN Q S, et al. AI-assisted CT imaging analysis for COVID-19 screening: building and deploying a medical AI system[J]. Applied Soft Computing,2021,98:106897. DOI: 10.1016/j.asoc.2020.106897.
[19]	AMYAR A, MODZELEWSKI R, LI H, et al. Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: classification and segmentation[J]. Computers in Biology and Medicine,2020,126:104037. DOI: 10.1016/j.compbiomed.2020.104037.
[20]	ISOLA P, ZHU J Y, ZHOU T H, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017.
[21]	MIRZA M, OSINDERO S. Conditional generative adversarial nets[J]. arXiv preprint arXiv: 1411.1784, 2014.
[22]	MA J , WANG Y , AN X , et al. Towards Efficient COVID-19 CT Annotation: A Benchmark for Lung and Infection Segmentation[J]. arXiv, 2020.
[23]	SONG Y, ZHENG S J, LI L, et al. Deep learning enables accurate diagnosis of novel coronavirus (COVID-19) with CT images[J]. IEEE/ACM Transactions on Computational Biology and Bioinformatics,2021,18(6):2775-2780. DOI: 10.1109/TCBB.2021.3065361.
[24]	MISHRA A K, DAS S K, ROY P, et al. Identifying COVID19 from chest CT images: a deep convolutional neural networks based approach[J]. Journal of Healthcare Engineering, 2020, 2020: 8843664.
[25]	JOKANDAN A S, ASGHARNEZHAD H, JOKANDAN S S, et al. An uncertainty-aware transfer learning-based framework for covid-19 diagnosis[J]. arXiv preprint arXiv: 2007.14846, 2020.