CellViT: Vision Transformers for precise cell segmentation and classification

被引:24
作者
Hoerst, Fabian [1 ,2 ,15 ]
Rempe, Moritz [1 ,2 ]
Heine, Lukas [1 ,2 ]
Seibold, Constantin [1 ,3 ]
Keyl, Julius [1 ,4 ]
Baldini, Giulia [1 ,5 ]
Ugurel, Selma [6 ,7 ]
Siveke, Jens [8 ,9 ,10 ,11 ]
Gruenwald, Barbara [12 ,13 ]
Egger, Jan [1 ,2 ]
Kleesiek, Jens [1 ,2 ,7 ,14 ]
机构
[1] Univ Hosp Essen AoR, Inst AI Med IKIM, D-45131 Essen, Germany
[2] Univ Hosp Essen AoR, Canc Res Ctr Cologne Essen CCCE, West German Canc Ctr Essen, D-45147 Essen, Germany
[3] Univ Hosp Essen AoR, Clin Nucl Med, D-45147 Essen, Germany
[4] Univ Hosp Essen AoR, Inst Pathol, D-45147 Essen, Germany
[5] Univ Hosp Essen AoR, Inst Intervent & Diagnost Radiol & Neuroradiol, D-45147 Essen, Germany
[6] Univ Hosp Essen AoR, Dept Dermatol, D-45147 Essen, Germany
[7] German Canc Consortium DKTK, Partner Site Essen, D-69120 Heidelberg, Germany
[8] German Canc Res Ctr, partnership German Canc Res Ctr DKFZ, Partner Site Essen, D-45147 Essen, Germany
[9] Univ Hosp Essen AoR, Univ Hosp Essen, West German Canc Ctr, Partner Site Essen, D-45147 Essen, Germany
[10] Univ Hosp Essen AoR, Univ Duisburg Essen, Bridge Inst Expt Tumor Therapy BIT, D-45147 Essen, Germany
[11] Univ Duisburg Essen, Univ Hosp Essen AoR, West German Canc Ctr Essen, Div Solid Tumor Translat Oncol DKTK, D-45147 Essen, Germany
[12] Univ Hosp Essen AoR, West German Canc Ctr, Dept Urol, D-45147 Essen, Germany
[13] Princess Margaret Canc Ctr, Toronto, ON M5G 2M9, Canada
[14] TU Dortmund Univ, Dept Phys, D-44227 Dortmund, Germany
[15] Insitute Artificial Intelligence Med, Girardetstr 2, D-45131 Essen, Germany
关键词
Cell segmentation; Digital pathology; Deep learning; Vision transformer; NUCLEI SEGMENTATION; IMAGE SEGMENTATION; NET; NETWORK;
D O I
10.1016/j.media.2024.103143
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Nuclei detection and segmentation in hematoxylin and eosin -stained (H&E) tissue images are important clinical tasks and crucial for a wide range of applications. However, it is a challenging task due to nuclei variances in staining and size, overlapping boundaries, and nuclei clustering. While convolutional neural networks have been extensively used for this task, we explore the potential of Transformer -based networks in combination with large scale pre -training in this domain. Therefore, we introduce a new method for automated instance segmentation of cell nuclei in digitized tissue samples using a deep learning architecture based on Vision Transformer called CellViT. CellViT is trained and evaluated on the PanNuke dataset, which is one of the most challenging nuclei instance segmentation datasets, consisting of nearly 200,000 annotated nuclei into 5 clinically important classes in 19 tissue types. We demonstrate the superiority of large-scale in -domain and out -of -domain pre -trained Vision Transformers by leveraging the recently published Segment Anything Model and a ViT-encoder pre -trained on 104 million histological image patches - achieving state-of-the-art nuclei detection and instance segmentation performance on the PanNuke dataset with a mean panoptic quality of 0.50 and an F 1 -detection score of 0.83. The code is publicly available at https://github.com/TIO-IKIM/CellViT.
引用
收藏
页数:16
相关论文
共 74 条
  • [1] Abraham N, 2019, I S BIOMED IMAGING, P683, DOI 10.1109/ISBI.2019.8759329
  • [2] An Integrated Region-, Boundary-, Shape-Based Active Contour for Multiple Object Overlap Resolution in Histological Imagery
    Ali, Sahirzeeshan
    Madabhushi, Anant
    [J]. IEEE TRANSACTIONS ON MEDICAL IMAGING, 2012, 31 (07) : 1448 - 1460
  • [3] QuPath: Open source software for digital pathology image analysis
    Bankhead, Peter
    Loughrey, Maurice B.
    Fernandez, Jose A.
    Dombrowski, Yvonne
    Mcart, Darragh G.
    Dunne, Philip D.
    McQuaid, Stephen
    Gray, Ronan T.
    Murray, Liam J.
    Coleman, Helen G.
    James, Jacqueline A.
    Salto-Tellez, Manuel
    Hamilton, Peter W.
    [J]. SCIENTIFIC REPORTS, 2017, 7
  • [4] Bommasani R., 2021, arXiv, DOI [DOI 10.48550/ARXIV.2108.07258, 10.48550/arXiv.2108.07258]
  • [5] Albumentations: Fast and Flexible Image Augmentations
    Buslaev, Alexander
    Iglovikov, Vladimir I.
    Khvedchenya, Eugene
    Parinov, Alex
    Druzhinin, Mikhail
    Kalinin, Alexandr A.
    [J]. INFORMATION, 2020, 11 (02)
  • [6] Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
    Campanella, Gabriele
    Hanna, Matthew G.
    Geneslaw, Luke
    Miraflor, Allen
    Silva, Vitor Werneck Krauss
    Busam, Klaus J.
    Brogi, Edi
    Reuter, Victor E.
    Klimstra, David S.
    Fuchs, Thomas J.
    [J]. NATURE MEDICINE, 2019, 25 (08) : 1301 - +
  • [7] Caron M, 2020, ADV NEUR IN, V33
  • [8] Emerging Properties in Self-Supervised Vision Transformers
    Caron, Mathilde
    Touvron, Hugo
    Misra, Ishan
    Jegou, Herve
    Mairal, Julien
    Bojanowski, Piotr
    Joulin, Armand
    [J]. 2021 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2021), 2021, : 9630 - 9640
  • [9] DCAN: Deep Contour-Aware Networks for Accurate Gland Segmentation
    Chen, Hao
    Qi, Xiaojuan
    Yu, Lequan
    Heng, Pheng-Ann
    [J]. 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), 2016, : 2487 - 2496
  • [10] Chen Jieneng, 2021, arXiv, DOI DOI 10.48550/ARXIV.2102.04306