A deep learning framework deploying segment anything to detect pan-cancer mitotic figures from haematoxylin and eosin-stained slides.

Publisher: Nature Publishing Group UK Country of Publication: England NLM ID: 101719179 Publication Model: Electronic Cited Medium: Internet ISSN: 2399-3642 (Electronic) Linking ISSN: 23993642 NLM ISO Abbreviation: Commun Biol Subsets: MEDLINE

Original Publication: London, United Kingdom : Nature Publishing Group UK, [2018]-

Deep Learning* ; Mitosis* ; Eosine Yellowish-(YS)* ; Hematoxylin*; Humans ; Neoplasms/pathology ; Image Processing, Computer-Assisted/methods ; Staining and Labeling/methods

Mitotic activity is an important feature for grading several cancer types. However, counting mitotic figures (cells in division) is a time-consuming and laborious task prone to inter-observer variation. Inaccurate recognition of MFs can lead to incorrect grading and hence potential suboptimal treatment. This study presents an artificial intelligence-based approach to detect mitotic figures in digitised whole-slide images stained with haematoxylin and eosin. Advances in this area are hampered by the small size and variety of datasets available. To address this, we create the largest dataset of mitotic figures (N = 74,620), combining an in-house dataset of soft tissue tumours with five open-source datasets. We then employ a two-stage framework, named the Optimised Mitoses Generator Network (OMG-Net), to identify mitotic figures. This framework first deploys the Segment Anything Model to automatically outline cells, followed by an adapted ResNet18 that distinguishes mitotic figures. OMG-Net achieves an F1 score of 0.84 in detecting pan-cancer mitotic figures, including human breast carcinoma, neuroendocrine tumours, and melanoma. It outperforms previous state-of-the-art models in hold-out test sets. To summarise, our study introduces a generalisable data creation and curation pipeline and a high-performance detection model, which can largely contribute to the field of computer-aided mitotic figure detection.
Competing Interests: Competing interests: The authors declare no competing interests.
(© 2024. The Author(s).)

Williams, G. H. & Stoeber, K. The cell cycle and cancer. J. Pathol. 226, 352–364 (2012). (PMID: 10.1002/path.302221990031)
Cree, I. A. et al. Counting mitoses: SI(ze) matters! Mod. Pathol. 34, 1651–1657 (2021). (PMID: 10.1038/s41379-021-00825-7340790718376633)
Malon, C. et al. Mitotic figure recognition: agreement among pathologists and computerized detector. Anal. Cell Pathol. 35, 97–100 (2012). (PMID: 10.1155/2012/385271)
Veta, M., van Diest, P. J., Jiwa, M., Al-Janabi, S. & Pluim, J. P. Mitosis counting in breast cancer: object-level interobserver agreement and comparison to an automatic method. PLoS ONE 11, e0161286 (2016). (PMID: 10.1371/journal.pone.0161286275297014987048)
Robbins, P. et al. Histological grading of breast carcinomas: a study of interobserver agreement. Hum. Pathol. 26, 873–879 (1995). (PMID: 10.1016/0046-8177(95)90010-17635449)
Roux, L. et al. Mitosis detection in breast cancer histological images An ICPR 2012 contest. J. Pathol. Inf. 30, 8 (2013).
Roux, L. et al. MITOS-ATYPIA-14 [Data set]. Grand Challenge https://mitos-atypia-14.grand-challenge.org/ (2014).
Veta, M. et al. Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge. Med. Image Anal. 54, 111–121 (2019). (PMID: 10.1016/j.media.2019.02.01230861443)
Veta, M. et al. TUPAC16 challenge [Data set]. Grand Challenge https://tupac.grand-challenge.org/ (2021).
Irshad, H. Automated mitosis detection in histopathology using morphological and multi-channel statistics features. J. Pathol. Inf. 4, 10 (2013). (PMID: 10.4103/2153-3539.112695)
Tashk, A., Helfroush, M. S., Danyali, H. & Akbarzadeh, M. An automatic mitosis detection method for breast cancer histopathology slide images based on objective and pixel-wise textural features classification. In Proc. 5th Conference on Information and Knowledge Technology 406–410 (IEEE, 2013).
Paul, A., Dey, A., Mukherjee, D. P., Sivaswamy, J. & Tourani, V. Regenerative random forest with automatic feature selection to detect mitosis in histopathological breast cancer images. In Proc. Medical Image Computing and Computer-Assisted Intervention 94–102 (Springer, 2015).
Mahmood, T., Arsalan, M., Owais, M., Lee, M. B. & Park, K. R. Artificial intelligence-based mitosis detection in breast cancer histopathology images using faster R-CNN and deep CNNs. J. Clin. Med. 9, 749 (2020). (PMID: 10.3390/jcm9030749321642987141212)
Sebai, M., Wang, X. & Wang, T. MaskMitosis: a deep learning framework for fully supervised, weakly supervised, and unsupervised mitosis detection in histopathology images. Med. Biol. Eng. Comput. 58, 1603–1623 (2020). (PMID: 10.1007/s11517-020-02175-z32445109)
Li, C., Wang, X., Liu, W. & Latecki, L. J. DeepMitosis: mitosis detection via deep detection, verification and segmentation networks. Med. Image Anal. 45, 121–133 (2018). (PMID: 10.1016/j.media.2017.12.00229455111)
Cai, D., Sun, X., Zhou, N., Han, X. & Yao, J. Efficient mitosis detection in breast cancer histology images by RCNN. In Proc. 16th International Symposium on Biomedical Imaging 919–922 (IEEE, 2019).
Aubreville, M. et al. A comprehensive multi-domain dataset for mitotic figure detection. Sci. Data 10, 484 (2023). (PMID: 10.1038/s41597-023-02327-43749153610368709)
Aubreville, M. et al. MItosis DOmain generalization challenge. 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021).[Data set]. Zenodo https://doi.org/10.5281/zenodo.4573978 (2021).
Aubreville, M. et al. MItosis DOmain generalization challenge 2022. 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). [Data set]. Zenodo https://doi.org/10.5281/zenodo.6362337 (2022).
Meyer, J. et al. Breast carcinoma malignancy grading by Bloom–Richardson system vs proliferation index: reproducibility of grade and advantages of proliferation index. Mod. Pathol. 18, 1067–1078 (2005). (PMID: 10.1038/modpathol.380038815920556)
Bertram, C. A. et al. Computerized calculation of mitotic count distribution in canine cutaneous mast cell tumor sections: mitotic count is area dependent. Vet. Pathol. 57, 214–226 (2020). (PMID: 10.1177/030098581989068631808382)
van Diest, P. J. et al. Reproducibility of mitosis counting in 2469 breast cancer specimens: results from the Multicenter Morphometric Mammary Carcinoma Project. Hum. Pathol. 23, 603–607 (1992). (PMID: 10.1016/0046-8177(92)90313-R1592381)
Aubreville, M. et al. A completely annotated whole slide image dataset of canine breast cancer to aid human breast cancer research. Sci. Data 7, 417 (2020). (PMID: 10.1038/s41597-020-00756-z332471167699627)
Bertram, C. A. et al. A large-scale dataset for mitotic figure assessment on whole slide images of canine cutaneous mast cell tumor. Sci. Data 6, 274 (2019). (PMID: 10.1038/s41597-019-0290-4317541056872565)
Coindre, J. M. Grading of soft tissue sarcomas: review and update. Arch. Pathol. Lab. Med. 130, 1448–1453 (2006). (PMID: 10.5858/2006-130-1448-GOSTSR17090186)
He, K., Gkioxari, G., Dollár, P. & Girshick, R. Mask R-CNN. arXiv, https://doi.org/10.48550/arXiv.1703.06870 (2018).
Lou, W. et al. Structure embedded nucleus classification for histopathology images. IEEE Transactions on Medical Imaging 43, 3149–3160 (2024).
Kirillov, A. et al. Segment anything. arXiv. https://doi.org/10.48550/arXiv.2304.02643 (2023).
Kim, J. Y. et al. The value of phosphohistone H3 as a proliferation marker for evaluating invasive breast cancers: a comparative study with Ki67. Oncotarget 8, 65064–65076 (2017). (PMID: 10.18632/oncotarget.17775290294125630312)
Tacha, D. E. A New rabbit monoclonal phospho-histone H3 (pHH3) hybridoma: an immunohistochemical comparison study with a rabbit polyclonal pHH3. https://api.semanticscholar.org/CorpusID:54518201 (2015).
Hendzel, M. J. et al. Mitosis-specific phosphorylation of histone H3 initiates primarily within pericentromeric heterochromatin during G2 and spreads in an ordered fashion coincident with mitotic chromosome condensation. Chromosoma 106, 348–360 (1997). (PMID: 10.1007/s0041200502569362543)
You, C. et al. Incremental learning meets transfer learning: application to multi-site prostate MRI segmentation. In Proc. Distributed, Collaborative, and Federated Learning, and Affordable AI Healthcare for Resource Diverse Global Health (DeCaF 2022, FAIR 2022) 13573, 3–16 (Springer, 2022).
Aubreville, M. et al. Domain generalization across tumor types, laboratories, and species—insights from the 2022 edition of the mitosis domain generalization challenge. Med. Image Anal. 94, 103155 (2024). (PMID: 10.1016/j.media.2024.10315538537415)
Jahanifar, M. et al. Mitosis detection, fast and slow: robust and efficient detection of mitotic figures. Med. Image Anal. 94, 103132 (2024). (PMID: 10.1016/j.media.2024.10313238442527)
Kotte, S. et al. A deep learning-based ensemble model for generalized mitosis detection in H &E stained whole slide images. In Proc. Mitosis Domain Generalization and Diabetic Retinopathy Analysis (MIDOG 2022, DRAC 2022) 13597, 221–225 (Springer, 2023).
Lafarge, M. W. & Koelzer, V. H. Fine-grained hard-negative mining: generalizing mitosis detection with a fifth of the MIDOG 2022 dataset. In Proc. Mitosis Domain Generalization And Diabetic Retinopathy Analysis (MIDOG 2022, DRAC 2022) 13597, 226–233 (Springer, 2023).
Ren, S., He, K., Girshick, R. & Sun, J. Faster R-CNN: towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems 1 (NIPS’15), 91–99 (MIT Press, Cambridge, MA, United States, 2015).
Lin, T. et al. Focal loss for dense object detection. In 2017 IEEE International Conference on Computer Vision (ICCV), 2999–3007 (Venice, Italy, 2017).
Redmon, J., Divvala, S., Girshick R. & Farhadi, A. You only look once: unified, real-time object detection. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 779–788 (Las Vegas, NV, United States, 2016).
Chen, Z., Badrinarayanan, V., Lee, C. & Rabinovich, A. GradNorm: gradient normalization for adaptive loss balancing in deep multitask networks. In Proceedings of the 35th International Conference on Machine Learning (PMLR) 80, 794–803 (Stockholmsmässan, Stockholm Sweden, 2018).
Argyriou, A., Evgeniou, T. & Pontil, M. Multi-task feature learning. In Advances in Neural Information Processing Systems 19 (NIPS 2006) (MIT Press, 2006).
Zhao, Y. et al. DETRs beat YOLOs on real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16965–16974 (Seattle, WA, United States, 2024).
Çayır, S. et al. MITNET: a novel dataset and a two-stage deep learning approach for mitosis recognition in whole slide images of breast cancer tissue. Neural Comput. Appl. 34, 17837–17851 (2022). (PMID: 10.1007/s00521-022-07441-9)
Sohail, A. et al. A multi-phase deep CNN-based mitosis detection framework for breast cancer histopathological images. Sci. Rep. 11, 6215 (2021). (PMID: 10.1038/s41598-021-85652-1337376327973714)
Israel, U. et al. A foundation model for cell segmentation. arXiv. https://doi.org/10.48550/arXiv.2311.11004 (2023).
Archit, A. et al. Segment anything for microscopy. bioRxiv. https://doi.org/10.1101/2023.08.21.554208 (2023).
Hörst, F. et al. CellViT: vision transformers for precise cell segmentation and classification. Med. Image Anal. 94, 103143 (2024). (PMID: 10.1016/j.media.2024.10314338507894)
Graham, S. et al. Lizard: a large-scale dataset for colonic nuclear instance segmentation and classification. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 684–693 (Montreal, BC, Canada, 2021).
Voig, B. et al. Investigation of semi- and self-supervised learning methods in the histopathological domain. J. Pathol. Inform. 14, 100305 (2023). (PMID: 10.1016/j.jpi.2023.100305)
You, C. et al. Rethinking semi-supervised medical image segmentation: a variance-reduction perspective. Adv. Neural Inf. Process Syst. 36, 9984–10021 (2023). (PMID: 3881311411136570)
Elmaci, İ., Altinoz, M. A., Sari, R. & Bolukbasi, F. H. Phosphorylated histone H3 (PHH3) as a novel cell proliferation marker and prognosticator for meningeal tumors: a short review. Appl Immunohistochem. Mol. Morphol. 26, 627–631 (2018). (PMID: 10.1097/PAI.000000000000049928777144)
Fischler, M. A. & Bolles, R. C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 341–395 (1981).
Ribalta, T., McCutcheon, I. E., Aldape, K. D., Bruner, J. M. & Fuller, G. N. The mitosis-specific antibody anti-phosphohistone-H3 (PHH3) facilitates rapid reliable grading of meningiomas according to WHO 2000 criteria. Am. J. Surg. Pathol. 28, 1532–1536 (2004). (PMID: 10.1097/01.pas.0000141389.06925.d515489659)
Hosang, J., Benenson, R. & Schiele, B. Learning non-maximum suppression, In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6469–6477 (Honolulu, HI, USA, 2017).
Loshchilov, I. & Hutter, F. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations (ICLR 2019) (New Orleans, Louisiana, United States, 2019).
Loshchilov, I. & Hutter, F. SGDR: Stochastic Gradient Descent with Warm Restarts. In 5th International Conference on Learning Representations (ICLR 2017) (Palais des Congrès Neptune, Toulon, France, 2017).
Ruifrok, A. C. & Johnston, D. A. Quantification of histochemical staining by color deconvolution. Anal. Quant. Cytol. Histol. 23, 291–299 (2001). (PMID: 11531144)
Tellez, D. et al. Whole-slide mitosis detection in H&E breast histology using PHH3 as a reference to train distilled stain-invariant convolutional networks. IEEE Trans. Med. Imaging 37, 2126–2136 (2018). (PMID: 10.1109/TMI.2018.282019929994086)
Nachar, N. The Mann-Whitney U: a test for assessing whether two independent samples come from the same distribution. Tutor. Quant. Methods Psychol. 4, 13–20 (2008).
Shen, Z. et al. OMG-Octo: Uniformised large scale database of mitotic figures in Haematoxylin and Eosin-stained slides. [Data set]. Zenodo. https://doi.org/10.5281/zenodo.14246170 (2024).

SUK18.2021 Sarcoma UK

TDQ283MPCW (Eosine Yellowish-(YS))
YKM8PY2Z55 (Hematoxylin)

Date Created: 20241220 Date Completed: 20241220 Latest Revision: 20241220

20241220

10.1038/s42003-024-07398-6

39702417