Machine Learning and Digital Histopathology Analysis for Tissue Characterization and Treatment Response Prediction in Breast Cancer

Breast cancer is the most common type of diagnosed cancer and the leading cause of cancer-related death in women. Early diagnosis and prognosis in breast cancer patients can permit more therapeutic options and possibly improve their survival and quality of life. The gold standard approach for breast cancer diagnosis and characterization is histopathology assessment on biopsy specimens, which is time and resource-demanding. In this dissertation project, state-of-the-art machine learning (ML) methods have been developed and investigated for breast tissue characterization, nuclei segmentation, and chemotherapy response prediction in breast cancer patients using pre-treatment digitized histopathology images. First, a novel multi-scale attention-guided deep learning model is introduced to characterize breast tissue on digital pathology images according to four histological types. Evaluation results on the test set show the effectiveness of the proposed approach in accurate histopathology image classification with an accuracy of 97.5%. In the next step, a cascaded deep-learning-based model is proposed to delineate tumor nuclei in digital pathology images accurately, which is an essential step for extracting hand-crafted quantitative features for analysis with conventional ML models. The proposed model could achieve an F1 score of 0.83 on an independent test set. At the end, two novel ML frameworks are introduced and investigated for chemotherapy response prediction. In the first approach, a digital histopathology image analysis framework has been developed to extract various subsets of quantitative features from the segmented digitized slides for conventional ML model development. Several ML experiments have been conducted with different feature sets to develop prediction models of therapy response using a gradient boosting machine with decision trees. The proposed model with the optimal feature set could achieve an accuracy of 84%, sensitivity of 85% and specificity of 82% on an independent test set. The second approach introduces a hierarchical self-attention-guided deep learning framework to predict breast cancer response to chemotherapy using digital histopathology images of pre‑treatment tumor biopsies. The whole slide images (WSIs) are processed automatically through the proposed hierarchical framework consisting of patch-level and tumor-level processing modules followed by a patient-level response prediction component. A combination of convolutional and transformer modules is utilized at each processing level. The proposed framework could outperform the conventional ML models with a test accuracy, sensitivity, and specificity of 86%, 87%, and 83%, respectively. The proposed methods and the reported results in this dissertation are steps toward streamlining the histopathology workflow and implementing response-guided precision oncology for breast cancer patients.