Explainable Hybrid Multi-Feature Fusion Model with Transformer for Medical Image Classification

Abstract

Breast Cancer disease detection is one of the important areas. Breast classification is an essential part in computer-aided diagnosis, which allows detecting and treating different diseases at the earliest stage. Nonetheless, traditional deep learning architectures usually use convolutional neural networks (CNNs) only, which might not be able to extract complementary information in handcrafted features. To overcome this drawback, this paper presents a proposed explainable hybrid multi-feature fusion framework, which combines deep and handcrafted features with a Transformer-based fusion mechanism. Deep features are computed with the help of a pre-trained ResNet50 architecture, whereas handcrafted features, such as texture and color descriptors, are calculated with the help of Gray-Level Co-occurrence Matrix (GLCM) and histogram-based approaches. These heterogeneous features are then mapped into a shared feature space and combined using a Transformer encoder to effectively capture inter-feature relationships.

The model suggested is tested on a multi-class breast cancer dataset and shows a better performance as compared to baseline CNN models. The accuracy, precision, recall, F1-score, and area under the curve (AUC) are used to conduct comprehensive experiments. Moreover, explainability is added with the help of Grad-CAM to visualize attentions of the model, and t-SNE analysis is used to illustrate that the fusion has enhanced the separability of features. The findings indicate that the suggested hybrid system has a substantial positive impact on the classification and interpretability of the data, which is why it is a promising solution to the analysis of real-life medical images.