The paradigm of precision medicine has transformed oncology, moving beyond one-size-fits-all treatments to strategies tailored to the specific molecular drivers of a patient’s cancer. This approach relies on identifying genomic biomarkers that predict sensitivity or resistance to targeted therapies and immunotherapies. Landmark examples include the use of tyrosine kinase inhibitors for Non-Small Cell Lung Cancer (NSCLC) with EGFR mutations, trastuzumab for HER2-amplified breast cancer, and BRAF inhibitors for melanomas harboring the BRAF V600E mutation. The approval of pembrolizumab for any solid tumor with high microsatellite instability (MSI-H) marked a shift towards ‘site-agnostic’ treatments based purely on a genomic signature. These advances are powered by technologies like Next-Generation Sequencing (NGS) and liquid biopsies.
Despite this progress, significant challenges persist. Intra-tumor heterogeneity, where different cells within the same tumor possess distinct mutations, can lead to incomplete treatment responses. Furthermore, cancers can evolve, developing acquired resistance to therapies that were initially effective. The sheer volume and complexity of genomic data also present hurdles for clinical interpretation. A single data modality, such as genomics, provides an incomplete picture. To build a more holistic understanding of a patient’s disease and predict treatment outcomes more accurately, there is a growing need to integrate data from multiple sources, including molecular profiles, medical imaging, and clinical history from electronic health records (EHRs).
This paper reviews the application of multi-modal machine learning (ML) and Explainable AI (XAI) to address these challenges in precision oncology. The primary objective is to synthesize the current landscape of advanced computational models that fuse heterogeneous data for predicting treatment response. We explore how emerging architectures like Graph Neural Networks (GNNs) and Transformers are being leveraged and argue that the integration of XAI is not merely an accessory but a prerequisite for the clinical translation, regulatory approval, and ultimate adoption of these powerful tools by clinicians.
The integration of diverse data modalities is foundational to advancing precision oncology. The ultimate goal is to combine genomics, proteomics, medical imaging (radiology and digital histopathology), EHRs, and even real-world data from wearables to create a comprehensive ‘digital twin’ of a patient.7 This multi-scale view allows for a deeply personalized and adaptive treatment strategy. A key area within this domain is radiogenomics, which connects imaging features with genomic characteristics, offering the potential for non-invasive ‘virtual biopsies’.5
To combine these data, researchers employ several fusion strategies. Early fusion concatenates feature vectors from different modalities at the input level, while late fusion builds separate models for each modality and integrates their predictions at the end. Intermediate fusion, considered a more sophisticated approach, creates a joint feature representation during the learning process, allowing modalities to inform one another.3,4 The choice of strategy is often task-dependent and determined empirically, as no single method is universally superior.3
Recent advances have been driven by deep learning architectures capable of handling such complexity. Graph Neural Networks (GNNs) and Transformers have become state-of-the-art.2 GNNs are ideal for representing different data types as nodes in a graph (e.g., genes, cells) and modeling the complex relationships between them as edges, as seen in models like EMOGI and MTGCN that integrate multi-omics data with Protein-Protein Interaction (PPI) networks.8 Transformers, originally from natural language processing, use self-attention mechanisms to process diverse data types in a unified manner, capturing global dependencies.2 This has been applied to sequence data like DNA/RNA and drug structures in models such as MuAt and DeepTTA.8
The development of large-scale foundation models pretrained on vast datasets represents another significant leap. Models like CONCH, a vision-language model for pathology pretrained on over a million image-caption pairs, can be fine-tuned for specific tasks with high accuracy.9 Frameworks like HoneyBee streamline the creation of machine learning-ready datasets by generating embeddings from raw data using specialized foundation models for each modality, such as UNI for histopathology and GatorTron for clinical text.10 Fine-tuning these models on specific tasks has been shown to dramatically improve performance, as demonstrated by an increase in cancer-type classification accuracy from 0.889 to 0.976 after fine-tuning a text model.10
This paper employs a systematic synthesis of recent academic literature to review the current state of multi-modal machine learning and explainable AI in the context of precision oncology, with a specific focus on treatment response prediction. The analysis is guided by the primary objective of AI in this domain: to provide robust decision support for clinicians. The methodology involved identifying and evaluating key studies and reviews that describe the integration of multiple data modalities, the performance of predictive models, the application of XAI techniques, and the persistent challenges related to clinical translation.
The core technologies analyzed include advanced deep learning architectures and the XAI methods used to interpret them. For data fusion, we focus on studies utilizing GNNs and Transformers, which are increasingly favored for their ability to model complex, relational data.2,8 For data modalities, the review encompasses the integration of histopathology (whole-slide images), genomics (e.g., mutation status, RNA-seq), radiology (e.g., CT, MRI, PET), and clinical data from EHRs.4,6,10 For explainability, we examine a range of techniques tailored to different data types, including attention heatmaps and Class Activation Methods (CAMs) for imaging, as well as Integrated Gradients, Layer-wise Relevance Propagation (LRP), and SHAP (SHapley Additive exPlanations) for molecular and tabular data.4,8,17 The synthesis also incorporates findings from studies on interactive XAI platforms that facilitate human-in-the-loop analysis.13
This section presents key insights from recent studies on multi-modal machine learning and explainable AI in precision oncology. It highlights the performance of advanced models, the role of explainability in clinical trust, and the challenges that hinder real-world implementation.
Multi-modal models consistently demonstrate superior or competitive performance compared to their unimodal counterparts. For instance, the Brim model, which integrates histopathology, genomics, and transcriptomics for pan-cancer survival prediction, achieved an average concordance index (C-index) of 0.682, outperforming unimodal models.1 Other frameworks like PORPOISE and ClamSurvival reported similar pan-cancer prognostic performance with C-indices of 0.64 and 0.65, respectively.8 In the domain of radiogenomics, deep learning models have shown remarkable accuracy. The RadGenNets model, fusing PET/CT scans with clinical/genomic data, predicted EGFR, KRAS, and ALK mutations in NSCLC patients with an Area Under the Curve (AUC) exceeding 0.94 for each.6 A broader review of 51 radiogenomics studies found a mean accuracy of 84.34% and a mean AUC of 85.42%.5 This promise has led to commercialization, such as the ArteraAI Prostate Test, which uses histology and clinical data for prognosis prediction.11
The primary objective of XAI in the clinical setting is to empower clinicians with transparent decision support.17 XAI techniques are tailored to the data modality. For histopathology and radiology, attention heatmaps and CAMs highlight the specific image regions driving a prediction.4 For molecular and EHR data, methods like Integrated Gradients assign attribution scores to features like gene mutations, revealing their impact on the model’s output.4 These techniques are being integrated directly into advanced architectures. The GNN-based model EMOGI uses Layer-wise Relevance Propagation (LRP) to identify key features in PPI networks, while the Transformer-based model MuAt uses its attention mechanism to link predictions to known mutational signatures.8
A significant advancement is the development of interactive XAI platforms. CLARUS, for example, is designed for GNNs in biomedical research and enables human-in-the-loop analysis.13 It allows experts to visualize patient-specific interaction networks and ask manual counterfactual questions by adding or deleting nodes (e.g., proteins) and edges (e.g., interactions) to observe the effect on the model’s prediction.13 In a case study on kidney cancer, researchers used CLARUS to systematically delete nodes and edges guided by XAI relevance scores, which ultimately caused the GNN’s prediction to flip, thereby identifying a critical signaling pathway.13 This interactive capability, which includes the ability to retrain the model based on user changes, distinguishes it from less dynamic tools and represents a key step toward building clinician trust.13
Despite high reported accuracies, the path to clinical implementation is fraught with challenges. A primary issue is the lack of reproducibility and robust validation. The clinical translation of radiogenomics is slow due to heterogeneous imaging protocols, small retrospective cohorts that risk overfitting, and a general failure to move beyond establishing correlation to proving causation.5 A stark example is the external validation of a CT-based radiogenomics model for EGFR mutation detection, where the AUC plummeted to approximately 0.5, performing worse than a model using only clinical variables.14 This highlights the need for feature stability; one study found that a prognostic model built with high-repeatability radiomic features was significantly more generalizable than one built with unstable features.15
Data availability and quality are also major hurdles. In large datasets like The Cancer Genome Atlas (TCGA), missing data is common; for instance, only a fraction of glioma cases have corresponding radiology and genomics data.4 While solutions like synthetic data generation are being explored, they introduce their own complexities.4 Furthermore, a systematic review found that while deep learning models reported slightly better performance than handcrafted radiomics, the methodological quality of the studies was often lower, partly due to a lack of standardization for deep learning features.16 Crucially, none of the reviewed studies were prospectively registered or assessed for clinical utility.12,16
The concept of a ‘digital twin’ also faces significant barriers. A review of digital twins in oncology found a profound lack of clinical validation and data reliability, with most solutions using synthetic or limited data.7 Technical challenges in data integration, scalability, and the substantial resource requirements (e.g., high-performance computing) further hinder broad application.7 Finally, mechanistic models based on differential equations or agent-based simulations suggest that purely data-driven approaches may miss key biological dynamics, such as how the spatial distribution of resistant cells can undermine certain treatment strategies like adaptive therapy.18,19
The findings present a dual narrative for AI in precision oncology. On one hand, multi-modal models demonstrate immense technical potential, achieving high accuracy on curated, retrospective datasets.1,6,8 On the other hand, their real-world clinical utility remains largely unproven, with significant roadblocks in validation, reproducibility, and trust.5,12,14 The stark contrast between the high AUCs reported for models like RadGenNets6 and the failure of other radiogenomics models in external validation14 underscores a critical gap between development and deployment. This gap suggests that performance on a training set is a poor proxy for clinical value.
The “black box” nature of complex deep learning models is a primary barrier to clinical adoption, limiting clinician trust and slowing regulatory acceptance.17 This is where XAI becomes indispensable. It is not merely a feature for transparency but a necessary tool for validation and debugging. By allowing a clinician to see *why* a model made a certain prediction—for instance, by highlighting a specific tumor region on a pathology slide or a key gene mutation—XAI provides a basis for the clinician to apply their own expertise, verify the model’s reasoning, and ultimately accept or reject its recommendation. Interactive platforms like CLARUS13 represent the next frontier, transforming XAI from a passive reporting tool into an active, collaborative instrument for scientific discovery and clinical inquiry. This directly addresses the core need to empower the human expert in the loop.
The research also highlights a need to move beyond correlation towards a more causal understanding. Current radiogenomics models establish statistical associations but cannot replace definitive tissue-based genomic analysis.5 Integrating data-driven ML with mechanistic models, which simulate biological processes like tumor growth and resistance acquisition,18,19 could provide a more robust foundation for prediction by grounding statistical patterns in biological principles.
Finally, the translation of these technologies is constrained by regulatory and ethical frameworks that are struggling to keep pace. The FDA’s proposed “Predetermined Change Control Plan” for adaptive AI is an attempt to address continuously learning models, but it raises significant concerns about auditability and validation.11,17 Moreover, ethical issues of data privacy, decision-making responsibility, and fairness are largely unaddressed in current research.7 Without careful attention, there is a substantial risk of deploying models trained on biased data, which could amplify existing healthcare inequalities.7
Multi-modal machine learning, powered by advanced architectures like GNNs and Transformers, holds transformative potential for predicting treatment response in precision oncology. By integrating genomics, imaging, and clinical records, these models can create a holistic patient view that surpasses the predictive power of any single data source. However, the path from promising retrospective results to routine clinical use is obstructed by major challenges, including a lack of prospective validation, poor reproducibility, and the inherent opacity of complex models.
For these powerful tools to be successfully translated, a paradigm shift is required. Future research must prioritize large-scale, multi-institutional prospective trials to demonstrate tangible clinical utility and cost-effectiveness. Concurrently, the development of robust, clinician-centric Explainable AI is paramount to building trust and ensuring that AI serves as a decision-support tool, not a decision-making oracle. Future work should focus on advancing interactive XAI platforms, developing standardized methods for data handling and model validation, and integrating mechanistic modeling to deepen biological understanding. Finally, a collaborative effort among researchers, clinicians, regulators, and ethicists is essential to establish clear and adaptive frameworks that can govern these dynamic technologies, ensuring they are deployed safely, effectively, and equitably.
