The role of explainability and transparency in fostering trust in AI healthcare systems: a systematic literature review, open issues and potential solutions

The healthcare sector has advanced significantly as a result of the ability of artificial intelligence (AI) to solve cognitive problems that once required human intelligence. As artificial intelligence finds more applications in healthcare, trustworthiness must be guaranteed. Even while AI has the potential to improve healthcare, there are still challenging issues because it is yet to be widely adopted, especially when it comes to transparency. Concerns about comprehending the internal workings of AI models, possible biases, model robustness, and generalizability are raised by their opacity which makes them function like black boxes. A solution for worries over the transparency of AI algorithms is explainable AI. Explainable AI seeks to enhance AI explainability and analytical capabilities, particularly in vital industries like healthcare. Even though earlier research has examined several explainable AI-related topics, such as a lexicon, industry-specific overviews, and applications in the healthcare industry, a thorough analysis concentrating on the function of explainable AI in building trust in AI healthcare systems is required. In an effort to close this gap, a systematic literature review that adheres to PRISMA principles that analyze relevant papers that were published between 2015 and 2023 was done in this paper. To determine the critical role that explainable AI plays in fostering trust, this study examines widely utilized methodologies, machine learning and deep learning techniques, datasets, performance measures and validation procedures used in AI healthcare research. In addition, research issues and potential research directions are also discussed in this research. Thus, this systematic review provides a thorough summary of the present status of research on explainability and transparency in AI healthcare systems, thus illuminating crucial factors that affect user trust. The results are intended to assist researchers, policymakers and healthcare professionals in developing a more transparent, responsible and reliable AI system in the healthcare sector. © The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2024.