The Role of XAI in Advice-Taking from a Clinical Decision Support System: A Comparative User Study of Feature Contribution-Based and Example-Based Explanations
<p>Interaction between healthcare practitioners and a CDSS, classified into over-reliance, self-reliance, and appropriate reliance.</p> "> Figure 2
<p>Workflow Overview. The order of the explanations and patients/cases is randomized.</p> "> Figure 3
<p>CDSS prediction and explanation by feature contribution for an exemplar case.</p> "> Figure 4
<p>CDSS prediction and explanation by example for an exemplar case.</p> "> Figure 5
<p>WOA for explanation by feature contribution and explanation by example. This figure shows that our CDSS with either type of explanation has a significant impact on the decision-making (WOA significantly greater than zero).</p> "> Figure 6
<p>WOA for explanation by feature contribution and explanation by example for correctly and incorrectly predicted cases.</p> "> Figure 7
<p>Healthcare practitioners’ preference for explanations in a CDSS.</p> "> Figure 8
<p>Comparison of the overall WOA of an explainable CDSS between subgroups of participants. (<b>a</b>) Comparison between obstetricians and other HCPs (midwives and dietitians). (<b>b</b>) Comparison between HCPs with ⩽ and >10 years of experience. (<b>c</b>) Comparison between HCPs with and without prior experience of CDSS use. (<b>d</b>) Comparison between all HCPs who were less and more inclined to CDSS use. (<b>e</b>) Comparison between obstetricians who were less and more inclined to CDSS use. Significant differences are highlighted in bold. HCP: healthcare practitioner.</p> "> Figure 9
<p>WOA for feature contribution and example for participants who preferred explanation by feature contribution.</p> ">
Abstract
:1. Introduction
- Does an explainable CDSS have any impact on healthcare practitioners’ decision-making?
- Which XAI method leads to overall higher advice-taking for both correctly and incorrectly predicted cases? Is it associated with healthcare practitioners’ clinical expertise?
- Do these XAI methods lead to appropriate advice-taking, over-reliance or self-reliance? Are they associated with healthcare practitioners’ clinical expertise?
- Which XAI method or combination do healthcare practitioners prefer in a CDSS?
- Is the advice-taking from an explainable CDSS associated with healthcare practitioners’ expertise, prior experience of CDSS, their attitude towards the use of CDSS, etc.?
2. Materials and Methods
2.1. Participants Recruitment
2.2. CDSS for the Prediction of GDM
2.3. Explanation by Feature Contribution
2.4. Explanation by Example
2.5. Decision-Making Task
2.6. Experimental Design and Data Collection
3. Results
3.1. RQ1: Impact on Decision-Making
3.2. RQ2: Overall Advice-Taking
3.3. RQ3: Appropriate Advice-Taking, Over-Reliance or Self-Reliance
3.4. RQ4: Preferred XAI Method
“I don’t think comparing to one similar patient is helpful, this does not predict the risk to our patient. If comparing to other patients it should be a comparison to a large group of similar patients, not just one case. ” (Obstetrician)
“visually clearer” (Dietitian)
3.5. RQ5: Factors Associated with Advice-Taking
3.6. Others
“… I can see its clinical applicability”. (Obstetrician)
“… Height matters. ” (Dietitian)
“… there is no internationally accepted definition of GDM and no accepted screening protocol … I think it is hard to understand the added clinical benefit of a risk prediction tool …”. (Obstetrician)
4. Discussion
5. Conclusions
Supplementary Materials
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
CDSS | Clinical Decision Support System |
XAI | Explainable Artificial Intelligence |
AI | Artificial Intelligence |
ANN | Artificial Neural Network |
SVM | Support Vector Machine |
RQ | Research Question |
GP | General Practitioner |
PEARS | Pregnancy Exercise and Nutrition Research Study |
GDM | Gestational Diabetes Mellitus |
SHAP | Shapley Additive Explanations |
BMI | Body Mass Index |
WOA | Weight of Advice |
HCP | Healthcare Practitioner |
KNN | K-Nearest Neighbours |
References
- Sutton, R.T.; Pincock, D.; Baumgart, D.C.; Sadowski, D.C.; Fedorak, R.N.; Kroeker, K.I. An overview of clinical decision support systems: Benefits, risks, and strategies for success. NPJ Digit. Med. 2020, 3, 1–10. [Google Scholar] [CrossRef] [Green Version]
- Rajpurkar, P.; Chen, E.; Banerjee, O.; Topol, E.J. AI in health and medicine. Nat. Med. 2022, 28, 31–38. [Google Scholar] [CrossRef] [PubMed]
- Corny, J.; Rajkumar, A.; Martin, O.; Dode, X.; Lajonchère, J.P.; Billuart, O.; Bézie, Y.; Buronfosse, A. A machine learning–based clinical decision support system to identify prescriptions with a high risk of medication error. J. Am. Med. Inform. Assoc. 2020, 27, 1688–1694. [Google Scholar] [CrossRef] [PubMed]
- Ragab, M.; Albukhari, A.; Alyami, J.; Mansour, R.F. Ensemble deep-learning-enabled clinical decision support system for breast cancer diagnosis and classification on ultrasound images. Biology 2022, 11, 439. [Google Scholar] [CrossRef] [PubMed]
- Massafra, R.; Latorre, A.; Fanizzi, A.; Bellotti, R.; Didonna, V.; Giotta, F.; La Forgia, D.; Nardone, A.; Pastena, M.; Ressa, C.M.; et al. A clinical decision support system for predicting invasive breast cancer recurrence: Preliminary results. Front. Oncol. 2021, 11, 576007. [Google Scholar] [CrossRef] [PubMed]
- Choi, G.H.; Yun, J.; Choi, J.; Lee, D.; Shim, J.H.; Lee, H.C.; Chung, Y.H.; Lee, Y.S.; Park, B.; Kim, N.; et al. Development of machine learning-based clinical decision support system for hepatocellular carcinoma. Sci. Rep. 2020, 10, 14855. [Google Scholar] [CrossRef] [PubMed]
- Antoniadi, A.M.; Galvin, M.; Heverin, M.; Hardiman, O.; Mooney, C. Development of an explainable clinical decision support system for the prediction of patient quality of life in amyotrophic lateral sclerosis. In Proceedings of the 36th Annual ACM Symposium on Applied Computing, Virtual Event, Republic of Korea, 22–26 March 2021; pp. 594–602. [Google Scholar]
- Wei, L.; Ventura, S.; Ryan, M.A.; Mathieson, S.; Boylan, G.B.; Lowery, M.; Mooney, C. Deep-spindle: An automated sleep spindle detection system for analysis of infant sleep spindles. Comput. Biol. Med. 2022, 150, 106096. [Google Scholar] [CrossRef] [PubMed]
- Safdar, S.; Zafar, S.; Zafar, N.; Khan, N.F. Machine learning based decision support systems (DSS) for heart disease diagnosis: A review. Artif. Intell. Rev. 2018, 50, 597–623. [Google Scholar] [CrossRef]
- Bertl, M.; Metsallik, J.; Ross, P. A systematic literature review of AI-based digital decision support systems for post-traumatic stress disorder. Front. Psychiatry 2022, 13, 923613. [Google Scholar] [CrossRef]
- Mazo, C.; Aura, C.; Rahman, A.; Gallagher, W.M.; Mooney, C. Application of Artificial Intelligence Techniques to Predict Risk of Recurrence of Breast Cancer: A Systematic Review. J. Pers. Med. 2022, 12, 1496. [Google Scholar] [CrossRef] [PubMed]
- Caruana, R.; Lou, Y.; Gehrke, J.; Koch, P.; Sturm, M.; Elhadad, N. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, 10–13 August 2015; pp. 1721–1730. [Google Scholar]
- Antoniadi, A.M.; Du, Y.; Guendouz, Y.; Wei, L.; Mazo, C.; Becker, B.A.; Mooney, C. Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review. Appl. Sci. 2021, 11, 5088. [Google Scholar] [CrossRef]
- World Health Organization. Ethics and Governance of Artificial Intelligence for Health: WHO Guidance. 2021. Available online: https://www.who.int/publications/i/item/9789240029200 (accessed on 4 October 2022).
- Panigutti, C.; Beretta, A.; Giannotti, F.; Pedreschi, D. Understanding the Impact of Explanations on Advice-Taking: A User Study for AI-Based Clinical Decision Support Systems. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April–5 May 2022; Association for Computing Machinery: New York, NY, USA, 2022. [Google Scholar]
- Antoniadi, A.M.; Galvin, M.; Heverin, M.; Wei, L.; Hardiman, O.; Mooney, C. A Clinical Decision Support System for the Prediction of Quality of Life in ALS. J. Pers. Med. 2022, 12, 435. [Google Scholar] [CrossRef] [PubMed]
- Schoonderwoerd, T.A.; Jorritsma, W.; Neerincx, M.A.; van den Bosch, K. Human-centered XAI: Developing design patterns for explanations of clinical decision support systems. Int. J. -Hum.-Comput. Stud. 2021, 154, 102684. [Google Scholar] [CrossRef]
- Hwang, J.; Lee, T.; Lee, H.; Byun, S. A Clinical Decision Support System for Sleep Staging Tasks with Explanations from Artificial Intelligence: User-Centered Design and Evaluation Study. J. Med. Internet Res. 2022, 24, e28659. [Google Scholar] [CrossRef] [PubMed]
- Bussone, A.; Stumpf, S.; O’Sullivan, D. The role of explanations on trust and reliance in clinical decision support systems. In Proceedings of the 2015 international conference on healthcare informatics, Dallas, TX, USA, 21–23 October 2015; IEEE; IEEE: Dallas, TX, USA, 2015; pp. 160–169. [Google Scholar]
- Parasuraman, R.; Riley, V. Humans and automation: Use, misuse, disuse, abuse. Hum. Factors 1997, 39, 230–253. [Google Scholar] [CrossRef]
- Wang, X.; Yin, M. Are explanations helpful? a comparative study of the effects of explanations in ai-assisted decision-making. In Proceedings of the 26th International Conference on Intelligent User Interfaces, College Station, TX, USA, 14–17 April 2021; pp. 318–328. [Google Scholar]
- Du, Y.; Rafferty, A.R.; McAuliffe, F.M.; Wei, L.; Mooney, C. An explainable machine learning-based clinical decision support system for prediction of gestational diabetes mellitus. Sci. Rep. 2022, 12, 1170. [Google Scholar] [CrossRef] [PubMed]
- Kennelly, M.A.; Ainscough, K.; Lindsay, K.L.; O’Sullivan, E.; Gibney, E.R.; McCarthy, M.; Segurado, R.; DeVito, G.; Maguire, O.; Smith, T.; et al. Pregnancy exercise and nutrition with smartphone application support: A randomized controlled trial. Obstet. Gynecol. 2018, 131, 818–826. [Google Scholar] [CrossRef] [PubMed]
- Lundberg, S.M.; Lee, S.I. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 2017, 30, 4765–4774. [Google Scholar]
- Hu, C.A.; Chen, C.M.; Fang, Y.C.; Liang, S.J.; Wang, H.C.; Fang, W.F.; Sheu, C.C.; Perng, W.C.; Yang, K.Y.; Kao, K.C.; et al. Using a machine learning approach to predict mortality in critically ill influenza patients: A cross-sectional retrospective multicentre study in Taiwan. BMJ Open 2020, 10, e033898. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Harvey, N.; Fischer, I. Taking advice: Accepting help, improving judgment, and sharing responsibility. Organ. Behav. Hum. Decis. Process. 1997, 70, 117–133. [Google Scholar] [CrossRef] [Green Version]
- Yaniv, I. Receiving other people’s advice: Influence and benefit. Organ. Behav. Hum. Decis. Process. 2004, 93, 1–13. [Google Scholar] [CrossRef]
- Hsu, K.H.; Chiu, C.; Chiu, N.H.; Lee, P.C.; Chiu, W.K.; Liu, T.H.; Hwang, C.J. A case-based classifier for hypertension detection. Knowl.-Based Syst. 2011, 24, 33–39. [Google Scholar] [CrossRef]
- Cai, C.J.; Jongejan, J.; Holbrook, J. The Effects of Example-Based Explanations in a Machine Learning Interface. In Proceedings of the 24th International Conference on Intelligent User Interfaces, Marina del Ray, CA, USA, 17–20 March 2019; Association for Computing Machinery: New York, NY, USA, 2019; pp. 258–262. [Google Scholar]
- McDermid, J.A.; Jia, Y.; Porter, Z.; Habli, I. Artificial intelligence explainability: The technical and ethical dimensions. Philos. Trans. R. Soc. A 2021, 379, 20200363. [Google Scholar] [CrossRef]
- Kenny, E.M.; Keane, M.T. Explaining Deep Learning using examples: Optimal feature weighting methods for twin systems using post-hoc, explanation-by-example in XAI. Knowl.-Based Syst. 2021, 233, 107530. [Google Scholar] [CrossRef]
- Caruana, R.; Kangarloo, H.; Dionisio, J.D.; Sinha, U.; Johnson, D. Case-based explanation of non-case-based learning methods. In Proceedings of the AMIA Symposium, American Medical Informatics Association, Washington, DC, USA, 6–10 November 1999; p. 212. [Google Scholar]
- van der Waa, J.; Nieuwburg, E.; Cremers, A.; Neerincx, M. Evaluating XAI: A comparison of rule-based and example-based explanations. Artif. Intell. 2021, 291, 103404. [Google Scholar] [CrossRef]
- Matthews, M.; Doherty, G.; Coyle, D.; Sharry, J. Designing mobile applications to support mental health interventions. In Handbook of Research on User Interface Design and Evaluation for Mobile Technology; IGI Global: Hershey, PA, USA, 2008; pp. 635–656. [Google Scholar]
- Kolachalama, V.B.; Garg, P.S. Machine learning and medical education. NPJ Digit. Med. 2018, 1, 1–3. [Google Scholar] [CrossRef]
Feature Contribution | Example | ||||
---|---|---|---|---|---|
No. of Participants | WOA Median | WOA Median | V | p-Value | |
All | 25 | 0.4 | 0.380 | 536.5 | 0.775 |
Type of HCP1 | |||||
Obstetricians | 19 | 0.5 | 0.4 | 304 | 0.654 |
Others (midwives, dietitians) | 6 | 0.367 | 0.236 | 36 | 0.824 |
Years of experience | |||||
>10 years | 12 | 0.317 | 0.314 | 113.5 | 0.958 |
⩽10 years | 13 | 0.5 | 0.4 | 164.5 | 0.790 |
Feature Contribution | Example | ||||||||
---|---|---|---|---|---|---|---|---|---|
No. of Participants | WOA Median | V | p-Value | WOA Median | V | p-Value | |||
Correct | Incorrect | Correct | Incorrect | ||||||
Type of HCP1 | |||||||||
Obstetricians | 19 | 0.6 | 0.5 | 94.5 | 0.711 | 0.333 | 0.5 | 32 | 0.066 |
Others (midwives, dietitians) | 6 | 0.45 | 0.333 | 12 | 0.281 | 0.236 | 0.271 | 12 | 0.844 |
Years of experience | |||||||||
>10 years | 12 | 0.4 | 0.317 | 44 | 0.351 | 0.25 | 0.392 | 25 | 0.301 |
⩽10 years | 13 | 0.5 | 0.5 | 39 | 1 | 0.333 | 0.429 | 22 | 0.610 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Du, Y.; Antoniadi, A.M.; McNestry, C.; McAuliffe, F.M.; Mooney, C. The Role of XAI in Advice-Taking from a Clinical Decision Support System: A Comparative User Study of Feature Contribution-Based and Example-Based Explanations. Appl. Sci. 2022, 12, 10323. https://doi.org/10.3390/app122010323
Du Y, Antoniadi AM, McNestry C, McAuliffe FM, Mooney C. The Role of XAI in Advice-Taking from a Clinical Decision Support System: A Comparative User Study of Feature Contribution-Based and Example-Based Explanations. Applied Sciences. 2022; 12(20):10323. https://doi.org/10.3390/app122010323
Chicago/Turabian StyleDu, Yuhan, Anna Markella Antoniadi, Catherine McNestry, Fionnuala M. McAuliffe, and Catherine Mooney. 2022. "The Role of XAI in Advice-Taking from a Clinical Decision Support System: A Comparative User Study of Feature Contribution-Based and Example-Based Explanations" Applied Sciences 12, no. 20: 10323. https://doi.org/10.3390/app122010323