Explainable AI for Medical and Healthcare

Data

Dr Pratheepan Yogarajah

4th May 2024

 Medical/Healthcare Applications

• Nishaanthini Gnanavel
➢ (Undergraduate Final Year student at University of
Moratuwa, Sri Lanka).

• Mohan Bhandari
➢ (PhD Student, AIT, Thailand)
 4 CAMPUSES

28/122
 Intelligent Systems Research Centre
(ISRC)

Human Centred Computing

Artificial Intelligence (AI)
AI
AI is already having huge impact on
our life
 AI is already making decisions for us
Facebook AI algorithm decides which post we much see
 AI is already making decisions for us
Amazon AI algorithm recommends products we can purchase
 AI is already making decisions for us
Netflix AI algorithm recommends movies we should watch
 Machine Learning

▪ In general, the main idea behind ML is to carry out

predictions.
▪ An important second idea is to carry out inference or interpret
or explain the model.
▪ The big idea – we don’t only want to have good predictions
but also know why the model has made those predictions.
➢Why do I feel sick?
➢Why do I have high blood pressure?
 Machine Learning

captures trains predicts

AI Predictions
Real World Data
Model
Netflix recommendation

Netflix AI
Model

Why? AI Magic?
 Medical Diagnostic

Medical AI
Model

Why? AI Magic?
AI Model Understanding
 Why Model Understanding?
This model is
relying on
incorrect
Input features to make
this prediction!!
Let me fix the
model
Model understanding facilitates
Model Understanding
debugging.

Predictive
Prediction = Siberian Husky
Model
 Why Model Understanding?
This prediction
is biased. Race
and gender are
Defendant Details being used to
make the
prediction!!
Model Understanding
Race
Model understanding facilitates bias
Crimes detection.
Gender

Predictive
Model Prediction = Risky to Release
 Why Model Understanding?
Model Understanding
I have some means
for recourse. Let
Increase
me go and work on
Loan Applicant Details salary by
my promotion and
50K + pay
Model understanding
credit card helps provide pay my bills on
time.
recoursebills
to on time
individuals who are
for next 3
adversely affected
months toby model predictions.
get a loan

Predictive
Prediction = Denied Loan
Model
 Why Model Understanding?
Model Understanding This model is using
irrelevant features
Patient Data when predicting on
If gender = female,
if ID_num > 200, then sick female subpopulation. I
should not trust its
25, Female, Cold If gender = male, predictions for that
32, Male, NoModel if cold = true and cough = true,
31, Male, Cough
understanding
then sick helps assess
if and when group.
to trust model predictions when making
Predictions
decisions.
Healthy
Sick
Predictive Sick
Model .
.
Healthy
Healthy
Sick
 Why Model Understanding?

Model Understanding This model is using

Patient Data
irrelevant features
If gender = female, when predicting on
if ID_num > 200, then sick female
25, Female, Cold subpopulation. This
If gender = male, cannot be approved!
32, Male, No if cold = true and cough = true,
31, Male, Cough then sick
.
Predictions

Healthy
Sick
Predictive Sick
Model .
.
Healthy
Healthy
Sick
 Why Model Understanding?

Utility Stakeholders

End users (e.g., loan applicants)

Debugging
Decision makers (e.g., doctors,
Bias Detection
judges)
Recourse
Regulatory agencies (e.g., FDA,
European commission)
If and when to trust model
predictions
Researchers and engineers
How can we understand AI model?
Explainable AI
 Types of Data

Tabular Data Image Data Text Data

 Types of Explanations

Why did

IF Proline>=990.0 THEN Wine=1 you predict

that?

IF Color intensity<=3.85 AND

Color intensity<=3.52 THEN Because of
$Arg0$
Wine=2 Attacking
IF Flavanoids<=1.41 AND Arg1
Attacking
Proline>=470.0 THEN Wine=3 Arg3

IF Proline>=680.0 AND But this

Alcohol>=12.93 THEN Wine=1 $Arg1$ is
valid too!
IF Hue>=0.69 THEN Wine=2 That’s true,

IF TRUE THEN Wine=2

but $Arg2$
defends
$Arg0$!

Textual Form Visual Form Graphical Form Dialectical Form

 Explainable AI

Prediction
Data
Model Why?

Predictions

How?

Interpretability Explainability
(Ante-hoc (Post-hoc
explainability) explainability)
 Approaches in Explainable AI
Explainable AI

Ante-Hoc Post-Hoc
Explainability Explainability
Approach that entail baking explainability
into a model from the beginning.

For example: Linear Regression Model • Can make predictions.

• But also, determine which features are
important; which are the strongest predictors.

Coefficients tell us how correlated x1 and

x2 are with the outcome.
 Approaches in Explainable AI
Explainable AI

Ante-Hoc Post-Hoc
Explainability Explainability
• Linear Regression Model - coefficients
• Tree-based models (decision trees, etc.)
 Example of a Decision Tree
Model: Decision Tree

Splitting Attributes
Tid Refund Marital Taxable
Status Income Cheat

1 Yes Single 125K No

2 No Married 100K No Refund
No
Yes No
3 No Single 70K
4 Yes Married 120K No NO Marital Status
5 No Divorced 95K Yes Single, Divorced Married
6 No Married 60K No
7 Yes Divorced 220K No Taxable Income NO
8 No Single 85K Yes < 80K > 80K
9 No Married 75K No
NO YES
10 No Single 90K Yes
10

Training Data
 Apply Model to Test Data

Test Data
Refund
Refund Marital Taxable
Yes No
Status Income Cheat

NO Marital Status 10
No Married 80K ?

Single, Divorced Married

Taxable Income NO
< 80K > 80K

NO YES
 Apply Model to Test Data

Test Data
Refund Marital Taxable
Refund Status Income Cheat
Yes No No Married 80K ?
10

NO Marital Status
Single, Divorced Married

Taxable Income NO
< 80K > 80K

NO YES
 Apply Model to Test Data

Test Data
Refund Marital Taxable
Refund Status Income Cheat
Yes No
No Married 80K ?
10

NO Marital Status
Single, Divorced Married

Taxable Income NO
< 80K > 80K

NO YES
 Apply Model to Test Data

Test Data
Refund Marital Taxable
Refund Status Income Cheat
Yes No No Married 80K ?
10

NO Marital Status
Single, Divorced Married

Taxable Income NO
< 80K > 80K

NO YES
 Apply Model to Test Data

Test Data
Refund Refund Marital Taxable
Yes No Status Income Cheat

No Married 80K ?
NO Marital Status 10

Single, Divorced Married

Taxable Income NO
Assign Cheat to “No”
< 80K > 80K

NO YES
 Approaches in Explainable AI
Explainable AI

Ante-Hoc Post-Hoc
Explainability Explainability
• Linear Regression Model - coefficients
• Tree-based models – feature importance
• Rule based model
 Example of a Rule Based Model

Classification
Training
Algorithms
Data

NAME RANK YEARS TENURED

M ike A ssistant P rof 3 no Classifier
M ary A ssistant P rof 7 yes (Model)
B ill P rofessor 2 yes
Jim A ssociate P rof 7 yes
D ave A ssistant P rof 6 no
A nne A ssociate P rof 3 no IF rank = ‘professor’
OR years > 6
THEN tenured = ‘yes’
 Example of a Rule Based Model

Classifier

Testing
Data Unseen Data

(Jeff, Professor, 4)
NAME RANK YEARS TENURED
T om A ssistant P rof 2 no Tenured?
M erlisa A ssociate P rof 7 no
G eorge P rofessor 5 yes Yes
Joseph A ssistant P rof 7 yes
 Approaches in Explainable AI

Take 1: Build inherently interpretable predictive models

[ Letham and Rudin 2015; Lakkaraju et. al. 2016 ]

 Approaches in Explainable AI
Explainable AI

Ante-Hoc Post-Hoc
Explainability Explainability
• Methods that explain/interpret a model but are
independent of the underlying model.
• Model is a black-box
• Post-hoc: applied after the complex black-box model
has been trained.

• Some known black box models are:

• SVM, Neural Networks, Tree Ensemble, etc.
 Approaches in Explainable AI
Explainable AI

Ante-Hoc Post-Hoc
Explainability Explainability
 Deep Learning Neural Network
Training Data

Input
(unlabeled image)

Low-level Neurons
features to respond to 1st Layer
high-level simple shapes
features
Neurons respond
to more complex 2nd Layer
structures

Neurons respond
to highly complex, nth Layer
abstract concepts

Automatic algorithm
(feature extraction and classification)
https://www.xenonstack.com/
http://fortune.com/ .

Approved for public release: distribution unlimited. 41

 Approaches in Explainable AI
Explainable AI

Ante-Hoc Post-Hoc
Explainability Explainability
Advantages:
• Not dependent on the underlying
model.
• Can train more complex models
such as neural nets.
• This allows for training non-linear
models and still being able to
explain them
Tradeoff between AI accuracy and
explainability

There is an inherent tension

between machine learning
performance and explainability:
usually, the best-performing
methods are the least transparent,
and the ones providing a clear
explanation are less accurate

[ Cireşan et. al. 2012, Caruana et. al. 2006, Frosst et. al. 2017, Stewart 2020]
Tradeoff between AI accuracy and
explainability

• the most accurate methods,

such as convolutional
neural nets (CNNs), provide
no explanations;
• understandable methods,
such as rule-based, tend to
be less accurate

[ Cireşan et. al. 2012, Caruana et. al. 2006, Frosst et. al. 2017, Stewart 2020]
 Approaches in Explainable AI
Take 2: Explain pre-built models in a Ante-hoc manner

Explainer

[ Ribeiro et. al. 2016, 2018; Lakkaraju et. al. 2019]

 Approaches in Explainable AI
Explainable AI

Ante-Hoc Post-Hoc
Explainability Explainability
• Approach that allow models to
be trained normally with
explainability only being
incorporated at testing time.
• Example, SHAP, LIME, Grad-
CAM, etc.
 Post-Hoc Approaches

• Nishaanthini Gnanavel
➢ (Undergraduate Final Year student at Moratuwa University,
Sri Lanka).

• Mohan Bhandari
➢ (PhD Student, AIT, Thailand)