Explaining and Checking Fairness
for Predictive Models
Przemysław Biecek
3rd Workshop
eXplaining Knowledge Discovery in Data Mining 2021
Materials: h ps://tinyurl.com/xkdd-fairness
Slack: h ps://tinyurl.com/xkdd21
https://github.com/ModelOriented/fairmodels
tt
tt
Do algorithms
discriminate?
https://www.propublica.org/article/facebook-ads-can-still-discriminate-against-women-and-older-workers-despite-a-civil-rights-settlement
Racist Soap Dispenser
https://twitter.com/nke_ise/status/897756900753891328
Cathy O'Neil:
The era of blind faith
black boxes
in big data must end
• “You don’t see a lot of skepticism,” she says. “The algorithms are like shiny new toys
that we can’t resist using. We trust them so much that we project meaning on to them.
• Ultimately algorithms, according to O’Neil, reinforce discrimination and widen
inequality, “using people’s fear and trust of mathematics to prevent them from asking
questions”
https://www.theguardian.com/books/2016/oct/27/cathy-oneil-weapons-of-math-
destruction-algorithms-big-data
.
What does it mean to discriminate?
https://fra.europa.eu/sites/default/ les/fra_uploads/fra-2018-handbook-non-
discrimination-law-2018_en.pdf
https://fra.europa.eu/en/publication/2018/handbook-european-non-discrimination-law-2018-edition
fi
PROTECTED GROUNDS
- Sex
- Gender identity
- Sexual orientation
- Disability
- Age
- Race, ethnicity, colour and membership of
a national minority
- Nationality or national origin
- Religion or belief
- Social origin, birth and property
- Language
- Political or other opinion
https://fra.europa.eu/sites/default/ les/fra_uploads/fra-2018-handbook-non-
discrimination-law-2018_en.pdf
https://fra.europa.eu/en/publication/2018/handbook-european-non-discrimination-law-2018-edition
fi
Moritz Hardt 2020, Fairness and Machine Learning (MLSS)
https://www.youtube.com/watch?v=Igq_S_7IfOU
Is di erent treatment always
a discrimination?
ff
Sex and gender di erences and biases in arti cial intelligence for biomedicine and healthcare
Cirillo et al 2020
https://www.nature.com/articles/s41746-020-0288-5
ff
fi
Sex and gender di erences and biases in arti cial intelligence for biomedicine and healthcare
Cirillo et al 2020
https://www.nature.com/articles/s41746-020-0288-5
ff
fi
Think about the whole process
bias may be everywhere
Some sources of bias
• Historical bias. The data are correctly sampled and correspond well to the observed relationships, but due
to di erent treatment in the past some prejudices are encoded in the data. Think about gender and
occupation stereotypes.
• Representation bias. The available data is not a representative sample of the population of interest. Think
about the available facial images of actors, often white men. Or genetic sequences of covid variants, mostly
collected in developed European countries. Or crime statistics in the regions to which the police are
directed.
• Measurement bias. The variable of interest is not directly observable or is di cult to measure and the way
it is measured may be distorted by other factors. Think of the results of the mathematics skills assessment
(e.g. PISA) measured by tasks on computers not that widely available in some countries.
• Evaluation bias. The evaluation of the algorithm is performed on a population that does not represent all
groups. Think of a lung screening algorithm tested primarily on a population of smokers (older men).
• Proxy bias. The algorithm uses variables that are proxies for protected a ributes. Think of male/female
only schools where the gender e ect can be hidden under the school e ect.
partially based on Cirillo et al, 2020. https://www.nature.com/articles/s41746-020-0288-5
ff
ff
ff
tt
ffi
Some sources of bias
• Historical bias. The data are correctly sampled and correspond well to the observed relationships, but due
to di erent treatment in the past some prejudices are encoded in the data. Think about gender and
occupation stereotypes.
• Representation bias. The available data is not a representative sample of the population of interest. Think
about the available facial images of actors, often white men. Or genetic sequences of covid variants, mostly
collected in developed European countries. Or crime statistics in the regions to which the police are
directed.
• Measurement bias. The variable of interest is not directly observable or is di cult to measure and the way
it is measured may be distorted by other factors. Think of the results of the mathematics skills assessment
(e.g. PISA) measured by tasks on computers not that widely available in some countries.
• Evaluation bias. The evaluation of the algorithm is performed on a population that does not represent all
groups. Think of a lung screening algorithm tested primarily on a population of smokers (older men).
• Proxy bias. The algorithm uses variables that are proxies for protected a ributes. Think of male/female
only schools where the gender e ect can be hidden under the school e ect.
partially based on Cirillo et al, 2020. https://www.nature.com/articles/s41746-020-0288-5
ff
ff
ff
tt
ffi
Some sources of bias
• Historical bias. The data are correctly sampled and correspond well to the observed relationships, but due
to di erent treatment in the past some prejudices are encoded in the data. Think about gender and
occupation stereotypes.
• Representation bias. The available data is not a representative sample of the population of interest. Think
about the available facial images of actors, often white men. Or genetic sequences of covid variants, mostly
collected in developed European countries. Or crime statistics in the regions to which the police are
directed.
• Measurement bias. The variable of interest is not directly observable or is di cult to measure and the way
it is measured may be distorted by other factors. Think of the results of the mathematics skills assessment
(e.g. PISA) measured by tasks on computers not that widely available in some countries.
• Evaluation bias. The evaluation of the algorithm is performed on a population that does not represent all
groups. Think of a lung screening algorithm tested primarily on a population of smokers (older men).
• Proxy bias. The algorithm uses variables that are proxies for protected a ributes. Think of male/female
only schools where the gender e ect can be hidden under the school e ect.
partially based on Cirillo et al, 2020. https://www.nature.com/articles/s41746-020-0288-5
ff
ff
ff
tt
ffi
Some sources of bias
• Historical bias. The data are correctly sampled and correspond well to the observed relationships, but due
to di erent treatment in the past some prejudices are encoded in the data. Think about gender and
occupation stereotypes.
• Representation bias. The available data is not a representative sample of the population of interest. Think
about the available facial images of actors, often white men. Or genetic sequences of covid variants, mostly
collected in developed European countries. Or crime statistics in the regions to which the police are
directed.
• Measurement bias. The variable of interest is not directly observable or is di cult to measure and the way
it is measured may be distorted by other factors. Think of the results of the mathematics skills assessment
(e.g. PISA) measured by tasks on computers not that widely available in some countries.
• Evaluation bias. The evaluation of the algorithm is performed on a population that does not represent all
groups. Think of a lung screening algorithm tested primarily on a population of smokers (older men).
• Proxy bias. The algorithm uses variables that are proxies for protected a ributes. Think of male/female
only schools where the gender e ect can be hidden under the school e ect.
partially based on Cirillo et al, 2020. https://www.nature.com/articles/s41746-020-0288-5
ff
ff
ff
tt
ffi
Some sources of bias
• Historical bias. The data are correctly sampled and correspond well to the observed relationships, but due
to di erent treatment in the past some prejudices are encoded in the data. Think about gender and
occupation stereotypes.
• Representation bias. The available data is not a representative sample of the population of interest. Think
about the available facial images of actors, often white men. Or genetic sequences of covid variants, mostly
collected in developed European countries. Or crime statistics in the regions to which the police are
directed.
• Measurement bias. The variable of interest is not directly observable or is di cult to measure and the way
it is measured may be distorted by other factors. Think of the results of the mathematics skills assessment
(e.g. PISA) measured by tasks on computers not that widely available in some countries.
• Evaluation bias. The evaluation of the algorithm is performed on a population that does not represent all
groups. Think of a lung screening algorithm tested primarily on a population of smokers (older men).
• Proxy bias. The algorithm uses variables that are proxies for protected a ributes. Think of male/female
only schools where the gender e ect can be hidden under the school e ect.
partially based on Cirillo et al, 2020. https://www.nature.com/articles/s41746-020-0288-5
ff
ff
ff
tt
ffi
Explanatory Model Analysis
https://ema.drwhy.ai/
possible source of bias
possible source of bias
possible source of bias
possible source of bias
possible source of bias
Explanatory Model Analysis
https://ema.drwhy.ai/
An interesting example is the StreetBumps project.
The city of Boston released an application for mobile phones
that allows to identify potholes based on vibrations
measured by accelerometer.
It is a very innovative idea, but when analyzing such data
one has to take into account the representativeness of the
collected data.
Much more information about potholes will come from the
neighborhoods where wealthier and younger people live,
who use mobile phone more often.
https://hbr.org/2013/04/the-hidden-biases-in-big-data
Bias encoded in
data embeddings
Man is to Computer Programmer as Woman is to Homemaker?
Debiasing Word Embeddings
Bolukbasi et al, 2016
https://arxiv.org/pdf/1607.06520.pdf
Measuring Bias in Contextualized Word Representations
Bolukbasi et al 2016, h ps://arxiv.org/pdf/1607.06520.pdf
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems
Kiritchenko et al 2018, h ps://arxiv.org/pdf/1805.04508.pdf
Learning Gender-Neutral Word Embeddings
Zhao et al 2018, h ps://aclanthology.org/D18-1521/
Mitigating Gender Bias in Natural Language Processing: Literature Review
Sun et al 2019, h ps://arxiv.org/pdf/1906.08976.pdf
Reducing Gender Bias in Word-Level Language Models with a Gender-Equalizing Loss Function
Qian et al 2019, h ps://arxiv.org/pdf/1905.12801.pdf
Identifying and Reducing Gender Bias in Word-Level Language Models
Bordia et al 2019, h ps://aclanthology.org/N19-3002.pdf
Investigating Gender Bias in Language Models Using Causal Mediation Analysis
Vig et al 2020, h ps://proceedings.neurips.cc/paper/2020/ le/92650b2e92217715fe312e6fa7b90d82-Paper.pdf
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models
de Vassimon Manela et al 2021, h ps://arxiv.org/pdf/2101.09688.pdf
tt
tt
tt
tt
tt
tt
tt
tt
fi
Is it always about higher
scores?
Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classi cation
Buolamwini and Gebru, 2018
https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf
fi
Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classi cation
Buolamwini and Gebru, 2018
Lower performance in a group can generate a
problem in future data.
Think about credit lines.
If in group A loans are assigned with high
accuracy and in group B loans are assigned
with low accuracy then over time we will get
a dataset where in group A many loans are
paid and in group B not paid.
This is only because the people who were
assigned credits in group B were more
random.
https://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf
fi
Disregarding sensitive
a ributes is not a solution
tt
The risk of
,,gaming the fairness’’
https://arxiv.org/abs/2007.09969
https://arxiv.org/abs/1901.09749
Fairness measures
Notation
true class, (1 - is preferred, favourable outcome)
predicted score
decision
protected a ribute
tt
Group fairness / statistical parity / independence / demographic parity
Predicted class is independent from protected a ribute
,,four- fth rule'' - selection rate for any disadvantaged group that is less than
four- fths of that for the group with the highest rate
Sounds like a good idea, but is easy to fool.
For example, in class a we use a valid classi er while in class b we make
decisions randomly.
Perfect classi cator does not satisfy this parity if classes are not balanced.
fi
fi
fi
fi
tt
Group fairness / statistical parity / independence / demographic parity
Predicted class is independent from protected a ribute
,,four- fth rule'' - selection rate for any disadvantaged group that is less than
four- fths of that for the group with the highest rate
Sounds like a good idea, but is easy to fool.
For example, in class a we use a valid classi er while in class b we make
decisions randomly.
Perfect classi cator does not satisfy this parity if classes are not balanced.
fi
fi
fi
fi
tt
Group fairness / statistical parity / independence / demographic parity
Predicted class is independent from protected a ribute
,,four- fth rule'' - selection rate for any disadvantaged group that is less than
four- fths of that for the group with the highest rate
Sounds like a good idea, but is easy to fool.
For example, in class a we use a valid classi er while in class b we make
decisions randomly.
Perfect classi cator does not satisfy this parity if classes are not balanced.
fi
fi
fi
fi
tt
Equal opportunity
Equal True Positive Rate TPR = TP/(TP+FN) for each subgroup
If he/she pays the credit then should
have an equal chance to get it.
Predictive equality
Equal False Positive Rate FPR = FP/(FP+TN) for each subgroup
Equalized odds, Separation, Positive Rate Parity
Equal True Positive Rate TPR = TP/(TP+FN) for each subgroup and
equal False Positive Rate FPR = FP/(FP+TN) for each subgroup
Predicted class is independent from protected a ribute given true class
tt
Positive Predictive Parity
Equal Positive Predictive Value PPV = TP / (TP + FP) for each subgroup
If he/she gets the credit then should
have an equal chance to pay it.
Negative Predictive Parity
Equal Negative Predictive Value NPV = NP / (NP + NP) for each subgroup
Predictive Rate Parity, Sufficiency
Equal Positive Predictive Value PPV = TP / (TP + FP) for each subgroup and
equal Negative Predictive Value NPV = NP / (NP + NP) for each subgroup
True class is independent from protected a ribute given predicted class
tt
Whether to sentence a prisoner
Demographic parity:
the rate of sentenced prisoners should be equal in
each group
fair from society's perspective
Equal opportunity:
The fraction of innocents sentenced should be
equal in subgroups
fair from the prisoner's perspective (ProPublica)
Predictive Rate Parity:
Among the convicted, there should be an equal
fraction of innocents
fair from the judge's perspective (Northpointe)
The Fairness Trade-off (the impossibility theorem)
Except for trivial cases all these criteria cannot be satis ed jointly.
In fact each two out of {Su ciency, Separation, Independence}
are mutually exclusive.
https://fairmlbook.org/
ffi
fi
Bias mitigation strategies
Data Pre-processing
change data to improve model performance, for example, use
subsampling or case weighting
Model In-processing
modify the optimized criterion to include fairness functions, e.g.
through adversarial training
Model Post-processing
modify the resulting model scores or nal decisions, e.g., using
di erent thresholds
ff
fi
Hands on example
with the fairmodels package
Wiśniewski et al, 2020, https://arxiv.org/abs/2104.00507
The four- fths rule (Code of Federal Regulations, 1978)
"A selection rate for any race, sex, or ethnic group which is less than four- fths (4 / 5) (or
eighty percent) of the rate for the group with the highest rate will generally be regarded by
the Federal enforcement agencies as evidence of adverse impact[...]."
fi
fi
http://aif360.mybluemix.net/data
http://aif360.mybluemix.net/data
http://aif360.mybluemix.net/data
Special thanks to MI2 DataLab:
Hubert Baniecki (modelStudio)
Ewa Baranowska (drifter)
Alicja Gosiewska (auditor)
Aleksandra Grudziąż (survxai)
Adam Izdebski (describe)
Questions? Comments? Ewelina Karbowiak (EIX)
Marcin Kosiński (archivist)
Ania Kozak (vivo)
Michał Kuźba (xaibot)
Szymon Maksymiuk (DALEXtra)
Magda Młynarczyk (cr17)
Aleksandra Paluszyńska (randomForestExplainer)
Kasia Pękała (triplot)
Przemysław Biecek Piotr Piątyszek (Arena)
Hanna Piotrowska (DrWhy theme)
Adam Rydelek (xai2cloud)
Agnieszka Sitko (factorMerger)
Jakub Wiśniewski (fairModels)
https://www.linkedin.com/in/pbiecek/ … and others from MI2DataLab
