讲解 program、辅导 Java/C++程序语言
Smart Industry Operations 2024-2025
Individual Assignment
Classification, Bias, and Fairness
Context of the Assignment
The context of this assignment is the use of predictive AI tools to support decision making in
operational settings. The specific operational setting is in the criminal justice system and the case of
interest is the COMPAS system, referred to in the lectures:
COMPAS: Correctional Offender Management Profiling for Alternative Sanction
The purpose of the assignment is to understand by applying in practice
- How classification methods work and how they can be appropriately assessed
- Understand aspects of bias in AI outcomes
- Make suggestions about how such bias can be mitigated.
The COMPAS system was operationalised in the United States and was provided to the states’
justice systems by private companies. The motivation for the justice system was to have a more
efficient system in place, which would also potentially avoid human bias in making a decision. The
decision is about whether to deny or grant an offender release from prison on the basis of a risk
assessment regarding whether the specific person was at risk of offence recidivism. The risk
assessment is based on a form such as:
www.documentcloud.org/documents/2702103-Sample-Risk-Assessment-COMPAS-CORE
The COMPAS system was designed to use collected data from such risk assessments to make a
prediction about whether an offender under consideration was predicted to be at an risk or not of
recidivism.
Case Data
To perform the tasks in this assignment, you are provided with real data from the operational use
of the COMPAS system. The data set is structured as follows:
age: A numerical variable, marking the age of the person
juv_fel_count: A numerical variable, marking the number of juvenile felonies
juv_misd_count: A numerical variable, counting the number of juvenile misdemeanours
juv_other_count: A numerical variable, indicating the number of juvenile convictions that were
neither felonies nor misdemeanors
priors_count: A numerical variable, providing the number of prior crimes committed
is_recid: Binary variable, stating whether the person recidivate within 2 years (1:yes, 0:no)?
sex: Categorical, the gender of the defendant (broken down in 2 dummy variables) race: Categorical, the race of the defendant, broken down in dummy variables
c_charge_degree: Categorical, the degree by which the person is currently being charged with. The
categories are: felonies, misdemeanors, and infractions, ordered from most serious to least. These
are further broken down to charge types (felonies or misdemeanours), and in each category from
the 1st (most serious offenses) to the least severe (F1-3, M1-MO3). Finally there is an attribute for
other offences and this is a binary attribute (0 or 1).
compas_score: This is the final prediction that the COMPAS system has made for each person,
being at risk or no of offence recidivism. 1: high/medium risk, 0: low risk.
The provided datasets are:
A. recidivism-risk.csv
A complete dataset, populated with all above information. You will use this for your analysis and for
classification modelling training purposes.
B. . recidivism-risk-predict.csv
This is a similar dataset to A, but on this one you have no access to the compass_score. You will use
this to predict the compass score (classify unknown cases).
Assignment Questions
A.1. Exploratory Data Analysis (10% of Assignment 1 mark)
In this part you are expected to:
A1.1. Explore the variables, their types, and their basic statistics.
A1.2. Analyse further the data regarding data distributions, range of values, existence of outliers
and correlations between attributes, as well as between input attributes and compass_score. To
what extent is the dataset balanced regarding the different categories of sex and race and across
the age ranges?
A.2. Classification (30% of Assignment 1 mark)
In this part you are expected to develop classifier models. You will have to consider how best to use
your training data (recidivism-risk.csv) and you are asked to apply the developed models to the
“recidivism-risk-predict.csv” data.
A2.1. Apply a decision tree classifier, choosing different tree depths on the recidivism-risk.csv data.
Motivate your solution analysis in relation to overfit and generalization. Report and analyse
performance using different performance metrics. Analyse your findings. Finally, choose a
developed model and apply it to the recidivism-risk-predict.csv data to produce your predictions.
A2.2. Apply a random forest classifier. random forest regression choosing different number of
estimators and tree depths. Motivate your solution and analysis in relation to overfit and
generalization. Report and analyse performance using different performance metrics. Analyse your
findings. Finally, choose a developed model and apply it to the recidivism-risk-predict.csv data to
produce your predictions. A2.3. Apply a Naïve Bayes classifier. Report and analyse performance using different performance
metrics. Analyse your findings. Finally, choose a developed model and apply it to the recidivismrisk-predict.csv
data to produce your predictions.
A2.4. Apply a support vector classifier. Report and analyse performance using different
performance metrics for three different ‘gamma’ values. Analyse your findings. Finally, choose a
developed model and apply it to the recidivism-risk-predict.csv data to produce your predictions.
A2.5. Make a comparative analysis across all classifier experiments. Make a reasoned choice of a
classifier to select and motivate the choice referring to the evidence obtained from performance
metrics.
A.3. Bias Analysis and Management (50% of Assignment 2 mark)
In this part you are expected to further analyse the data regarding potential bias. Specifically
consider the characteristics ‘age’, and sex (female, male), and race (all categories).
A3.1.1-A3.1.4. Perform exactly the same experiments as in part A.2. but without taking into account
the ‘race’ attributes. Analyse the difference of the obtained results compared to A.2. (A3.1.1 is for
decision tree, A3.1.2 for random forest, A3.1.3 for Naïve Bayes, A3.1.4 for support vector classifier).
Consider also your observations from part A.1. regarding the distribution of data in the recidivismrisk.csv
dataset with regard to the race attributes in your analysis.
A3.2.1-A3.2.4. Perform exactly the same experiments as in part A.2. but without taking into account
the ‘sex’ attributes. Analyse the difference of the obtained results compared to A.2. (A3.2.1 is for
decision tree, A3.2.2 for random forest, A3.2.3 for Naïve Bayes, A3.2.4 for support vector classifier).
Consider also your observations from part A.1. regarding the distribution of data in the recidivismrisk.csv
dataset with regard to the sex attributes in your analysis.
A3.3.1-A3.3.4. Perform exactly the same experiments as in part A.2. but without taking into account
the ‘sex’ and ‘race’ attributes. Analyse the difference of the obtained results compared to A.2.
(A3.3.1 is for decision tree, A3.3.2 for random forest, A3.3.3 for Naïve Bayes, A3.3.4 for support
vector classifier). Consider also your observations from part A.1. regarding the distribution of data
in the recidivism-risk.csv dataset with regard to the sex attributes in your analysis.
A.4. Overall comparisons and analysis (10% of Assignment 2 mark))
In this part you are expected to:
Discuss comparatively the obtained results highlight only what you see as most interesting
regarding the obtained performance and/or aspects of data unbalance, and fairness, motivating
your analysis on the basis of the obtained evidence. What would be your concluding
recommendations?.
A.5. Bias Management and Mitigation (up to 15% extra of Assignment 2 mark, capped to
maximum assignment mark)
This is a bonus part of the assignment for the teams that aim to ensure a very high mark. In this
part you are free to work creatively on the basis of what you have seen in the lectures and
practicals. The aim in this part is to apply bias management and mitigation methods as applicable to
either the race or the sex attributes in the dataset. You are not expected to do this exhaustively but
to work on limited experiments of your choice (for example regarding one of the two sensitive attributes, sex or race). The aim of this extra ‘question’ is to trigger you into creative thinking and
action, rather than instruct you to perform a very specific task.
A general remark on grading: Note that the quality of your analysis in each step will be taken into
account. Make sure your code is well-documented and your report is readable, such that what is
done is clear and motivated. We will evaluate your contribution based on what is explained and
documented in your Jupyter notebook report.