How to spot unintended biases in machine learning

Unintended biases in machine learning (ML) models have the potential to introduce undue discrimination and exacerbate social inequalities. Academics have proposed various technical and non-technical methods intended to assist practitioners in assessing these biases. While frameworks for identifying risks of harm due to unintended biases have been proposed, they have not yet been operationalised into practical tools to assist industry practitioners.

This blog post summarises a recent paper by Lee and Singh (2021) on the “Risk Identification Questionnaire for Unintended Bias in Machine Learning Development Lifecycle.” This paper makes three key contributions:

• Mapping prior work on bias assessment methods to phases of a standard organisational risk management process (RMP)
• Introducing a bias risk identification questionnaire as a practical tool
• Validating the need and usefulness of the questionnaire through a survey of industry practitioners.

Importantly, 86% of practitioners agree that the questionnaire helps them “to proactively diagnose unexpected issues.”

What are unintended biases?

Instead of attempting to define a contextually complex concept such as fairness, recent work has suggested it may be more helpful to identify potential biases that skew the outcome in unintended, undesirable ways. While downstream harms are often blamed on “biased data,” they arise from distinct categories of biases that each aligns to an ML development process. In each stage of model development, there are decisions made that could result in skewing of the outcome in a way that is discriminatory against certain subgroups, e.g. in data collection and labelling methods, feature engineering, etc.

Here are the six types of biases:

1. Historical bias: misalignment between the world as-is and the values or objectives required from the ML model;
2. Representation bias: under-representation or failure for a population to generalise for groups in population;
3. Measurement bias: choosing and utilising features/labels that are noisy proxies for real-world quantities;
4. Aggregation bias: inappropriate combination of heterogeneous, distinct groups into a single model;
5. Evaluation bias: use of inappropriate performance metrics or the testing / external benchmark that does not represent the entire population; and
6. Deployment bias: inappropriate use or interpretation of model in a live environment.

These biases occur through a series of decisions made that could result in skewing of the outcome in a way that is discriminatory against certain subgroups, e.g. in data collection and labelling methods, feature engineering, etc., illustrated in Figure 1. As such, instead of “fairness,” the authors refer to unintended bias with an eye to any aspects of the data, model, and processes in the lifecycle that may result in a negative impact, especially on historically marginalised groups.

Figure 1: Bias in ML development lifecycle

Mapping the bias assessment methods to organisational risk management process (RMP)

Figure 2: Bias assessment methods mapped to the ISO31000:2018 Risk Management Process (RMP)

There are numerous tools proposed to help practitioners assess the risk of unintended biases, such as impact assessments, logging templates, and mitigation strategies, but there has not yet a systematic effort to attempt to integrate these methods into an organisation’s risk management process (RMP). In this paper, the authors link each key method to the RMP in ISO 31000, a standardised organisational risk framework.

The authors then point out a gap in the risk identification process: while there have been frameworks and taxonomies introduced on bias risk types, these have not been operationalised into what can be practically implemented. This is the motivation for their paper.

Risk identification questionnaire

The authors introduce a risk identification questionnaire to help detect the risk for each type of bias in each phase of the ML development lifecycle. The questionnaire’s scope is highlighted in Figure 2. By understanding not only how the model may be biased but also why the bias exists through an explicit identification of the risk type, the questionnaire allows for a more targeted assessment of impact and design of a mitigation strategy.

A sample of the questionnaire is below in Figure 3. The full questionnaire is available here.

Figure 3: Sample snapshot of the questionnaire 

Questionnaire applied to insurance fraud

The authors use a real-life case study of predicting insurance fraud. The summary of their case study can be found below.

The first section of the questionnaire aims to understand the context for the model, especially highlighting any potential competing objectives. The model developer identified three possible positive impact of fraud prediction:

• Higher accuracy would reduce claims costs and thus offer cheaper insurance premiums
• Higher accuracy would reduce the money available to criminal groups
• Genuinely honest claims are paid more quickly and with fewer intrusive processes

The developer also identified three potential negative impacts:

• False positive: honest claimants can often feel persecuted and may withdraw their claims
• Bias: it could be seen as a deliberate bar to making insurance claims
• Representational: fraud classification may be taken as an indication of criminality and as such may reinforce biases

This allows the developer to think about the potential trade-offs between the benefits and harms of the model. For example, the developer could build multiple models and understand their impact on certain sub-groups, e.g. false positive rate for minority ethnic groups, as well as the positive impact of an accurate model, e.g. £1 million in fraud predicted.

The remaining questionnaire highlights potential biases throughout the lifecycle. Here are some examples of the different types of biases:

1. Historical bias: identification of potential criminal acts is regularly accused of racial or faith-based biases
2. Representation bias: Any claim that has not been investigated can only be assumed to be honest, and it is known that a significant percentage of non-obvious fraud is missed
3. Measurement bias: Attempts to geo-locate fraud patterns can cause unintended correlations with particular racial groups
4. Aggregation bias: One model may not be able to represent the many possible fraud scenarios may have occurred
5. Evaluation bias: The model may be over-fitting to accuracy metrics, when the relative importance of False Positive and False Negative results may vary according to business appetites and claim types
6. Deployment bias: Fraud models feed human investigators, which may continue to reinforce other model biases with the feedback mechanism

From these biases, it is apparent that the best solution to mitigate the bias risk may not be technological, but rather, should be addressed at the source with the people or the process. For example, measurement bias should be tested by assessing whether investigators are entering data reflecting their subconscious biases, e.g. flagging men vs. women as more suspicious. The developer specifically noted claimants who do not speak English to a good level may have a degraded experience and potentially result in lower data quality for their claims. If this is true, the mitigation is not at the level of the algorithm because the problem is not algorithmic. The organisation may consider staff retraining on subconscious biases or hiring staff who speak multiple languages.

See the full paper for a more complete analysis of bias types and potential mitigation strategies.

Figure 4: Case study of biases in insurance fraud prediction

Survey results of practitioners

 To validate the questionnaire, the authors conducted a survey of 105 practitioners. The results were largely positive. 90% of practitioners believe the “ability to proactively diagnose unexpected issue” is extremely/very important. The vast majority (86%) of them agree that the proposed questionnaire meets this need.

In answering what they found the most helpful about the questionnaire, practitioners commented that the “breakdown of different types of biases,” “clear structure,” “standardizing model assessment,” and “concrete concepts” are the most helpful aspects of the questionnaire, helping practitioners “think about bias in a systematic way.”

More broadly than the risk diagnosis, the questionnaire was found to give the users more familiarity with the model. 77% believe “better understanding of model risk” is extremely/very important, with 83% agreeing the questionnaire helps them achieve this goal.

Takeaways

This paper aimed to propose a risk identification methodology for potential unintended biases in ML development lifecycle, aligned to a standard enterprise risk management framework. The authors built a questionnaire and walked through a real-life use case on potential biases in an ML algorithm to predict fraudulent insurance claims. The questionnaire was validated with industry practitioners, which had a strong positive reception overall.

To ensure the end-to-end risk management of ML models and their potential to perpetuate unintended harmful biases, a targeted and systematic bias risk identification methodology is necessary. To promote adoption, risk identification methods should be easy to integrate into an organisation’s existing processes and risk frameworks and allow for appropriate mitigation strategies to be formulated.

Michelle Seng Ah Lee works in Risk Analytics leading the AI ethics offerings at Deloitte UK. She is also pursuing a PhD at the University of Cambridge on this topic in the Computer Science and Technology department. She can be reached at: michellealee@deloitte.co.uk.