As machine learning models have become entrenched in industry, attention has turned to their ethics and auditability. Whilst traditional models such as a SVM with a linear kernel are explainable, they are not complex, and thus cannot capture some of the non-linear relationships other machine learning models do. Recent machine learning models, such as neural networks, are powerful due to their complexity, yet it’s this very characteristic that makes it almost impossible for humans to understand how they work. Why has a machine learning model decided a loan should be approved for a certain person? The model would likely have hundreds of inputs, all mapped in such intricate ways that millions of parameters are involved in any decision. A human cannot process such vast quantities of non-linear and sometimes invertable information, making the process a black box. Without being able to comprehend why a model has made a decision means it’s difficult to assess for bias, be that from the human who made the model or the dataset itself. Regulations aiming to tackle the unauditable nature of models are being enforced, such as GDPR’s right to an explanation of any automated decision. It is clear data scientists need to consider not just how they make a model that performs the best, but one that is also explainable.      

When using sophisticated machine learning models such as deep learning or ensemble methods, the field of eXplainable AI (XAI) aims to create algorithms that make complex models understandable by humans. Largely, this is done at the local level. This means XAI algorithms aim to explain why a single decision was made: why was one person’s loan application accepted or rejected? Instead of being presented with the millions of intricately linked parameters that have been used to come to this decision, the feature inputs are given a weighting of how significant they were to the decision. For example, for a loan application, a person’s credit score could be shown to be important, whilst the neighbourhood they live in was deemed to have no bearing. If their neighbourhood was shown to be significant, this may highlight ethical limitations of the model.

XAI has been hailed as a way to make the unexplainable explainable and has recently become a popular research topic, driven by ethical and regulatory demands. If machine learning models can be explained, humans will have more control of the automated systems that they are building. However, research into XAI is in its infancy. The focus has been on making algorithms that work rather than integrating their evaluation into the framework that data scientists use. Much like a machine learning model is evaluated for its accuracy, XAI methods should always be evaluated for their performance. Although not knowing why a machine learning model came to a decision is ethically dangerous, it is more dangerous to be sure a decision was made for the wrong reasons.

There is an array of ways XAI methods are designed. Some work through surrogate models, such as a popular algorithm called LIME. This algorithm builds a very simple linear model around a single decision point. The model is simple enough that a human can understand which inputs have been significant. Other XAI methods use game theory to work out the marginal contribution of inputs, such as SHAP. Both of these methods lead to approximations of the complex machine learning model and thus are very unlikely to be 100% accurate. Despite it being known that XAI methods aren’t infallible, there are limited methods out there to determine the robustness of XAI. XAI is simply too new to have a sophisticated framework available to assess its accuracy, like machine learning models do.

So, what can we do to make sure the XAI methods we use aren’t giving misleading results? A simple test that a data scientist can perform is to remove the inputs that have been deemed significant by the XAI method and rerun the machine learning model. If the XAI was correct, the machine learning model’s prediction should change. If the prediction is similar, then the input highlighted by the XAI method did not have a material impact on the prediction and therefore the XAI method was not accurate. In addition, XAI methods are deemed to be robust if they give similar results for similar examples. For example, if two people with very similar financial information and therefore loan decision are explained, the same inputs should be highlighted as significant. If they are not, potentially the XAI method is uncertain on the significant inputs and therefore should not be trusted.

Evaluating XAI can also be more intricate than the techniques described above. Different XAI methods require different evaluation techniques. LIME, for example, builds surrogate models that can be evaluated for their faithfulness to the machine learning model. SHAP has many parameters that must be altered to see the effect they have on the explanations. There are more XAI methods that can be counted, and each one requires its own bespoke evaluation techniques, which relies on knowledge of how the XAI works. This is a difficult task that requires in-depth knowledge and experience of XAI methods.

To summarise, XAI methods are crucial to being able to make auditable and ethical machine learning model pipelines. But this is an area of research in its infancy: general evaluation methods have not yet been created. It is at the discretion of the data scientist to determine how to evaluate the accuracy and robustness of XAI methods. Whilst machine learning model frameworks with sophisticated evaluation are making machine learning model development easier to be robust, XAI methods are far from this level of maturity. It is only through careful examination and comprehension of XAI methods through which a data scientist can be sure the employed XAI is robust and representative of the complex machine learning model being explained. No good data scientist would deploy a machine learning model without thorough evaluation, nor should they deploy XAI methods in the same manner. The only difference between the two is the way in which we evaluate XAI methods is largely undefined. Defining the evaluation will be the continued focus of research in the near future. Until then, XAI methods should not be accepted as accurate until evidence is provided by the data scientist to suggest so, using some of the techniques detailed in this blog.