The “gift box” of ML techniques for financial services

The use of data driven analytics is rising and financial services firms are increasingly experimenting with advanced analytics and machine learning (ML) techniques to solve various business problems, generate insights from data, as well as enhance decision-making in existing risk management process. Banks’ adoption of ML techniques is moving with special caution in the space of regulated models, where the standard of proof has historically been set higher than for unregulated models.

In the UK, the Bank of England (BoE) has been signalling openness to innovation and adoption of AI/ML techniques in financial services. Their recent  Staff Working Paper on “An interpretable machine learning workflow with an application to economic forecasting” advocates the use of advanced quantitative methods to improve “traditional” statistical approaches. In this blog, we discuss how the Paper sets a higher standard of using ML techniques in an explainable way and share our experience in using ML elements to bring incremental enhancements to “traditional” credit risk modelling problems.

Setting the bar for ML use

To explore possible benefits of ML on a real-life example, the BoE Working Paper considers the problem of forecasting the US unemployment rate using a publicly available FRED-MD macroeconomic dataset. The authors pick the most relevant macroeconomic drivers of unemployment and use “traditional” statistical models to set the basis of comparison with more-advanced ML challenger models.

ML techniques are vulnerable to the perception of being a black boxes that lack transparency or explainability.  The BoE addressed explainability by calculating Shapley values that describe importance of each driver in the ultimate model.

Whilst the BoE Working Paper demonstrates application of ML approaches and explainability, banks tend to prefer incremental process enhancements over paradigm shifts in their modelling approaches. A lower-risk first step, is to introduce ML techniques to specific aspects of model selection workflows within traditional, regression-based, frameworks.

Combining ML techniques with traditional frameworks

We summarise below two case studies where ML techniques are helpfully infused into traditional workflows.

Segmentation

Within traditional logistic regression, variability in risk factors and relationship with the dependent variable (as well as variability response to exogenous factors) is commonly encountered, and overcome via segmentation. Traditionally, segmentation is aligned to intuitive customer segments such as product groups. 

As the market goes through economic cycles, different portfolio segments emerge, leading to deterioration of a generalised PD model’s performance. ML approaches, combined with domain knowledge, can be strong in identifying relevant customer segments and driving greater stability and hence longevity of segmented logit models.

Business Rules

Logit models assume an additive response to the input risk drivers. This typically results in the exclusion of risk drivers where the response is not additive, or performance deterioration in those regions of the data space where the generalised additive relationship does not hold.

In this example, model development process may start with a “traditional” logit reference model regression. This benchmark model is then challenged with a more complex and better-performing ML model. To get the best of this tandem, we identify regions of the data space where the reference model is out-performed by the challenger. The regions are filtered using a manual review for intuitiveness, to create simple and explainable business rules that override the reference model and improve its performance. Overrides to the model output are preferred to using ML for feature engineering, for their explainabilty.

Conclusions

The BoE’s paper establishes a benchmark for the standard of analysis required for mainstream adoption of ML techniques. Whilst unlikely to trigger a paradigm shift in firms’ attitudes to, and use of, ML techniques, this could be the time for firms to consider whether the incorporation of ML techniques into existing modelling workflows delivers business benefits without the maintenance and governance overheads of a full switch to ML model frameworks.