I've been highlighting questionable modelling practices by some retail banks under the auspices of Basel II compliance, as well as offering alternate best-practice guidance.
Today, in my third entry in the series, I’ll focus on the misuse of the Gini coefficient measure, which generally falls into two categories:
- “Chasing the Gini”
Working with one client, for example, we were advised that they had increased the Gini by over 20 points on a new model. However, when the model was implemented, there was no improvement seen in business performance. As it turns out, the increase stemmed from impacts very high up the score range on a minority of applications, resulting in little overall impact on the credit risk decisions made.
All measures have their pros and cons, and as part of my best-practice recommendation, be sure you consider:
- A range of predictive measures when developing a model, such as divergence, K-S and KS score point/percentile, R-squared, etc.
- How the model will perform when implemented, looking at trade-off curves, etc.
- That you’re not “over-fitting” the model to the development data. Instead, focus on developing robust models that will perform well over time.
- Other factors such as model interpretability and implementation costs.
- Setting of minimum Gini performance thresholds
This highlights a lack of understanding by the internal governance team that such measures can be influenced by the model design. Education on such factors is crucial to the successful review and validation of models.
If you missed my earlier posts in this series, be sure to check out my first and second entries. All the banks I’ve been blogging about have Advanced IRB status and experienced credit risk managers. That suggests one overall takeaway from this blog series: while Basel II compliance and credit risk management complement each other, one cannot be substituted for the other. Getting them to work together leads to better credit risk assessment and increased profitability.
For additional modelling best practices, I once again encourage you to download these two FICO ebooks: High-Performance Predictive Analytics and Comply and Compete: Model Management Best Practices (registration required).