Welcome to the latest Model Management Monday. This is the fifth in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance.
Best Practice #5: Validate Model Effectiveness
Once you have developed a model, you need to validate that it works according to your business objectives. You also need to revalidate on an ongoing basis—once a year at a minimum, but more often in a dynamic economy and/or where policy changes may impact model effectiveness.
The validation evaluates the stability of model inputs and outputs, and measures your model’s effectiveness. As models age, their predictive power diminishes. Regular validations provide an early indication that a model may benefit from a redevelopment or realignment.
Overall, you should:
- Strive for clarity, consistency. Regulators want to see that you validate on a regular cadence, producing a consistent collection of reports, and that your process is repeatable. Regulators also want to know that you have a process in place to determine when further scrutiny is warranted, and that you document the results of your investigation, including any actions you are taking (such as more frequent reassessment, recalibration or rebuilding) when a model falls below an identified threshold.
- Create a supervisory review. In the US, the OCC/Fed requires your validation processes be reviewed by parties independent of those developing the model and designing and implementing the validation process. Globally, Basel puts an equally strong emphasis on governance. An independent reviewer should have the authority to challenge model developers, so this input is considered carefully rather than summarily overruled. Be sure to capture details from each review, including formal sign-off along with specific concerns and the agreed-upon resolutions.
- Never validate in a standalone environment. Models and the scores they produce rarely operate in a vacuum; rather, they are intimately tied to business rules and decision strategies. Verifying that a model score rank-orders risk is important. But to truly understand a model’s effectiveness, you must also consider its interactions with your decision strategies. Prior to implementing a newly developed model, a best practice is to simulate the impact of the model with respect to the strategies applied to your customer portfolio.
- Include standard performance measures. Your validation checklist should include standard measures (K-S, divergence, ROC area, Gini coefficient, etc.), along with metrics that ensure the model rank-orders by score range (PDO). Measure both model performance, as well as score and attribute stability.
Regulators want you to demonstrate that you are following an approved process for validating your models, and that the results of each validation and any remediation actions are clearly documented.
For more details on this and other best practices, download the FICO Insights white paper, "Comply and Compete: Model Management Best Practices" or Martin Butler’s paper on Model Management and Governance. And check our blog next Monday for my next post in this series.
