All posts by Richard Schiffman

Risk & Compliance Model Management Best Practices: Part 9

May112015

Welcome to our last Model Management Monday. This is the ninth and final part in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance.   Best Practice #9: Document Thoroughly With the increased use of predictive models in automated decisioning, regulators worldwide place tremendous importance on documentation and oversight. To answer regulator questions for details of how a particular model was developed, along with who approved the model at key stages of its development, you need the right tools in place to quickly retrieve the supporting evidence. With that in mind, you should keep an inventory of every model within your operating environment, cataloguing its purpose, usage and restrictions on use. Technical documentation should be captured describing the modeling methodology used, along with supporting evidence for why the final model inputs were chosen. Your documentation should be detailed enough so that... [Read More]

Leave a comment

Risk & Compliance Model Management Best Practices: Part 8

May042015

Welcome to the latest Model Management Monday. This is the seventh in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance. Best Practice #8: Monitor Overrides Anytime you override a score, regulators will require that you document and monitor that decision carefully. Your overrides should be based on clear and consistent guidelines. Regulators will ask questions such as: What is your policy for allowing an override? What authority level do you require for override approval? How many overrides are you doing every month? How do you determine that each type of override is appropriate? All overrides should be assigned an override reason code for tracking in order to evaluate an underwriter’s decisions. Use codes that allow for efficient or effective analysis. Strive to eliminate vague codes such as “general” or “miscellaneous.” Reasons for high-side overrides (accounts that score above the... [Read More]

Leave a comment

Risk & Compliance Model Management Best Practices: Part 7

Apr272015

Welcome to the latest Model Management Monday. This is the eighth in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance. Best Practice #7: Defend Decision Strategies No matter how complex your decision strategies, regulators will expect you to explain and defend them with empirical results. Regulators will want to know how you develop, track and implement your strategies. You must also show the results of your strategies, including the realized losses, gains and exposures arising from your decisions. Most importantly, regulators will want to know how you balance the need to increase profits with the need to contain risk. Carefully document all your strategy decisions, as well as changes to those strategies. You should document what your subpopulations are, what actions you’ve taken and where cutoff scores are applied. You must also demonstrate that your segments are homogeneous and... [Read More]

Leave a comment

Risk & Compliance Model Management Best Practices: Part 6

Apr202015

Welcome to the latest Model Management Monday. This is the sixth in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance. Best Practice #6: Track Performance Over time, many factors can impact model performance. These include shifts in population makeup or behavior, economic changes, impacts of marketing campaigns, and changes to credit and collection policies. Regulators expect you to monitor models on a continual basis, so you can recalibrate and rebuild them in a timely manner. Tracking outcomes is vital to understanding how well your models and business strategies are performing. This requires capturing what was known at the time of a decision, what actions were taken and what the resulting outcomes were. Automating the generation of model monitoring reports provides faster feedback about the effectiveness of your models, and makes it easier to identify and adjust for emerging trends... [Read More]

Leave a comment

Risk & Compliance Model Management Best Practices: Part 5

Apr132015

Welcome to the latest Model Management Monday. This is the fifth in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance.   Best Practice #5: Validate Model Effectiveness Once you have developed a model, you need to validate that it works according to your business objectives. You also need to revalidate on an ongoing basis—once a year at a minimum, but more often in a dynamic economy and/or where policy changes may impact model effectiveness. The validation evaluates the stability of model inputs and outputs, and measures your model’s effectiveness. As models age, their predictive power diminishes. Regular validations provide an early indication that a model may benefit from a redevelopment or realignment. Overall, you should: Strive for clarity, consistency. Regulators want to see that you validate on a regular cadence, producing a consistent collection of reports, and that your process... [Read More]

Leave a comment

Risk & Compliance Model Management Best Practices: Part 4

Apr062015

Welcome to the latest Model Management Monday. This is the fourth in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance. Best Practice #4: Choose the Right Model Type Financial institutions should select a model type appropriate for data type and decision area, and one that will provide robust predictions. For both business and regulatory purposes, you should also consider the following: Transparency. Your model type should be easy to understand and explain, both internally as well as externally to regulators and customers. Look for interpretable features that allow you to identify and explain what is driving a score result. A risk model should include reason codes, which many regulators require you give to customers when declining a request for credit. Reason codes identify the factors that had the greatest negative impact on the score. Palatability. Regulators will ask about... [Read More]

Leave a comment

Risk & Compliance Model Management Best Practices: Part 3

Mar302015

Welcome to the latest Model Management Monday. This is the third in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance. Best Practice #3: Ensure Segmentation Transparency Regulators require that you clearly document how you segmented the subpopulations within your portfolio and how you determined the unique actions you take against each. You also need to demonstrate that your segmentation supports your business objectives. Regulators will ask whether you defined your subpopulation empirically or by a domain expert, and how your segmentation fits in with your decision strategies. In some countries, you may also need to demonstrate that your segmentation does not discriminate based on age, race or gender. Basel mandates that if you segment by product, you must do it under the umbrella of Residential Mortgage (RM), Qualified Revolving Retail Exposures (QRRE) and Other Retail (OR). The key to... [Read More]

Leave a comment

Risk & Compliance Model Management Best Practices: Part 2

Mar232015

Welcome to another Model Management Monday. This is the second in my blog series on model management, each post highlighting a best practice that supports both compliance and improved performance. Best Practice #2: Prepare a Suitable Data Sample Regulators require you demonstrate your model validation sampling techniques are complete, responsible and relevant, since incorrect or inaccurate sampling can impact model performance. This holds true for both the initial validation after you develop the model, as well as your ongoing model validations. For your initial validation, the sample you use should be independent of the development sample. This can inform whether a model is over-fit to training data, and provides a more realistic benchmark for how the model is likely to perform in production. For ongoing validation of models, we recommend that you: Avoid sampling when possible. It is best to use all records from a given time period to validate... [Read More]

Leave a comment

Risk & Compliance Model Management Best Practices: Part 1

Mar162015

Welcome to the inaugural Model Management Monday. As I explained in my last post, I’m publishing a series of blog posts, with each post highlighting a best practice that supports the dual goals of compliance and improved model performance. Here’s the first in the series. Best Practice #1: Review Credit Policies Regularly Regulators will ask about your credit risk policies, since they reflect your organization’s broader objectives in terms of risk appetite. That’s why it’s important to have clearly stated policies and review these regularly. A thorough review should ask: Does each policy serve a purpose? Policies have a tendency to become part of corporate culture. Regular policy reviews ensure you are not retaining a policy when it is no longer useful in the current business environment or overlooking newly emerging requirements. Are your policies defensible? Examine your policy requirements closely to determine if they are truly indicative of risk... [Read More]

Leave a comment

Risk & Compliance Regulatory Scrutiny of Models Comes with a Silver Lining

Mar112015

There’s no doubt that increased regulatory scrutiny of predictive models has been a burden on financial institutions. But there is a silver lining: it has brought about a renewed focus on model management. And once an institution has the people, processes and technology in place for properly managing and tracking their models, it can go from merely complying with regulations to evaluating and refining model performance in ways that control losses and boost portfolio profitability. Compliance and performance – it’s a win-win. Of course, all this is easier said than done. Indeed, setting up an effective model management infrastructure remains one of the toughest challenges facing our clients, who often struggle to: Manage growing model portfolios. The sheer number of predictive models is increasing rapidly; large lenders may have thousands in production. This presents a mammoth challenge to manage and track them, and ensure they continue to perform well. Respond... [Read More]

Leave a comment