As I engage clients about the recently released FICO® Score 9, one of the questions that comes up frequently is how effective will the score be if it is used in a more turbulent economic period. The FICO® Score 9 development sample was based on an October 2011 to October 2013 timeframe. For many lenders, this represents an extremely clean vintage as losses were extremely low. They wondered how a score developed on data that was so pristine would hold up under a more unsettled time period.
Over the last 25 years, there is an extraordinary body of work that demonstrates that the FICO® Score performs well in both good economic times and bad. Since the first FICO® Score was released in 1989, we’ve validated various versions of the score across many different points of the economic cycle and for a wide variety of industries (auto, bankcard, mortgage, etc.) for both account management and originations. In the hundreds of times we validated the score, we have seen that it successfully rank-orders risk no matter where we are on the economic cycle – a confirmation of the incredible robustness of the FICO® Score.
Let me share some of the latest research to back up my claim. With FICO® Score 9, we assessed the model’s performance on an out-of-time sample – a data sample from a different time period than what the model was developed on. Specifically, we assessed model predictiveness on a data sample from an April 2006 - April 2008 time period; the data used was the FICO® Score 8 development sample and captures the onset of the Great Recession. I’ve included the analysis for mortgage originations as that remains the poster child for an industry that has undergone dramatic change pre- and post-crisis.
Please see the trade-off curve in the chart below. For those of you not familiar with trade-off curves or lift curves, fear not. The punch line is that the curves are nearly identical, indicating that the FICO® Score 9 remains predictive even in the face of dramatically different market conditions. More importantly, this indicates that FICO® Score 9 is incredibly competitive with FICO® Score 8 when compared on a 2006-2008 data sample even though FICO® Score 8 has a significant “home court advantage” in this analysis! By “home court advantage” I mean the FICO® Score 8, unlike the FICO® Score 9, was specifically optimized on the 2006-2008 data set and you would have expected FICO® Score 8 to be the best performing by a large margin.
Trade-off curves are also referred to as Lorenz curves or lift curves. The x-axis represents the cumulative % of total accounts. The y-axis represents the cumulative % of bad accounts. As an example to interpret the above curve, by identifying the lowest 30% of all accounts, FICO® Score 9 (and FICO® Score 8) will be identifying or screening out approximately 62% of the total bads. A “high” lift curve indicates a model’s ability to effectively discriminate risk. The two curves above illustrate how close FICO® Score 9 is performing against a model optimized on that particular timeframe. We would expect to see FICO® Score 8 perform better, but for most of the score spectrum, the curves are identical indicating how well FICO® Score 9 performs on this out of time performance analysis.
These results underscore the robustness of the innovation we introduced with FICO® Score 9; the performance enhancements introduced are proven to hold up even in drastically different market conditions. Our analysis serves as strong evidence for the resiliency of FICO® Score 9. Lenders can be confident that as the economic environment changes, the FICO® Score, including FICO® Score 9, will hold up well.
