I've been blogging about the science behind who gets a FICO® Score and the three critical analytic questions that must be answered before scoring more consumers. In this post, I'll discuss recent FICO research addressing the third question: Is the relationship between risk and score consistent for potentially scorable and traditionally scorable consumers? (See my posts for question 1 and question 2.)
This consistency is essential to effective credit scoring, and it's what we model developers mean when we ask whether the “odds-to-score relationship” is appropriately aligned. In layman's terms, when two consumers with two completely different profiles have the same score, it should represent the same level of risk.
Here’s the challenge for scoring more consumers: Our research has shown that this consistent odds-to-score alignment doesn’t hold true for some currently unscorable consumers when compared with the larger traditionally scorable population.
Take, for example, credit files with no account reported in the last six months—which we consider to be “stale” files because they have no recent data. As I explained in a prior blog post, these consumers don't currently receive FICO® Scores. In our study, we relaxed the minimum score criteria and applied the current FICO Score algorithm to these consumers with stale files, in order to compare the odds-to-score relationship of this segment to traditionally scorable consumers.
We observed a troubling pattern, shown in the graphic below. Consumers with stale files typically show a flatter odds-to-score relationship. This means the score applied to these consumers is less effective at assessing risk. And the staler the credit file, the flatter the odds-to-score line.
Let me break down the graphic. The odds-to-score relationship for traditionally scorable consumers—with at least one account updated in the prior six months—is shown as “Reported 0-6 months.” For consumers with credit records last reported 7-20 months prior, the odds-to-score relationship is flatter than the baseline. For consumers with credit records 21 months and older, the odds-to-score relationship is flatter still. While the graphic shows results for auto loans, we saw similar odds-to-score patterns for bankcard and mortgage loans.
These results make logical sense. Over time, a consumer’s financial situation may change. Some consumers will see their financial status improve, some will experience problems, and some will remain stable. For traditionally scorable consumers, we have relatively fresh data available at the time of scoring, and strong evidence to trust its reliability. But for consumers where no account information has been reported for six months or longer, any possible change in financial status has gone unreported, and the data associated with the corresponding credit scores have become unreliable. As our research shows, the older the data, the less reliable the score.
An astute modeler would realize that we could potentially overcome this by developing a separate, specific scorecard for these consumers. This approach may yield better odds-to-score alignment with the traditionally scorable population.
However, our research showed that this approach was also not analytically sound. Consider once again consumers with stale files. Only 13% of these consumers in our study had classifiable performance during the subsequent 24-month performance window. That means nearly nine out of ten of these consumers would be excluded from the development of the scorecard. As I discussed in a prior blog post, it is not reasonable to assume that the risk patterns of such a minority would hold true for the larger group.
So do you accept a flatter odds-to-score relationship for these newly scorable records, or a model built on a highly truncated and biased population? Neither outcome is desirable if your business decisions rely on a meaningful credit score. Tweaking an algorithm to score more consumers is easy. Getting the risk prediction right requires more sophistication.
Stay tuned to the blog as we present additional research on the issue of scoring more consumers. To read all our latest research on this topic, I encourage you to download the new Insights white paper: To Score or Not to Score? (registration required).