Posted by Dr. Andrew Jennings, FICO Chief Research Officer and Head of FICO Labs
Credit issuers are very familiar with the concept of champion / challenger testing — putting two decision strategies into market to see which one performs better. But in a rapidly changing market, there’s another way to use challenger strategies: to gather data.
In a high-performing analytic organization, a certain proportion of challengers will be purposefully designed to produce “controlled variation.” If you test only challenger strategies that are close to how you currently do business, you will limit what you can learn from your data. If, however, you push the design of some challengers outside of the bounds of business as usual, you’ll introduce variation into your data and expand what you can learn from it.
In a practical situation, it is never as simple as the challenger beats the champion. Well-designed strategies don’t even set out with this goal in mind. They set out to push the relationships that underlie the decision model so that learning increases understanding. Think of them not as challengers to replace a champion, but as learning strategies. This cycle, in turn, leads to a new champion and another round of testing. By exploring a wider range of possibilities, you increase the chances of discovering unique insights that might lead to competitive differentiation, and you’re also less likely to be caught flat-footed when the forces of change shake up the status quo.
