Should you trust “click-click-click” modeling?

Many of today’s announcements around analytic innovation focus on the need to use analytics to crunch through petabytes of data. That’s true, but it’s only part of the story. Given today’s dynamic market conditions, businesses need analytics to make decisions for which they don’t have petabytes of data—or at least not petabytes of relevant data. That’s why you need to rely on analytic experts to get the most predictive value from what you have to work with.

Here’s an example of what I mean. You can make the performance of a credit approval model look wonderful in the development stage depending on how you assign future performance to “rejects”: those individuals who didn’t pass your past approval screens, and therefore whose performance is not captured in your development sample. Since improving operational performance depends on the “swap set”—who will you accept that you didn’t previously and vice versa—you need to estimate the unknown performance of the rejects, a process known as reject inference You can’t do this automatically. It requires a judgment call. Being too optimistic will make the model look great and perform badly. The reverse will lead to lost opportunity. 

I don’t think I’m going too far by saying that the real power in modeling has always come from the analytics expert, and this will not change. Expertise makes the difference between models that perform so-so and those that perform at a very high level. This performance edge comes from the expert’s ability to interpret nuances in the data in order to find the best predictive characteristics for a desired performance outcome. It comes from knowing how to validate models without “over-fitting” them to the data they were developed from, and how to fine-tune models to a company’s specific real-world business conditions.

Above all, the analyst’s understanding of the context in which a model will be used—and the decision it’s supposed to improve—is absolutely critical. I don’t believe you can get that by going click-click-click.