In my fraud predictions for 2016, I said that the meaning of “analytics” will continue to blur and get more confusing. Here’s why.
Increased demand for analytics has given rise to a plethora of companies suddenly claiming they understand and can “do” them. But the reality is that very few of these “analytics arriviste” companies have done the real work behind the scenes to make sure that analytic assumptions haven’t been violated. Even closer to the point, they haven’t done the important but unglamorous work of really understanding the data that drives both the decisions and the building of the analytics.
As FICO’s VP of Scores and Analytics Dr. Andrew Jennings says, “The word ‘analytics’ now means anything from advanced optimization to recording the number of cars in the car park. And because it is now so general, it has been divorced from the underlying technical considerations that were once associated with the word.”
As Andy notes, this manifests itself in things like “automated models.” “The algorithm has no knowledge of selection bias, sample truncation and so on, and so the result can be widely wrong no matter how clever the technique is,” Andy notes. “More data doesn’t solve this problem if the data all has the same limitation. For example, a model may observe that a price fall is associated with increased production, but completely miss the fact that there are no substitute products that the producer can switch to, so output increases to maintain earnings.”
Who creates bad analytics? People who aren’t well-trained — and we see more and more of those. Even as businesses “get smarter” by using more analytics, our Chief Analytics Officer Dr. Scott Zoldi says, “There is a dumbing down of analytics. Fifteen years ago we had PhDs with strong backgrounds in AI, statistics and mathematics. Today we struggle to find people with basic skills in these areas, but more people say they do ‘analytics.’ “
We hope that bad analytics get exposed and rejected by the market. Unfortunately, given the current demand, this is likely to happen after some businesses bet their future on the performance of these models. Oops.
The good news is that business folks are getting increasingly smarter about what analytics can do, how they work, and how to tell the good from the bad.
Case in point: Last spring FICO launched the FICO Academic Engagement Program, working with MBA students at San Diego State University. They participated in a semester-long program in which they used FICO’s Decision Management tools (FICO® Analytic Modeler Scorecard Pro and FICO® Analytic Modeler Decision Tree Pro) through the FICO® Analytic Cloud.
Through this program, MBA students can start to bridge the gap between the theory and the practice of analytics. “One of the shortcomings with statistical textbooks is that the data is often too clean and the relationships between variables are too perfect,” said Dr. Bruce Reinig, professor of Management Information Systems at San Diego State University, whose students were involved in the FICO Academic Engagement Program. “Students need experience working with real-world data in which values are sometimes unreliable or missing and you sometimes have to make decisions with incomplete information. The best way to accomplish this is to have students work on real problems for real organizations.”
We will see some bad analytics in 2016, no question about it. But the marketplace will be getting smarter at the same time. Watch out, wannabes.
Agree? Disagree? Tweet at me on Twitter @FraudBird.
