Skip to main content
Machine Learning vs. Human Expertise—Striking a Balance

With the 2013 INFORMS Annual Meeting now in full swing, I’d like to congratulate my colleague Gerald Fahner for being named a semifinalist for the INFORMS Innovation Award. His submission—titled “Imposing Domain Expertise on Algorithmic Learning to Construct Highly Predictive and Palatable Scorecards”—discusses an exciting new analytic approach that balances machine learning with human expertise.

At the heart of the new approach is the Tree Ensemble Model (TEM), a type of machine learning algorithm that holds great potential for improving predictions over today’s workhorse of scoring, logistic regression. TEMs can be quite useful in finding segmentation schemes that capture more complex customer behavior patterns with less danger of overfitting to “noise.”

But making TEM insights useful in operations requires human involvement. Domain expertise is essential to compensate for biases and “holes” in the development data and for bridging the gap to production data, which will be different and vary (often rapidly) over time. And in heavily regulated industries like banking, domain knowledge is especially critical to incorporating legal and operational requirements. In the US, for example, the Equal Credit Opportunity Act mandates that credit scoring models cannot give fewer points to applicants above a certain age, and this constraint must be built into the score formula.

To solve this challenge, we've come up with a best-of-both-worlds approach. Instead of deploying the TEM itself—essentially a “black box” of hundreds of decision trees, difficult to understand, deploy and explain to regulators—FICO has innovated a method of transmuting TEM insights into a segmented scorecard. Scorecards allow business experts to incorporate domain knowledge into predictions and customer treatments. The technique enables greater transparency and more straightforward implementation.

When we tested this approach on several custom data sets, we saw significant improvements over traditional approaches in credit risk, as well as in application fraud and other areas. We’re now using it successfully for clients outside of the lab.

For a more in-depth description of this technique, check out Gerald's post on our FICO Labs Blog. INFORMS Innovation Award finalists will be chosen later this year, and a winner will be announced next year. Best of luck to Gerald!

related posts