In the Harvard Business Review article “Good Data Won’t Guarantee Good Decisions,” Shvetank Shah, Andrew Horne and Jaime Capellá argued that “Big Data” won’t live up to its promises to lead to better business decisions, unless complemented with educated human judgment. It divided decision makers into three groups:
- Unquestioning empiricists who trust data analysis over judgment
- Visceral decision makers who rely on gut feeling
- Informed skeptics who balance judgment and data for their decisions
My clients and associates, who use predictive analytics to improve business decisions, tend to identify with the third group. Although we may at times feel attracted to one or the other extreme position, depending on the type of problem and data conditions.
While the article didn’t go into the guts of predictive analytics, there are obvious relations to practices for developing predictive models for consumer behavior. Various modeling decisions have to be made, including choice of a model type, data transformations, predictor selection and fit metrics. Do we want to leave all decisions to a computer? How much influence should domain experts exert on model construction? What is an effective “informed skeptic” approach to modeling consumer behavior? What are useful procedures for combining data-driven learning with educated judgment and subtle domain knowledge?
In my talk at FICO World I will discuss how we at FICO have embraced a pragmatic engineering approach for balancing data analysis with domain expertise. For example, decision makers from credit scoring may be familiar with the notion of the “weights engineering meeting." As a young project analyst, I learned how to engineer deployment-ready scorecards by listening closely to our clients’ wisdom and concerns, not just to their data. We would engage stakeholders from risk, marketing and IT and work together on an initial model design, tweak it, learn about sensitivities, thereby balancing the fitting of historic relations with task-related knowledge. With this approach, models become guided, but not driven, by the data. Opinions are expressed and discussed, reasoned judgment enters - the informed skeptics’ approach to modeling. The computational underpinning for performing this model refinement effectively is a numeric procedure for constrained optimization of designer objective functions within a class of generalized additive prediction models, also known as scorecard technology.
What’s more important than the technicalities is the principle and why it improves decisions. Besides resolving legal and operational constraints for credit scoring models, incorporating domain knowledge can be crucial to mitigate selection biases and speed up learning from data. It can inform and stabilize extrapolations into areas of prediction space where historic data support may lack, but where future operations may move.
Domain expertise is in demand when historic data lack representativeness. Reject inference is a classic example. This holds true not just for building scorecards, but even more so when developing action-effect models to predict causal effects of treatment decisions on future outcomes – an important and challenging task in the development of decision models. Aside from dealing with representativeness issues, there is another reason why domain knowledge should be embraced – learning a new task from empirical data generally works faster and better when knowledge from similar tasks can be transferred to learning the new task. This is the upshot from a lesser known area of machine learning called “transfer learning”. These fundamental benefits of combining data with expertise can accrue for any predictive modeling project, not just for credit scoring.
Fast forward 16 years from my first weights engineering meeting and the data universe has expanded dramatically with no signs of abating. “Big Data” and rapid shifts in consumer-business dynamics are in the headlines. In unison, powerful machine learning algorithms and high performance computing became attractive for data scientists. We can and should leverage the automation and scalability of hard-nosed machine learning tools in our quest to improve predictions. We can and should gain deeper insights into predictive relationships than what would be feasible with a tedious manual approach to data analysis.
New combinations of Big Data and machine learning will affect the balance between judgment and data. How far the pendulum may swing into the direction of the empiricists may be speculated about. My expectation is that for the foreseeable future, inclusion of domain knowledge into predictive models will continue to provide a competitive advantage. But how do all the new opportunities served up by machine learning square with the informed skeptic approach to modeling?
FICO Labs is pursuing exciting new developments. We developed a method of transmuting tree ensemble models into engineerable segmented scorecard systems that can be easily inspected, engineered, and deployed. Our approach productively combines machine learning and human expertise. It can learn the utmost from data, while retaining expert control over predictive relationships expressed by deployable models. By combining flexible and objective data-driven predictions with the capability to impose domain knowledge, the empiricist in us can rejoice that the specter of burying empirical evidence under biased, restrictive assumptions is averted, while the visceral decision maker in us retains the last word over what will be deployed.
If you care about this topic and would like to learn more, including examples of applications for credit scoring and insurance fraud, please join us for our FICO World 2013 session on “Machine Learning and Human Expertise - A Better Way to Engineer Predictive Models for Big Data”.
