By Shafi Rahman
How do you boil the ocean in a day? One bucket at a time. The trick is to boil multiple buckets simultaneously to significantly speed up the boiling process. There is an important concept here for all practitioners of analytics - extracting information from huge volume of data is tractable if the solution is designed intelligently.
In my previous blog, I discussed how you could gain insight into an individual’s preferences to establish a 1:1 customer dialogue. However, individuals literally sees thousands of offers a day, from buying toothpaste to taking out a new mortgage to enrolling in a college. The likelihood that any of these offers would meet an actual need is temporal in nature. An effective 1:1 customer dialogue would require one to accurately know the likelihood of any of these events happening to the customer in a given time frame. This requires processing terabytes of data and building thousands of predictive models for predicting the likelihood of occurrence of each of these events in the timeframe of interest.
Much before Big Data became a buzzword, scientists at FICO had quietly gone about creating a powerful framework to leverage terabytes of behavioral, transactional and other sources of data. The data is used to quickly create thousands of highly predictive models, and to predict the propensity of various events occurring for a given individual. These predictions have formed the basis of 1:1 customer dialogues for many FICO clients on a consistent and profitable basis for over half a decade.
Often building even a single predictive model is expensive and time consuming. FICO’s scientists achieved this feat by effectively merging the advances in analytic methodologies, our modeling tools, low-cost commodity hardware, the Linux operating system, coarse-grain distributed computing software and sound software engineering principles, in order to create highly predictive models. With the help of finely tuned processes and highly evolved quality control, FICO has been leveraging this framework for delivering thousands of predictive models every month to our clients. We have been doing this since “Big Data” was just obscure jargon, may I say, much like “boiling the ocean” is today.
