Interact Lisbon - Making Your Modeling Team World Class

by FICO

Last day of the conference and I am listening to Chisoo Lyons talk about building a world class analytic modeling team. Over the last few years Fair Isaac has been focusing less on developing models for customers and more on empowering customers and their analytic teams - both with software and methodology. Chisoo has been conducting some market research into the challenges and successes of Fair Isaac customers in the area of analytic modeling.

Chisoo began by discussing typical issues in an analytic modeling team including resource constraint, problems with quality and consistency, inefficient development with too much effort not linked to business impact and a lack of visibility and objectivity into how the team invests its time. While these are pretty standard they were the basis for some survey work on challenges and how strong performers differ from average performers as well as how the business and analytic people responded.

While everyone was focused on more effective models, strong performers were more focused on distinctive skills.
Average performers were more focused on productivity and less focused on new business areas and on process automation.
Human capital was a key issue, as you would expect
Business people wanted analysts to focus more on business problems while analytic teams focused on how to find people with both analytic and business skills.
It was also widely held that efficiency was an issue but that this created a tension because analytic modelers don't want to be restricted in terms of choosing techniques and creative approaches.
As you would expect, innovation was more of an issue for the largest, most established groups.

Chisoo then put up a "vicious cycle" slide showing how a lack of resources and inefficient tools make it hard to focus on business priorities. This in turn makes it hard to get results that have a real impact as well as a lack of visibility which makes it hard to get resources and so on. Breaking this cycle means focusing on results and the results that matter are the business results, not the analytic results.

The framework Chisoo uses to manage this is the People/Process/Yield framework. For world class organizations, she identified these characteristics:

People
- Learning environment, industry standard perception, careers can be made
- Training to drive the career of analysts onward and upward
- Retention is key so rotation, especially between business and analytic roles, and mentoring are very important
- Integration with the business so that analytic team is considered part of the business solution - shared objectives
Process
- Standardization drives better quality models and requires processes, prototypes, engagement points, milestones and so on
- Technology that is usable, standardized to enable collaboration but flexible so that new ways of solving problems can be tried

Chisoo then presented some case studies and how different companies are trying to maximize their analytic, or decision, yield.

ICICI Bank (from India) who are a new bank, and a new analytic group, are focused on recruitment, training and technology as they are just getting started
They are growing fast - have gone from 25 modelers to 100 in a couple of years and from 80,000 decisions impacted a month to 730,000 decisions/month. This makes recruitment an issue and ICICI leverages their commercial brand and focuses heavily in universities presenting themselves as a great place to get started thanks to their automated training and tool investment.
HSBC UK, a very established bank, had a group focused on BASEL II. Their focus area is integration of the analytic team with the business.
Investing in training, spending more time with the business and being more focused on showing the link to business results during development and when the results are presented. Trying to generate a mind shift from statistical analyst to commercial analyst.
CitiCards, a major US credit issuer. Focus on best practices beyond the algorithms.
Looking to bring in outside expertise in best practices for problem definition, implementation, communication, expectations and so on. Not the how to build the model steps but the rest of the process. Consistency is not about efficiency but about sustaining quality across the board.

Chisoo wrapped up by discussing the wide range of tasks in predictive analytic model development above and beyond the actual creation of the model using statistical techniques - the play book for developing effective models. This needs to cover metrics, interaction points, roadmap, resources, communication protocol, decision process, presenting results and more. Formalizing this, and making it standard, makes a big difference in becoming a world class organizations. Here is a summary of some of Chisoo's slides: