Continuing the series on Adaptive Control:
- Why do you need Adaptive Control?
- What's a basic technology approach for Adaptive Control? - This Post
- What is Champion/Challenger?
- What is Experimental Design?
- How do I do Decision Analysis?
To support adaptive control, your production environment needs to support the deployment of multiple Challenger approaches to the existing Champion while business users need access to simulation tools.
The production environment must allow a Decision Service to have not just its current or “Champion” implementation but also a number of potentially better strategies – “Challengers”. It must be possible to randomly take some of the transactions and run them not through the rules and analytics that are currently defined as the Champion but through one of the Challenger strategies. You need to capture the results, both immediate and longer term, in a way that allows you to know which decisions were taken in which way (Champion v Challenger) when you come to do analysis. Performance management dashboards and Key Performance Indicators need to show both the overall average across all the approaches, the Champion’s results and the Challenger results when the measures differ across them. Thus if a Challenger strategy sacrifices retention for profitability by more aggressively dropping unprofitable customers, measurement reports and dashboards showing retention need to show the results appropriately so as not to mislead those managing the decision. In general, only a small percentage of decisions will be taken with one of the Challenger strategies, the vast majority (more than 90%) are likely to be taken with the Champion.
A simulation/testing environment must be available. This needs to have access both to historical data and to randomly generated data. If analysis of customers rejected in the past can be done to infer what they might have looked like as customers, this kind of data can usefully be added. Then, if you change your customer acquisition or origination strategy, you will be able to do some analysis to see how this might have affected your customer base. Within this simulation environment, you need to be able to do some testing of how different rules and analytics might affect results in various “what-if” scenarios. This might be as simple as running a test Decision Service against the data and seeing what results you get to as complex as running formal simulation and optimization technologies.
Business users, those who understand how the business operates and what its measures and objectives are, must be able to interact with both environments. They must be able to analyze the results both of production Challengers and of simulations. They should be able to design and run simulations for new scenarios and they should be able to design new Challengers and push them to the IT department for final testing and production deployment. This requires a combination of reporting and dashboard technology with rule management applications.