Adaptive Control: Decision Analysis

Continuing (and completing) the series on Adaptive Control:

• Why do you need Adaptive Control?
• What's a basic technology approach for Adaptive Control?
• What is Champion/Challenger?
• What is Experimental Design?
• How do I do Decision Analysis? - This Post

Decision analysis is the formal comparison of different approaches - Champion or Challenger - to see which ones work best given the organizations current strategy. It might also involve analysis to assess which will be more successful in the future. The result of this analysis might be to promote a Challenger, remove a Challenger and design a new one or to change the underlying premise for the experimental design. Ensuring that business users, especially those with responsibility for maintaining the rules in a deployed Decision Service, have the right environment to monitor the data will help them keep the decision running optimally. These users should have performance management and reporting tools that use the data in the operational systems to track how well the overall processes and systems are behaving. This kind of monitoring will help them see when they might need to make a change to a threshold in a rule or perhaps add or remove rules. The results feed into the formal analysis of Champions and Challengers.

Comparison can be of actual results to predicted results or of the results of two alternatives. Such comparisons can be at a specific moment in time or over a more extended period. Statistical analysis and standard reporting tools are all used to understand how well the deployed rules and models are working, how Champions and Challengers compare and to formulate strategies for improving things. Profit, retention and other classic measures can be tracked and compared for each challenger and the champion. Remember that you must be able to tell which approach was used for each customer/account/transaction as otherwise no comparison is possible.

One of the most effective tools other than standard reporting is a swapset analysis, like that illustrated below. This compares how different approaches or strategies apply various actions to different segments of your customer base (a classic thing to vary between champions and challengers).

You can also use standard graphs to compare the distributions and results across segments for different approaches. If you have built a decision model and are able to develop optimal solutions based on those models then you can also report your actual results against what is called an efficient frontier. This plots the best result possible for given levels of constraint - in the example below the profit per account given a change in losses. If your results map to the efficient frontier then you are doing as well as your constraints allow. Otherwise not.

Finally you might want to read the section on Decision Yield as it contains some ideas for tracking decision yield over time and the various champion and challenger decisions could be tracked using those.

adaptive control, champion/challenger, decision model, EDM, enterprise decision management, predictive analytics, decision yield, optimization, decision optimization