Continuing the series on Adaptive Control:
- Why do you need Adaptive Control?
- What's a basic technology approach for Adaptive Control?
- What is Champion/Challenger?
- What is Experimental Design? - This Post
- How do I do Decision Analysis?
Experimental Design is a mature and extremely successful science dating back to the pioneering work of R. A. Fisher in the 1930s. It is designed to generate efficient experiments that will yield results suitable for accurate analysis including understanding causes of variation, predicting how changes in operating conditions would influence the outcome and the possibility of optimization to achieve a desired outcome. To apply this in Adaptive Control you must design Challengers in a systematic way.
Remember, by adding (say) a pair of Challenger strategies, you gain experience in a wider range of approaches. However, you might not pick the best areas for the Challengers - you might choose to vary aspects of the rules or models that are not the most important. Even so, you are likely to move towards the "optimal" approach in fewer steps than if you had only a single approach. Good design for the Challengers is critical, however. Consider the diagram below, which shows a danger with poorly defined Challengers. The best Challenger (Challenger 2) becomes the new Champion but there is no guarantee that it represents a move directly towards the optimal approach. In this case, it implies improvement in a direction that will result in New Challengers that are less optimal. Good Challenger design should involve a constant movement toward the optimal approach because, although an adaptive control infrastructure will correct and move you towards the optimal approach over time, you may not want to wait as long as multiple champion/challenger review cycles might take. If, for instance, you must wait a year to see how well a particular retention approach worked then you cannot afford to drift off the direct path to the best answer. Similarly if the optimal approach changes over time, and most do, then the delay imposed by a "bad" set of challengers may well be unacceptable.
In contrast, if you do a good job of designing your Challenger strategies (by using a good experimental design approach), you can expand your Challengers around your Champion in a way that maximizes the likelihood that you can get close to an optimal approach quickly. This requires both fine grained control over your Challengers and careful design of them so that you can infer likely results “between” Challengers and cover a wide range of possible variations. Below you can see how a range of Challengers can be modeled, using experimental design, around the current Champion to ensure that the new Champion is moving move directly and rapidly towards the optimal approach.
Experimental design requires a general idea of the predictive models being used in the decision and plausible ranges for the control factors. Data are then collected in the most efficient manner to provide sufficient coverage throughout the operating range of the decision, such that the model will yield accurate predictions and optimization results. Prior experience and theoretical insight into a problem help with the task to design the best experiments to allow for efficient, systematic acquisition of data. When designed appropriately, the number of Challengers or experiments required will be minimized.
I realize that this post is more of a call to action ("You should do this") and less of a how-to but it is a large subject and I don't feel qualified to go on about it at any length. There's more on the general approach to designing experiments, and a bunch of useful links, on Wikipedia.