Experimentation is in the news. Not surprisingly, it features large in my contributions and in the last couple of days it has also shown up as a tech-enabled business trend to watch in the McKinsey Quarterly and on James Taylor’s Decision Management blog.
Experimentation is old hat to many in the banking world, but that does not mean there isn’t room for improvement — or that industries that are coming to this topic with fresh thinking can’t get off on a better foot.
James talks about the systems and strategy required to support a culture where experimentation is the norm. By far the most important of these points is the latter. Of course you need the systems, but unless you understand the goals that investment will not yield the returns it could and should.
The investment in experimentation goes beyond the system infrastructure and people. Every experiment has an opportunity cost. One should either be doing more of it or less; the problem is that at the time you often don’t know, and so the emphasis falls on making the experiments as varied as possible and constructed in such a way as to be able to learn as quickly as possible. The benefit comes from being smarter, even in the short term, and certainly being better able to act over the medium term.
What makes a good experiment? A good experiment is one that builds on those already in place; it explores the decision space in a systematic way and allows the business to draw useful conclusions about behaviour. A good experiment isn’t one that seeks to beat any or all of those that are in place. This is a grand objective but it isn’t really achievable once you get beyond the most basic improvements. The objective is to gather data: to create variety and use the combination of the data from those many experiments and from the combined learning to restart the cycle. In the traditional language, “challengers” don’t beat “champions” one on one. Challengers only really win in combination!