Between the summer of 2011 and the summer of 2012 there was a 15,000 percent increase in analytic scientist job postings. Today, the median salary for an analytic scientist is in the six figures. By definition we're in the midst of a talent crunch. And we’re starting to see what happens at the edges when there aren’t enough quality analytic scientists or companies don't have the budget to hire one -- models behave badly.
Someone reminded me recently of an oldie but a goody from back in 2011 – the early days of the talent crunch. Michael Eisen, a biologist at UC Berkeley, touched upon this subject in his it is NOT junk blog, when one of his graduate students found two new copies of Peter Lawrence’s The Making of a Fly for $1,730,045.91 (+$3.99 shipping) on Amazon. Over the next several days, they watched the price climb to $23,698,655.93 (plus $3.99 shipping) before anyone noticed and the book returned to a reasonable $106.23.
Does this still happen today? Yes, in two extremes.
- Merchants undercut each other so much that they lose money on the transaction.
- On rare or limited items, prices get out of whack resulting in $7,000 price tags for something that is worth a few $100 at best.
So, what is happening? If you comb the Amazon discussion boards, the predominant theory is that merchants are greedy. But we know that doesn’t explain it.
- The Arbitrage Business Model enables a merchant to buy a product in one market and sell it in another. In the case of Amazon, often from another listed vendor.
- Ready access to APIs for adding and removing inventory from a site like Amazon, plus the transparent pricing, means that anyone with minimal skill can trade on any price discrepancies between marketplaces.
- Simplistic/Naive models. Poor pricing behavior happens when relatively simple assumptions without a lot of supporting analysis are made. Often the assumptions don't take a domain-specific objective into consideration. Injecting domain-specific knowledge into modeling is key to avoiding misleading predictions. Handling edge and corner cases is one of the most difficult parts of developing analytical solutions to problems, and lots of such cases can be induced when using only simple rules rather than probabilistic models to adjust prices.
- Immature markets. Without the capacity to “short” something, not to mention the inability to issue forward, future, or options contracts, speculators have little incentive to provide any immediate pricing discipline on Amazon. There’s no reason prices shouldn’t drift all over the place, irrespective of cause. It’s no big deal until somebody manages to make—or lose—some money off of it.
- Automatic execution of models requires frequent tracking and robust sanity checks. The model in the Amazon example wreaked havoc for about 51 days before one of the sellers noticed the issue and fixed it. Good analytic solutions come with better tracking than this!
This is only the tip of the iceberg. A talent crunch with healthy salaries is producing a rush of people calling themselves analytic scientists, without the proper experience or training. Have you seen any models behaving badly recently? We'd like to hear about them.