Skip to main content
Myth Busters: The Analytic Talent Crunch Will Constrain Big Data Innovation

Help Wanted
By Benjamin Baer

Earlier this month, we posted our first in a series of myth busters, inspired by the Discovery Channel’s television show MythBusters. Over the next several months we’ll tackle hot topics related to Big Data, analytics, customer engagement and mobile technology and we’ll determine whether the topic can be confirmed, is plausible or is busted (not true). 

Thomas Davenport and D.J. Patil brought the data scientist into the national spotlight in last October’s Harvard Business Review article titled Data Scientist: The Sexiest Job of the 21st Century, and reported that job postings for analytic scientists have jumped 15,000 percent between the summer of 2011 and 2012. Many believe the “shortage” will only get worse. McKinsey & Company predicted a 50 to 60 percent shortfall in analytic scientists in the US by 2018. And Gartner echoed this sentiment predicting that only one-third of 4.4 million global big data jobs will be filled by 2015.

By definition we have a talent crunch on our hands. But will it constrain Big Data innovation? We say no…busted. Here is why:

  1. Analytic technology will get simpler to use.  Over the next few years we will see reduced complexity in collecting, processing, analyzing and acting on Big Data. Big Data Analytics is not immune to Moore’s Law.  Over the last several decades processors became faster, computing power became cheaper, chips became smaller and applications became easier to use. Simple always wins, even with Big Data Analytics. There will be an explosion in tools geared for the business user, there already are a few available today…
  2. Machine learning and analytic solutions will do much of the heavy lifting.  More and more applications like customer engagement, debt management, and customer management will include fully integrated analytic capabilities. This along with improvements in machine learning will enable businesses to handle the analytic “drudgery at scale” and free their analytic scientist to focus on the high value projects.
  3. Cloud technology will deliver Big Data Analytics into more hands. Just as Cloud revolutionized ERP and CRM, it will change the dynamics of Big Data Analytics. It will give more people access to the powerful (and simpler) tools and solutions, and massive amounts of data – it will make Big Data palatable.
  4. People will become more comfortable with analytic tools, therefore freeing the analytic scientist to focus on the most complex projects. As tools become simpler, and more accessible they will become second nature to business professionals and IT.
  5. Data will no longer need to be pristine. Much of the time and expense in the analytics process today involves confirming that the data sample is accurate and useful. As FICO World keynote and Economist Data Editor Kenneth Cukier put it, Big Data by its nature is messy, and does not require the same devotion to precision and accuracy as when dealing with smaller data sets. When using large sample sizes of billions and billions of records (structured and unstructured), often messy is good enough.

While the analytic talent crunch is very real, it will not have as deafening an effect on Big Data innovation as some pundits would you have you believe. In the not too distant future, the analytic scientist will be able to focus on the most complex projects, and the business user will be able to competently handle the rest. Did we convince you?

related posts