Skip to main content
Are You Really Drowning in Data? Challenging the Big Data Assumption

By Josh Hemann

There is a lot of talk within the Big Data blogosphere about companies being awash in data and not knowing how to use it. (Try an internet search on “awash in data” and you’ll see what I mean.) These discussions assume copious data are sitting there, not being taken advantage of, and what follows are typically discussions of some new hardware, software or mathematics that will change that. I am not sure which companies these bloggers are referring to, but they certainly cannot be the ones I have worked with in Retail and CPG, as both a consultant as well as being in the trenches on a marketing team. In my experience in these settings, while potential data may be copious, what is actually available to work with is a small subset.

This oft-stated (and unstated) Big Data assumption is simply unfounded in Retail and CPG settings, but why? To answer this, it helps to remind ourselves of how most retailers’ IT departments and capabilities have evolved over the years: IT has been formed around the procurement, replenishment and distribution of merchandise. But much of what has made data become Big is because myriad human behaviors are now being measured and persisted. So, while there has been a lot of talk about customer-centricity and using Big Data to understand one’s customer base better, the fact is for years (decades) the questions that have been asked of data, and the efforts to answer them, have been almost exclusively merchandise-related. Further, these efforts are typically championed and funded by stakeholders on the Merchandise side of the org chart, not marketing or customer-focused parts of the org.

The side-effect of this evolution is that there are typically key pieces missing in the hardware, software, and analytic talent ecosystem that prevent meaningful data-driven decision making on customers. For example, even if customer-linked transaction data are stored they may not be aggregated in a way that makes subsequent analytics feasible (e.g. queries are too time-consuming and difficult to code or execute routinely). This reality begs a proverbial question: even if tons of data exist somewhere in a corporate IT department’s ether, if it is not easily accessible and usable, does it really exist? And does it exist in a way that can wash someone over?

There is no doubt we have more instrumentation and data that at any time in history. But most Retail and CPG companies are not drowning in it. In a sense drowning would be a good problem to have, because the life jacket would be a mix of the hardware, software and mathematics discussed in the blogosphere. Instead, the problem at this point is one of vision: how can we elevate customer understanding to the same level of importance as merchandise, with commensurate investment in process and optimization?

We can all name a select few companies that have mastered this elevation. How long will it take the majority of Retail and CPG companies to catch up to the blogosphere’s view of Big Data? In coming posts, we'll look at some concrete steps to connect (big) data to consumer-level business objectives.

related posts