In Part 1 of “A Year in the Life of Big Data” I explored the last 12 months in the big data market through the lens of the Gartner Hype Cycle reports. In the space of one year it went from the peak of hype to “commonplace.” Why is that and what can we expect to see at the upcoming Strata/Hadoop World event next week?
Gartner conducts an annual big data adoption survey that may provide some insights. Gartner’s 2014 big data Investment Survey (“Survey Analysis: Big Data Investment Grows But Deployments Remain Scarce in 2014,” Nick Heudecker & Lisa Kart, Gartner, September 9, 2014, subscription required) produced the following:
“Which of the following best describes your organization’s state of big data adoption?”
- 4%: Don’t know
- 24% No plans to invest at this time
- 13%: Knowledge gathering
- 19%: Developing strategy
- 27%: Piloting and experimenting
- 13% Deployed
“What are your organization’s top hurdles or challenges with big data?”
- Determining how to get value from big data
- Risk and governance issues
- Obtaining skills and capabilities needed
- Integrating multiple data sources
- Integrating big data technology with existing infrastructure
- Defining strategy
- Funding
- Infrastructure/architecture
- Leadership or organizational issues
- Understanding what “big data” is
Characterizing big data as an over-hyped technology certainly seems appropriate if the number one barrier to adoption cited is trying to determine whether it has value to the organization. Now, let’s flash forward to the newly-released 2015 big data investment survey (“Survey Analysis: Practical Challenges Mount as Big Data Moves to Mainstream,” Nick Heudecker and Lisa Kart, Gartner, September 3, 2015, subscription required.)
This year only 14% reported big data deployment and 25% had no plans or didn’t know when they would proceed with big data investment.
Looking at the big data challenges reported by respondents, we see a reordering of responses from the prior year’s data.
“What are your organization’s top hurdles or challenges with big data?”
- Determining value (down from 65% to 55%)
- Skills and capabilities (up from 30% to 36%)
- Risk and governance (up slightly from 32% to 33%)
- Defining strategy (up from 28% to 31%)
- Funding (up from 25% to 31%)
- Integrating multiple data sources (down from 30% to 26%)
- Integrating with existing infrastructure (down from 29% to 25%)
- Infrastructure/architecture (from 29% to 22%)
- Leadership or organizational issues (from 17% to 18%)
- Understanding what big data is (down from 17% to 15%)
Determining value is still the top answer cited, but has dropped in frequency from the prior year’s survey. So, given the trend discussed in part 1 of this blog post — that big data has gone from over-hyped to ubiquitous in the space of one year — and the above survey results, I propose the following:
- The drop in “determining value” responses indicates that organizations have identified projects and use cases, but are faced with the mundane realities of funding and staffing
- Skills gaps are even more of an issue and must be addressed by making existing experts more effective
- Challenges with integrating existing data sources and infrastructure may be slowing down big data projects
SnapLogic can help with the big data integration and with making the scarce big data experts more productive by lessening effort spent on manual data integration activities. But it’s up to vendors like us and events like Strata+Hadoop World to help showcase real-world use cases as well as the technologies that enable them. The upcoming Strata event should provide valuable insights into both for attendees.
So – hope to see you in New York for the conference to see what’s new and exciting in emerging big data technologies. Or, if we believe Gartner, see what’s entirely commonplace in analytics, BI and information infrastructure. Either way, we’re looking forward to the conference.
