Originally posted on InsideBigData.
Big data is a big deal – and a big opportunity. The challenge is that most big data progress has been limited to big companies with big engineering and data science teams. The systems can be complex, immature and hard to manage. That might be OK if you’re a well-trained developer in Silicon Valley, but it doesn’t play well if you’re a jack-of-all-trades IT leader at a bank in Atlanta or an on-the-move business leader in Amsterdam. How can you tap into big data if you don’t have an army of engineers who stay steeped in the latest tech?
Fortunately, help is on the way thanks to several trends that move tech from the foreground to the background.
First and foremost, big data is moving into the cloud. Amazon, Microsoft and Google now deliver Hadoop and Spark “as a service,” eliminating the need to spin up a cluster, research new software products or worry about version management. Companies are increasingly moving their big data workloads into the cloud, hiding complexity from their users while relying on the world’s best data center professionals to manage the tech. They want access to infrastructure when and how they want it, with just as much as they need but more.
Next up is the emergence of “serverless” computing. This builds on the cloud trend, while removing even more tech dependencies. Just load your data and start processing. Tell your cloud provider what you want to do, how much data you want to crunch and where you want to run it – and they will spin up the infrastructure when you need it, and spin it down when you don’t. That’s incredible for retailers and CPG companies, for example, who may have seasonal businesses and therefore fluctuating data needs throughout the year. They can do critical data analytics as needed without having to pay for robust infrastructure during the off-peak periods.
The third big trend is the move to self-service, fueled by new tools and platforms that democratize both data integration and data consumption. “Self-service integration” makes it fast and easy to connect systems, create data pipelines and automate processes without the need for intensive coding. Likewise, “self-service analytics” makes it easy for analysts and business users to manipulate data without IT intervention. On both fronts, self-service lowers the bar on tech skill requirements, opening the data door for millions of new users.
Last but certainly not least is machine learning and artificial intelligence, which are being used to embed more and more intelligence into applications. The systems simply learn from the data that flows through – and deliver predictive insights or recommend actions accordingly. For example, we can now offer “self-driving” integration that learns from millions of metadata elements and billions of data flows to give users expert step-by-step directions in building their data pipelines. That shortens the learning curve for business users and analysts while freeing tech teams for higher-value innovation and governance.
All four of these trends are powered by very advanced technologies, but what’s remarkable is how much they actually “hide” the tech to put more focus on the data and the business – exactly where the focus should be. It’s very similar to what we’re seeing with the incredibly intuitive consumer software that just works out of the box without a lot of training or manual intervention. Like those consumers, companies want to get more things done faster with convenience and fast access. They want cutting-edge capabilities, but they don’t want to deal with bugs and broken integrations and rapidly changing versions.
At the end of the day, most companies just want better data and faster answers – without the technology headaches. “I should just be able to write the query and get an answer back.” Thanks to the cloud, serverless computing, self-service platforms and self-learning technologies, we’re getting closer to that goal. While big data isn’t yet at the Google search or Alexa skill level of simplicity, we should all aspire to take it there.
