We’ve written a lot about the requirements and drivers for enterprise integration platform as a service (iPaaS) on the SnapLogic blog. Between recent industry analyst reports and a steady stream of media coverage of the importance of integration to cloud and big data success like this one: 5 Reasons Integration is Changing, I think it’s fair to say that integration is being “re-imagined” in the enterprise.
I recently wrote an article that first appeared on NetworkWorld on the topic of comparing iPaaS to legacy approaches to cloud and data integration, such as traditional ESB systems. I’ve reposted the article here for comment. As always, I’d appreciate your thoughts.
—-
iPaaS: A New Approach to Cloud Integration
Application integration has often been an exercise in frustration – long delays, high costs and over promises by vendors. How many ERP projects have you heard of that were canceled or shelved due to complex customization and integration challenges?
Integration though, is coming to a new place. Cloud technologies and open APIs are helping enterprises merge on-premise and off-premise systems without considerable coding and re-architecting. Instead of requiring specialists in SOA, enterprise service bus (ESB), extract transform and load (ETL) and data warehousing, organizations are hoping the concept of Integration Platform as a Service (iPaaS) can be used to integrate systems in half the time using technically-savvy generalists and increased involvement from lines of business.
As defined by Gartner, Forrester, Ovum and other analyst firms, iPaaS represents a new approach for enterprise IT organizations that are undergoing a rapid transition to the cloud and have big data plans.
Behind the move to more flexible, cloud integration platforms are two core trends: “cloudification,” the race to transform organizations to cloud-based architectures; and the need for agility, as business users expectations’ for rapid delivery of new web, social and mobile services continue to grow.
In our recent TechValidate survey we asked about the drivers for adopting a cloud integration platform and the top response was “speed and time to value.” Respondents also took issue with legacy, on-premises tools to address cloud integration requirements, with 43% noting the high expense of hardware and software purchases and configurations. More than a third (35%) said change management is painful in legacy tools where end point changes mean integration re-work.
Let’s look at the legacy approaches to the problem:
The Enterprise Service Bus
- ESB is a middleware architecture that was designed to manage access to applications and services and present a single and consistent interface to end-users. ESB incorporates the features required to implement a service-oriented architecture (SOA), and was appealing to enterprise IT organizations struggling with constantly changing application versions and upgrades.
- “Loose coupling” would introduce much more flexibility to application lifecycle management, it was the thought. Unfortunately for most, implementing the SOA and ESB vision was too expensive and unwieldy. IT organizations needed to install three environments (development, test and production), leading to delays.
- Secondly, ESBs were not very flexible in managing change, such as adding a new field to an end point, because of the inflexible underlying technologies. Thirdly, ESB projects required high-priced specialized integration experts. As a result, many IT organizations have continued to use the same old point-to-point enterprise application integration (EAI) methods of the past, which does not bode well for integrating the more dynamic cloud applications businesses now prefer, such as Salesforce and Workday.
ETL or Batch Data Integration
ETL is typically used for getting data in and out of a repository (data mart, data warehouse) for analytical purposes, and often addresses data cleansing and quality as well as master data management (MDM) requirements. With the onset of Hadoop to cost-effectively address the collection and storage of structured and unstructured data, however, the relevance of traditional rows-and-columns-centric ETL approaches is now in question.
XML-based Integration
Many application integration tools are XML-based, which over time has resulted in some technical shortcomings. A few of those include the fact that XML encoding tags are significant and can result in bloated payloads and the expensive overhead of repeatedly marshaling the data into and out of the document object model (DOM). What’s more, unlike JavaScript Object Notation (JSON), XML is not ideal for supporting the poly-structured information that’s becoming more common in today’s enterprise. XML-based tools were designed to handle smaller data sets at low latencies, causing problems when companies attempt to use such tools for high-volume, high-speed cloud application integration projects.
iPaaS attempts to solve many of the problems that legacy systems have not been able to do cost-effectively or within the faster requirements of agile-based development. iPaaS is a set of cloud-based services that enables both IT organizations and lines of business to develop, deploy, manage, govern and integrate applications and business systems.
Vendors provide the software and hardware infrastructure, as well as the tools for building/testing/deploying/monitoring and orchestrating integration flows. Solutions include pre-built connectors to support a variety of modern and legacy data sources and systems. While still early in enterprise IT adoption, iPaaS solutions are developed to meet the new cloud expectations of the business, built on modern lightweight and more flexible standards like JSON and REST, with the ability to scale in and out elastically when needed.
Since iPaaS abstracts the complexity, and in this case, the code, there is a perceived loss of functionality or flexibility for IT, but the trade-off is a gain in productivity.
What to Consider
In large organizations, moving to an iPaaS solution is often a step-by-step approach and many companies will retain ESB and other older architectures for a period of time as they modernize their application and data infrastructure. Here are some thoughts on how to approach iPaaS.
First, evaluate iPaaS providers for the following technical requirements:
Metadata-driven integrations vs. programmatic approaches- Drag-and-drop user experience that allows for some degree of self-service
- Pre-built connectivity (minimizing coding)
- Cloud-based management and monitoring, including comprehensive error management, transactional support, data transformation and other operations
- A hybrid deployment model that respects data gravity and allows processing to run close to the applications, regardless of where they reside.
In addition, if your organization is transforming to a cloud-based enterprise where agility is valued, be sure to dig into the platform aspects of iPaaS and ensure you don’t get locked into a technology that’s only suited for one style of integration. An iPaaS solution must be built to expose and consume micro-services and be able to handle real-time application integration, as well as the new big data integration requirements that are driving predictive analytics, digital marketing and customer-centricity initiatives in the modern enterprise.
To handle the new social, mobile, analytics, cloud and the Internet of Things (SMACT) data and API requirements seamlessly, an iPaaS needs to expand and contract compute capacity to handle variable workloads while streaming data into and out of a Hadoop-based analytics infrastructure.
While integration is certainly not a new enterprise IT challenge, there’s still a lot of old thinking and even older technology that must be reconsidered as cloud business application adoption grows and new approaches emerge to handle big data requirements. The good news is there is once again tremendous innovation happening in the integration market. As result, iPaaS is also gaining enterprise IT acceptance and adoption.
