Data to data-driven decision making through SnapLogic API Management

You have probably heard this a million times, but I’ll say it again. Only those decisions, whether technical or business, that are driven and backed by meaningful data, have the best chance at success. To enable organizations to adopt a data driven decision making style, APIs become critical as they act as an interface between the data consumer and a myriad of internal and external systems. Therefore, it’s important for organizations to define and invest in an API strategy. To define a solid API strategy, it is imperative to ask the right qualification questions that I had noted in my previous blog post to uncover assumptions and to propel the enterprise’s digital transformation initiatives forward. If you answer yes to any of the following questions, then you should make SnapLogic API Management, an extension of the SnapLogic Intelligent Integration Platform, your go-to solution:

1. Do you need to deliver APIs to a wide variety of users, including developers, integration specialists, architects, ad-hoc integrators, and citizen integrators (business savvy non-technical users)?
2. Does your organization have complex integration challenges that require automation?
3. Do you want a single unified API-centric platform to handle the entire API lifecycle – from creation to management to sharing of APIs?
4. Do you desire to optimize productivity while guaranteeing the efficient use of resources?

Implementing the integration use case is only the first part of a two-part solution. The second and the core aspect is to turn the implementation into an API and manage these APIs, to deliver an enhanced experience for API consumers, who may be technical or business users. The idea of leveraging APIs is to abstract the underlying implementation and only expose functionality that the API consumers are looking for.

The Business Problem

The volume of data being generated, transformed and consumed is growing at an exponential rate. While some of this data is structured, most of it is either in an unstructured or semi-structured format, leading to an increased variety and complexity of the data. Creating data taxonomy and enforcing a common data format could be a good way to manage the data variety but it is not always possible. As a result of this, organizations invest in systems and technologies that are compatible to keep up with the data’s volume/variety to solve this problem.

However, this investment only introduces an even bigger problem, which is the presence of numerous data silos, which means that data resides in disconnected silos such as databases, data lakes, data warehouses, CRM, ERP, HCM systems and so on. Additionally, numerous data silos lead to data access management and data inconsistency issues that could then lead to faulty analysis.

Here is a high-level view of what the enterprise ecosystem looks like, with a key requirement of enabling an effective medium of interoperability between the internal and external ecosystems:

The following section highlights a two part solution to improve developer productivity and experience (DX) while lowering total cost of ownership for the various digital transformation initiatives.

Two-part Solution

Part 1: Integration

Data volume is experiencing an exponential growth, so are the systems that can hold data. With every system offering a unique way to access data they hold, APIs being the most prevalent option, it becomes increasingly challenging for enterprises to build a digital and a connected ecosystem that would benefit a wide spectrum of users.

SnapLogic provides a comprehensive and a unified platform that follows a visual designer-based paradigm to meet integration requirements of enterprises. Organizations can easily interact with the various data silos, overcoming challenges around API inconsistencies and lack of documentation, and bring data into a common repository that is universally accessible across the organization. In other words, this is the integration aspect of the solution.

Part 2: API Management

Now that the integrations and APIs, whose goal is to solve a variety of use cases, ranging from data migration between and across public clouds to bi-directional data migration between SaaS applications and on-premise deployments, have been developed, it is important that API developers take meaningful steps to enable their organization’s API consumers to derive value from these APIs in a self-service manner.

Enter SnapLogic API Management. Any integration built on the SnapLogic platform can be easily turned into an API and published onto the API gateway. At the time of publishing APIs, SnapLogic developers have the option to apply a variety of API policies, ranging from security to traffic management. API developers, based on SLA requirements and API contracts,  can wrap these APIs as part of an API proxy, before opening up the APIs for intra or inter-organizational consumption.

Key API Management Capabilities

• API Gateway: SnapLogic API Management allows creation of API proxies, that allows for abstraction of backend services from API consumers. By leveraging the API gateway,  SnapLogic developers and administrators can easily manage the various aspects of the API life cycle. Here are some key highlights:
  • Management of Hybrid APIs
    • APIs deployed on Cloudplex (SnapLogic managed execution plane)
    • APIs deployed on Groundplex (Customer managed execution plane)
  • API protection and security
    • API Discoverability for consumers
    • ACL based authorization flows for requesting API access
    • OAuth2 authentication
    • Basic authentication
    • IP Restriction
    • CORS
  • API performance
    • API Throttling
    • Rate limiting
  • API Design and Creation
    • Request Transformation
    • Response Transformation
  • API Monitoring
    • API Usage
    • API Latency
  • API Documentation: Comprehensive documentation that is Open API Specification 2.0 compliant

• API Developer (consumer) Portal: Provides a frictionless experience for API consumers with self-service capabilities for API discovery and access. It also greatly simplifies the administration work for API managers that is necessary to deliver a robust integration API experience. Through this portal, developers, consumers, technical, and business users can easily access a central API repository to browse, discover, and consume APIs that were created in the SnapLogic platform, through an intuitive search interface. Another key part of this offering is the built-in API testing interface that enables API consumers to quickly validate the underlying functionality.

In the next section, I will walk you through a use case that involves developing an integration and exposing the integration as an API, to enable Marketing Operations administrators (API consumers) to quickly move from data to data-driven decision making.

Key personas:

• Anna, the API developer whose primary objectives include:
  • Establishing agile processes for API development 
  • Adopting tools and technologies that cater to both the novice and the expert API developer

• Adam, the API administrator whose primary objectives include:
  • Providing dedicated support for API development teams
  • Ensuring APIs meet SLAs defined as part of the API contracts

• Paul, the API consumer whose primary objectives include:
  • Providing data driven insights to the Marketing leadership

Key challenges:

1. Salesforce is the source system but access to it is limited to a few departments due to licensing constraints
2. Oracle database is the target system and therefore transforming the data to a relational representation is one of the most critical requirements
3. Without the ability to reuse APIs, developers spend their energy and time working on redundant  tasks; tasks that could have been easily automated through a collaborative API centric environment. Due to the  limited domain expertise that exists around design aspects of APIs, both from a functional and an implementation standpoint, lack of capabilities that promote API reusability, is another prevalent challenge

The solution:

Since data is represented uniquely across Salesforce and Oracle, one potential solution for Anna, the API Developer is to first build an integration channel between the two systems by leveraging a middleware layer such as Kafka. The image below highlights key parts of the workflow:

As part of this flow, whenever a new record is created in Salesforce by a Sales Operations administrator, an outbound trigger invokes a SnapLogic API, namely the ‘Message Producer API’ shown in the image below. This API has been designed in such a way that it will receive the incoming XML payload from Salesforce and will send an acknowledgment back to Salesforce indicating successful receipt of the payload. In addition to that, the API will also record the XML payload to ensure that the source data can be replayed and reviewed if the need for debugging arises. Finally, the API will publish the message onto the specified Kafka topic, besides marking the Salesforce opportunity as “processed.”

Next, the ‘Message Consumer API’ has been designed to deliver messages to an Oracle database, will consume messages from the Kafka topic and will acknowledge upon successful consumption. The publish-and-consume stages may be made synchronous or asynchronous through the API’s design. Finally, the API also formats the data into a relational record and inserts the message into a predefined Oracle table.

By leveraging SnapLogic API Management, Adam, the API administrator can effectively manage these APIs through the API gateway by applying additional API policies and easily publishing them onto the API gateway, thereby helping deliver value in addition to meeting SLAs and other contractual obligations. Once published, the APIs can be made discoverable on the API Developer portal with advanced authorization options to manage and govern API requests, enhancing security practices in the process.

And lastly, Paul, the API Consumer, who does not know about the existence of any of the backend systems such as Salesforce, Kafka and SnapLogic pipelines/APIs can derive value and data from the Oracle table by invoking the exposed APIs through the API Developer portal, in a self-service manner and on an on-demand basis.

Conclusion

With the power of SnapLogic API Management, available as part of the SnapLogic Intelligent Integration Platform, organizations can easily build out digital ecosystems to enable data-driven decision making across the enterprise and can effectively execute on digital transformation initiatives with improved developer experience (DX), customer experience (CX), increased team productivity and reduced total cost of ownership (TCO).

If you’d like to see SnapLogic API Management in action, you can request a demo.