OpenAI recently introduced GPT-4, the latest version of its language model. This iteration provides significant improvement in generation and reasoning capability over the GPT-3 models. We believe GPT-4 and successor LLMs will transform the automation of data and application integration.
At SnapLogic, our vision is to deliver frictionless integration and automation. We have helped pioneer the incorporation of AI and ML technology to streamline the integration, automation, and orchestration of data flows across the enterprises. Our Iris AI solution, trained against over petabytes of metadata, has already provided tremendous value to our customers over the last six years. We are now actively exploring the use of GPT-4 to further enhance our platform capabilities.
Large language models (LLMs) like GPT-4 can assist organizations to simplify and automate the integration of business processes, and unify data across applications into data warehouses for reporting and for descriptive and predictive analytics.
What we’ve seen so far
At SnapLogic we have been prototyping a variety of integration and automation use cases using ChatGPT (GPT-3.5) to connect intent expressed in business language with technical implementation to realize the business goal. We wanted to see how GPT-4’s increased reasoning and longer prompt support improved our integration-based use cases.
In the last two days, we have been rapidly testing and comparing our previous prompts against both GPT-3.5 and GPT-4. In addition, we have been experimenting with completely new and more demanding use cases. So far we are extremely impressed with the improved accuracy of the GPT-4 responses. It is worth noting that GPT-4 is currently more expensive per token (prompt length) and slightly slower in response generation. That said, the cost and time seem justified for the improved response accuracy.
Our interest is driven by the enormous pressure that enterprises face to increase agility while reducing costs. In our work, we find that process and data fragmentation from the simultaneous proliferation of departmental SaaS apps, the growing use of public and homegrown APIs, and the explosion of heterogeneous data from departmental business systems are some of the biggest obstacles to enterprise agility.
Use cases for AI in data integration and automation
We believe that LLMs can simplify data and application integrations in the following areas:
1. Extraction
SQL is the language of business data in both databases and cloud data warehouses. 65% of our customers’ pipelines include at least one SQL Snap. Authoring the queries within these Snaps can be a challenging process, requiring in-depth understanding of not only SQL but also the target database structures and nuances specific to each data store. GPT-4’s advanced NLP capabilities can simplify the formulation of SQL queries. With the ability to understand complex data structures and relationships, GPT-4 can help users generate more accurate and effective queries using natural language prompts. This can significantly reduce the time and technical expertise required to extract data from various sources.
2. Transformation
GPT-4 can simplify the data preparation process by automating the transformation of data from one format to another. With advanced machine learning capabilities, GPT-4 can identify patterns and relationships within the data and generate the logic required to automatically transform it into the desired format. This can significantly reduce the time and effort required for data preparation and increase the accuracy of the data.
3. Data pipeline creation
Using natural language prompts, GPT-4 has the ability to simplify the creation of data pipelines by guiding users through the process. By specifying their requirements using conversational language, users can minimize manual intervention, as GPT-4 generates the necessary processing flow, expressions, or scripts needed to create the pipeline. This can save organizations time and reduce errors in the pipeline creation process. Furthermore, GPT-4 can suggest the appropriate configurations for data connections, thereby eliminating the challenge of connecting to enterprise data.
4. Quality assurance
GPT-4 can help automate testing and quality assurance of data and app integration and orchestration processes. We have found that GPT-4 can summarize small data set samples and identify data quality issues such as data errors and conflicts. These responses can then be used to apply data cleaning transformations on a larger data set to produce high quality data for the target database or cloud data warehouse.
5. Troubleshooting
In case of any issues or errors in the data and app integration and orchestration processes, GPT-4 can help troubleshoot the problem. With the ability to summarize complex data structures and relationships as well as transform complicated system error messages into plain language, GPT-4 can quickly identify the root cause of the issue and suggest potential solutions. This can significantly reduce the time and effort required for troubleshooting and increase the efficiency of the system.
6. Insights
GPT-4’s advanced NLP capabilities can help users get insights against a subset of enterprise data by allowing them to ask questions of the data in natural language. The ability for non-technical subject matter experts to pose questions in their terms can significantly reduce the time and effort required for data analysis and increase the accuracy of the insights.
We believe that GPT-4’s advanced NLP, machine learning, and data processing capabilities offer significant opportunities to simplify and automate data and app integration and orchestration processes.
Continuing the conversation: what’s next?
At SnapLogic, we are excited to see the potential of GPT-4 and look forward to exploring its capabilities further as we continue to deliver our long-standing vision of frictionless integration and automation.
We are moving fast, and are confident that by collaborating with the SnapLogic community we will be able to deliver a new experience that is transformational in terms of time to value — delivering results to the business at both the top and bottom line.
Special thanks to Greg Benson, Chief Scientist, and Manish Rai, Vice President, Product Marketing at SnapLogic for their thoughts and contributions to this developing discussion.
To continue the conversation and learn more about these advances please watch the Reveal Event: The Future of Integration with Generative AI now available on-demand.
