SnapLogic today announced support for the Amazon Redshift console partner integration, a tool comprised of select Amazon Web Services (AWS) Partner solutions that have integrated with Amazon Redshift. With the new console partner integration, customers can accelerate data onboarding and produce valuable business insights in minutes.
SnapLogic is an AWS Partner Network (APN) Advanced Technology Partner with the AWS Data & Analytics Competency and Redshift Service Ready status, which further simplifies an organization’s move to a fully-managed data architecture.
By supporting the Amazon Redshift console partner integration, SnapLogic now makes it easy for AWS customers to quickly move data from hundreds of applications such as Salesforce, Workday, ServiceNow, Google Analytics, Facebook Ads, Slack, Jira, Splunk, and Marketo to an Amazon Redshift data warehouse in an efficient and streamlined way. The Amazon Redshift console partner integration allows customers to identify the partners relevant to their specific use case. Once a partner is selected, users are transferred to a dedicated landing page on the partner website, where sign-up and configuration can be completed.
This post will take you through detailed steps on how to automatically deploy an Amazon Redshift cluster, integrate SnapLogic from Amazon Redshift console for the cluster created, and get started with your first SnapLogic pipeline to ingest data into the cluster. These steps include:
- Create an Amazon Redshift cluster using CloudFormation
- Integrate the Amazon Redshift cluster created with SnapLogic from AWS console
- Sign up for SnapLogic trial and verify integration established to Redshift Cluster
- Run a pipeline, ingest data into Amazon Redshift cluster and verify it
Prerequisites
Here is the minimum required setup to complete the hands-on tutorial:
- Before starting this tutorial, you need to have an AWS account. In this exercise, you configure the required AWS resources using AWS CloudFormation in the us-west-1 region. If you haven’t signed up, please sign up for AWS here.
- If you need to connect your existing Amazon Redshift cluster with SnapLogic, you will first need to enable public accessibility for your cluster as well as whitelist SnapLogic’s ips for connectivity.
Note: CloudFormation below creates a new redshift cluster and automatically whitelist SnapLogic IPs
Creating an Amazon Redshift cluster
Now let’s go ahead a get started by creating a Redshift Cluster that we can integrate with SnapLogic. We have used the following AWS CloudFormation template to create a two-node Amazon Redshift instance running on dc2.large type instance by default (you may choose other instance types and node counts as well).
Click Launch Stack to initiate CloudFormation stack creation in the us-west-1 region.
On the Create Stack page, update RedshiftPassword, RedshiftNodeCount and RedshiftNodeType if needed or use the default, acknowledge the resource Capabilities and click Create Stack. Wait for the stack creation to complete, which can take up to 10 minutes to complete.
Once stack creation is completed, you can navigate to Amazon Redshift Console and see the cluster that just got created with the status listed as available.
Integrate the Amazon Redshift cluster created with SnapLogic from AWS console
Click on the Cluster Name as shown above so that you can see the details and verify them. After quick verification while on the cluster details page, click on the Add Partner Integrations button on the top.
On the Choose Partner page, scroll down and select SnapLogic. You can also find some details around common use-cases where SnapLogic helps customers. Scroll to the bottom and click Next.
On the Integrate with SnapLogic Page, verify the cluster details and click Add Partner button which will open the SnapLogic trial page for sign-up.
Sign-up for SnapLogic trial and automatically configure the Amazon Redshift Connection
On the trials page which opened up, provide your details including name, work email, job title, company name, etc. and agree the terms (If you have already created a SnapLogic trial account with your work email in the past, use +1 at the end of your alias to sign-up in order to ensure everything works smooth for you). There is also a box named Redshift Details where you should provide your cluster password that you updated during CloudFormation stack creation or the default password: Awsuser01.
Once all details are entered, click Start Your Free Trial button. This page has transferred all your cluster details and automatically set up a connection as well as a pipeline to help you get started quickly.
Once signed up, you would see a thank you page indicating instructions to get started have been sent to your email. You should have an email in your inbox showing you how to set up a password and login to the SnapLogic UI. Complete sign up by following the instructions.
Now let’s navigate to Amazon Redshift Console and click on the cluster you just added partner integrations to. Navigate to the Properties tab and scroll down to the Integrations section and see that SnapLogic is listed as an Active connection on the console indicating this Redshift cluster now has integrations to your SnapLogic trial environment that you just signed up for.
Now login to the SnapLogic UI after verifying your email address and changing your password. Once on the console, Navigate to Manager tab ???? Project Spaces ???? Redshift Console ???? “Username_Org” as shown below:
You will notice an account and a pipeline created automatically for you. Click on the Redshift-Trial-Account and verify it is pointing to the cluster you just created above. You can click Validate to ensure SnapLogic can connect to your Amazon Redshift cluster.
Run a pipeline, ingest data into Amazon Redshift cluster and verify it
Now let’s run a pipeline which is created and configured automatically for you to quickly experience the flexibility SnapLogic provides to ingest data into Amazon Redshift.
While on the Redshift Console page, click on ForecastAPI-to-Redshift pipeline, which will open the SnapLogic Designer user interface that has the pre-configured pipeline that pulls alert data from National Weather Service and ingests into Redshift. The pipeline looks like the below:
To execute the pipeline, click on the Execute Pipeline button on the top right corner:
This triggers the pipeline and starts ingesting the data into the Redshift cluster. Once execution is completed, the pipeline turns into Green.
Once execution is completed, navigate to Redshift Query Editor and provide connection details to the cluster if needed and connect it.
| database name: | snaplogic |
| user name: | snapuser |
| password: | <your custompassword> [default Awsuser01] |
Select the SnapLogic Database from dropdown, Schema sample_data, right-click on weather table, and select Preview data, and you should be able to see the data ingested by the SnapLogic pipeline.
With this, you have successfully tested and executed your first pipeline on the SnapLogic trial and ingested data into an Amazon Redshift cluster which is the last step in this hands-on exercise.
Clean Up
When you finish the steps and tested any additional pipelines, remember to clean up all other AWS resources you created using AWS CloudFormation to avoid incurring additional infrastructure costs. Use the CloudFormation console or AWS Command Line Interface (CLI) to delete the stack named SnapLogic-Redshift-Console-Blog.
Conclusion
In this post, you learned about Amazon Redshift’s new partner Integration capability from its console which allows customers to identify the partners relevant to their specific use case, create an Amazon Redshift cluster using CloudFormation, and integrate it with SnapLogic from Redshift console. Once directed to the partner page, you signed up for the SnapLogic trial and verified the integration established to Redshift Cluster, ran a SnapLogic pipeline, ingested data into Amazon Redshift cluster, and verified it.
To learn more about building a data warehouse with SnapLogic on AWS, get started with the AWS Quick Start.
