A Day in the Life of a DataStage Manager Part 2: Vision

Last week was a pivotal point in our use of IBM DataStage. Many insignificant issues have become massive roadblocks that need to be addressed. I need some clarity from our IT leadership and enterprise architecture team so that we can chart our path forward. I am looking forward to today’s planning meeting to discuss the IT vision and the future of our data platform.

A new week, with a vision for Enterprise IT

Today, we had a great meeting with the CIO on the vision for our enterprise tech stack and all-important data platform. We will continue to leverage cloud for the majority of enterprise applications but the CIO also expects that we will have a hybrid infrastructure for the foreseeable future. 

With this vision in mind, I raised the question of our aging middleware platform and the challenges we are facing with the upgrade, connectivity, lack of capacity, and costs. It was great to see that our CIO is fully committed to the vision and wants us to explore options so that we can modernize our technology stack. 

He also made it clear that we are not tied to a specific vendor and the decision will be made with the company’s best long-term interests in mind. 

Tuesday morning: adding DataStage capacity

While it is great that we are talking about the vision for the tech stack, we’ll still have business issues to resolve in the near term. The DataStage rep got back to me on adding more capacity for integrations with the new ERP instance — and it is not looking good. I knew we were in a long-term contract but I did not know it was a 7-year term! 

We are locked into our existing compute capacity for the next 2 years and we are already at 85% load. This is a critical use case, so we will incorporate it with our current capacity and stay within budget. But, that means we will have to use the rest of the capacity judiciously in the second half of the year. 

Tuesday afternoon: discussion with an enterprise architect

Now that we have our marching orders, I partnered with another manager whose team is using IBM DataStage and we set up a meeting with enterprise architects across our teams. Both reiterated our current challenges with DataStage and solicited feedback on the “must-haves” as we seek a new solution. 

From this highly interactive session, we came up with the following key criteria that we will look for in a new platform:

1. A cloud-native tool that also supports hybrid deployments
2. Connectivity to the latest version of modern applications and cloud data warehouses
3. A modern tool that would support modern APIs and data formats
4. Enterprise-grade scalability and security

We know that DataStage has a cloud version. But Is it any good? Will that meet our needs? With 2 years left in the existing DataStage contract, these discussions are happening at the right time. What other tools in the market can help us automate data pipelines and business processes? 

Making progress

I think we’re getting closer to a solution. We have a list of essential criteria for an integration platform to enable our vision of scale, but this is also an opportunity to add GenAI to our technology stack. Could there be a modern integration platform that meets our key criteria and enables us to advance productivity and automation initiatives with generative AI? Time to see what’s on the market.