Whenever I drive past a retailer that sells my company’s equipment, I am reminded of the immense impact my team is having in delivering products on time and in sufficient quantities to retailers around the world and it gives me a lot of joy! But my mind is brought back to our daily realities. Top of my mind is last Friday’s meeting where we were debating whether to upgrade to DataStage 11.7.x.
Weekend anxieties
The upgrade to the DataStage version 11.7.x will bring in some minor upgrades, but it also brings a lot of headaches. Since upgrades are complex with sparse documentation, I will have to pull my most experienced developers from an important business-critical project. We will have to make upgrades on all the servers running DataStage.
Once the upgrade is done, we will have to do extensive regression testing to verify that our existing integrations are still working as expected. This will take a few weeks. And then after all that, we will have to upgrade the thick clients on every developer’s laptop. At best, it will consume 3 to 4 weeks of my team’s time. Based on our prior experience, it might even take more than 2 months. With a growing backlog, I am in no position to stop new integration projects for months. It will have to wait!
Monday morning, 8:35 AM
The week has started with another challenge. Snowflake Summit was two months ago and they released a few amazing features such as the general availability of the Iceberg tables support and four new geospatial functions. Iceberg tables are great since we can leverage external compute engines on our Snowflake data. This helps us bring our data into an open format and gives us more flexibility in picking the right computing platform for the use case.
My team is really looking forward to incorporating those features into our environment but after a quick email exchange with DataStage’s customer success team, it looks like DataStage will not be supporting those features any time soon.
Monday morning, 11:30 AM
I just had a follow-up meeting with our VP of Finance. We are on the verge of a brand new acquisition in the European market. We will be adding quite a few customers and significant revenue from that acquisition, so we will have to spin up a new instance of our SAP S/4HANA ERP system and get data from that instance and other systems-of-record into our cloud data warehouse.
We are already at capacity of our DataStage servers, so I sent out a query to our DataStage sales representative. The rep told me that while we can add capacity, it won’t be cheap. If we use our budget for additional DataStage capacity, we will have to deprioritize our other IT investments in the second half of the year. Given how critical this acquisition is for the company, I don’t see any way around it!
Monday afternoon, 1:15 PM
Just checked on my Reddit post on a problem that my team is trying to solve with DataStage. No answers yet! I guess I will have to wait a few more days, maybe weeks?
Monday afternoon, 4:00 PM
Before I finish my day, I need to sync up with my HR partner, I have been planning for the impending departure of some of my experienced DataStage developers. I wanted to review a short list of candidates with her. I am looking to see what experience the candidates have and what their salary expectations are.
Sigh! My HR team has found some great experienced candidates but their salary expectations are high! I am in a pickle here! I wish to have clarity on our future middleware strategy so that I can make a firm decision on this hire. But without that clarity, I might have to pay up for the scarce DataStage talent.
The cycle continues
Managing DataStage for integrations is a continuous balancing act between managing complex upgrades, ensuring seamless delivery of new projects, and finding ways to adopt new features with this legacy product while dealing with daily operational challenges. What will tomorrow bring? Stay tuned for my next installment.
Learn more about how IBM DataStage compares with SnapLogic
