I recently wrote about how SnapLogic’s Hadooplex achieves elastic scalability, running as a native YARN application in a Hadoop cluster. As a I noted in the post:
“As the workload increases, the application master requests the YARN ResourceManager to spin up more Hadooplex nodes one at a time as shown in the diagram below. This scale out occurs dynamically until either the workload starts decreasing or a maximum number of Hadooplex nodes allowed has been met.
As the workload decreases, the nodes start spinning down. This is how SnapLogic achieves elastic scaling based on the workload volumes within a Hadoop cluster utilizing the YARN ResourceManager. This is possible only if an application is a native YARN application.”
I wanted to take this further by showing what this looks like in a SnapLogic Elastic Integration Platform demonstration. In this demo, you can see how how the Hadooplex, which is the run-time execution engine elastically scales depending on the workload.
You can read more about SnapLogic big data processing platforms in this paper and check out more SnapLogic demonstrations here. Be sure to also check out our upcoming webinar with Mark Madsen, which will focus on the new reference architecture for the enterprise data lake.
