Extract, Transform, and Load (ETL) code has been the way developers move and prepare data from one or more sources for years. When looking at the day-to-day workings of any large enterprise, you will find 10s or 100s of thousand ETL pipelines behind the scenes that represent the glue of how information flows.
Writing ETL code is relatively simple. Plus, it’s useful in answering almost any business question (i.e. Are my sales up or down? What’s the trends with service calls? How are different manager’s spending vs budget? Etc.). With the proliferation of commercial and open-source tools available, it’s no wonder why most companies have created a large inventory of ETL pipelines.
The issue is, when migrating from an on-premise data warehouse to a cloud-native equivalent, it’s difficult to simply unplug and repoint legacy ETL pipelines.
For one thing, commercial ETL vendors don’t have an easy button. Companies are blocked in their data warehouse to cloud migration efforts when stuck on older versions of their tools – be it Informatica, DataStage, or others. While some of these vendors offer modern, cloud-based ETL versions, companies do not want to increase the risk of their migration projects by including an ETL version upgrade at the same time. So, what’s the best solution?
