ETL Migration
Migrating your Legacy ETL Pipelines to Run in the Cloud – Automatically
For data warehouse migration projects, companies are already faced with an array of complex questions, tasks and decisions to address. Where will I migrate to? How do I jump-start the migration effort? How long will this take, and how long will this cost?
For companies reliant on third-party ETL tools, another question comes up – what do I do with my legacy ETL pipelines?
Writing ETL code is relatively simple. Plus, it’s useful in answering almost any business question (i.e. Are my sales up or down? What’s the trends with service calls? How are different manager’s spending vs budget? Etc.). With the proliferation of commercial and open-source tools available, it’s no wonder why most companies have created a large inventory of ETL pipelines. But how do you migrate your ETL pipelines to the cloud?
With the SHIFT™ Migration Suite we’ve automated the process to provide customers with the fastest and most cost-effective method to migrate ETL pipelines to the cloud.
1. Automating Planning & Assessment Activities
First, we leverage SHIFT™ CRAWLER to automatically scan the vast inventory of legacy ETL pipelines to uncover the lineage and source-to-target data flow to find out:
- What pipelines are feeding my data warehouse? And how many are there?
- What source systems are feeding my data warehouse?
- Which pipelines are not only feeding the data warehouse in-scope for migration, but also other systems?
By defining the lineage and inventory of all in-scope pipelines, we provide customers with quick, actionable insights to plan their migration path to the cloud, without risk of impacting business operations or other dependant systems.
2. Automating Code Translation
Then, we leverage SHIFT™ TRANSLATOR to translate all of the legacy ETL code to run natively against the cloud target. Specifically, SHIFT™ TRANSLATOR will automatically:- Translate all embedded SQL to the syntax of the target cloud platform (be it Snowflake, Redshift, or others).
- Convert all XML objects, including source/target connectors, tables, and orchestration logic to run against the target cloud platform
By automating this process, the SHIFT™ Migration Suite provides customers with an end-to-end automated solution to get legacy ETL pipelines running on the cloud fast, efficiently, and without impacting current business processes.
3. Managing and Accelerating Cut-Over
- SHIFT™ TESTER will automate key aspects of code and data validation testing when completing the migration project, in order to accelerate end-to-end testing efforts. This ensures all translated code compiles and executes with the expected results in the cloud target.
- SHIFT™ CRAWLER will identify dependencies to downstream applications like BI or Analytics tools, in order to define the appropriate migration approach while identifying optimization and consolidation opportunities along the way.
