Delta Lake and Talend

Modern data architecture with Delta Lake and Talend

Read Article

About Wavicle Data Solutions

Delta Lake and Talend

Modern data architecture with Delta Lake and Talend

Read Article
Delta Lake and Talend

Modern data architecture with Delta Lake and Talend

Read Article
Delta Lake and Talend

Modern data architecture with Delta Lake and Talend

Read Article

Wavicle Case Study

McDonald’s Accelerates Migration from Legacy ETL Platform to Talend by 80%

McDonald's

When a division of McDonald’s, the leading foodservice retailer, decided to migrate its legacy ETL platform to Talend, it worked with Wavicle’s data engineering experts to migrate data from IBM DataStage to Talend.

Talend Cloud migration

Data Engineering
Data Integration
AWS
IBM DataStage
Talend

Challenge

The McDonald’s organization responsible for global restaurant operations improvement captures and analyzes data about everything from food safety to speed of service and customer satisfaction. This data from over 37,000 restaurants across 108 markets worldwide, was stored in an on-premises IT ecosystem. To reduce its data center costs and improve flexibility and scalability of its data management environment, this group decided to migrate it’s entire on-prem IT ecosystem to Amazon Web Services (AWS).

The move to the cloud prompted the company to migrate its data integration tool as well. Concerned about diminishing support for DataStage, they enlisted Wavicle’s ETL migration services to migrate to Talend.

Solution

Wavicle’s ETL migration experts approached the work in three phases:

  • Analysis: Analyzing, categorizing, and scoping the work
  • Development: Converting ETL jobs from DataStage to Talend
  • Testing: Testing of output across both platforms to compare results

Our team leveraged ETL conversion accelerators and proprietary testing tools to automate and accelerate these time-consuming tasks and dramatically reduce the development time. To get started, they simply fed legacy ETL components to the ETL Conversion accelerator for analysis.

In this case, the analysis of ETL jobs identified a large inventory of DataStage components. Working closely with the group’s business analysts, we identified backup or obsolete jobs that could be eliminated from the conversion project. This left nearly 300 jobs to be analyzed and ranked for their level of complexity based on parameters such as number of nodes, objects, occurrences, SQL statements, system types used, and transformation expressions.

Using our ETL Conversion tool, we migrated these jobs to Talend and then used our Data Capturer tool to compare output across the legacy ETL and Talend platforms. These tools automated 80% of the development effort, requiring manual intervention only for the most complex jobs. Throughout the effort, we collaborated closely with McDonald’s cloud migration team to ensure the environment was set up on AWS for accurate testing.

Results

Using Wavicle’s accelerators and frameworks to automate the analysis, conversion, and testing tasks, we successfully converted all ETL jobs to Talend, saving 95% of the analysis work and 81% of the development effort.

ETL Conversion tool reduces analysis and development time

Without ETL Converter With ETL Converter Time Savings
Total number of jobs 283 Total number of jobs 283
Estimated analysis hours 1,056 Actual analysis hours 801 95%
Estimated conversion hours 10,864 Actual conversion hours 2,0772 81%

Additional value of the project includes:

  • Licensing cost savings: Move to Talend reduces ETL software licensing costs by 75%[3]
  • Ongoing software support: Access strong Talend support team and community vs. reduction in DataStage support
  • Integration: Talend provides seamless integration to other global data analytics solutions like RedShift, Tableau, Adhoc extracts etc.
  • Easier maintenance: Reduces costs of developer rates by cutting the time nearly in half to build, pull, and extract data elements with fewer errors than manual ETL
  • Reduced cycle time for enhancements: Developers have the ability to tap into the core products of Talend and use community-supported and created plugins to match any specific ETL need
  • Reduced infrastructure costs: Eliminating need for on-premises infrastructure saves data center operations and maintenance cost

[1] This effort denotes the time taken by converter to generate the detailed analysis reports. Additional effort spent to validate and document the results.

[2] This effort is for the overall conversion activity. Small amount of additional effort will be required to tweak the complex category jobs.

[3] Based on the published pricing

Download Case Study PDF