Migration to a unified data framework

Pilot Company Transforms Data Ecosystem to Unified Data Platform

Pilot Company relies on their data platform to support everything from travel center reporting to ensuring compliance with critical fuel regulations. With support ending for IBM Netezza, their long-time data storage solution, they needed to evaluate their migration options and select new solutions to achieve their business and technical goals.

Their primary objective was to develop a flexible environment with cloud-agnostic services and reporting layers. They also wanted to minimize infrastructure costs by significantly reducing data processing runtimes and overall computing expenses. Additionally, they aimed to ensure accurate reporting, maintain compliance with critical fuel regulations, and mitigate penalties and reputational risks.

Project goals:

• Decrease infrastructure costs by increasing speeds and optimizing compute costs
• Ensure reliable executive reporting
• Mitigate risk and maintain compliance with industry regulations
• Reduce latency of critical data pipelines from 8 hours to 4 hours, with a long-term goal of 30 minutes

Pilot Company already used AWS as a part of its ecosystem. The smooth expansion of AWS and implementation of a Snowflake cloud data platform perfectly aligned with the company’s needs. There was also an opportunity to further optimize infrastructure costs with Databricks by improving performance through distributed computing capabilities while decoupling storage and compute costs. 

Wavicle’s team worked with Pilot to leverage Amazon S3, Databricks, and Snowflake to create a well-integrated data processing and analytics ecosystem. Here’s a breakdown of how they serve individually:

• Amazon S3 acts as a scalable storage solution for raw and processed data 
• Databricks functions as a powerful computing engine for data transformation and analysis tasks 
• Snowflake serves as a reporting data platform, facilitating valuable business insights 

Together, this setup provides a cost-effective and efficient data solution with robust reporting capabilities. It also reduces costs by optimizing storage utilization, intelligently allocating resources, and decoupling storage and computing costs.

Amazon S3 implementation

First, data was transitioned from IBM Netezza to Amazon S3, making it Pilot’s primary storage platform. Amazon EMR was used to build data pipelines, simplifying migration. This has set up Pilot Company for future growth, ensuring scalability and flexibility for growing data storage needs. 

Databricks implementation

Next, as a pivotal step in the migration process, Databricks was used to support CI/CD pipelines and serve as Pilot’s computing engine. Concurrently, Databricks Spark SQL automated query optimization, eliminating the need for tuning SQL scripts manually. This resulted in significant cost savings, with expenses reduced to a fraction compared to Netezza.

The Databricks implementation also played a crucial role in fortifying data governance. Leveraging Delta tables and ACID transactions ensured data integrity, security, and compliance with critical fuel regulations. This strengthened the reliability and trustworthiness of the data infrastructure. 

Snowflake implementation

In addition to the primary components, Snowflake was included as an auxiliary unit to prepare data for reporting in Tableau dashboards, strengthening decision-making processes. Snowflake proved instrumental in streamlining data preparation for faster and more accurate dashboards. This improvement was highly beneficial for business leaders who needed timely insights into operational performance.

The integration of AWS, Databricks, and Snowflake successfully established a unified data solutions framework for the business. In addition to improving decision-making and facilitating faster and cost-effective analytics, this collaboration offered scalable storage, robust computing capabilities, and optimized data querying. Together, these platforms built a future-proof, efficient, and agile data ecosystem, empowering the organization to evolve in this changing data landscape.

New data solutions framework paves the way for a resilient data ecosystem

The unified data platform supports more than 50 terabytes of data, uses 500+ jobs, and supports 2,000+ users across Pilot Company. By streamlining data management and ensuring accurate, collaborative, and secure data access for decision-making, the company experienced significant improvements in the following areas:

• Refresh time to process information was reduced by 8x  
• Databricks optimization resulted in 7x cost savings  
• Databricks Spark SQL at scale eliminated the need for optimizing SQL scripts   
• Delta table ACID transactions improved data governance and data integrity 

The successful implementation of a unified data platform that included AWS, Databricks, and Snowflake helped Pilot Company reach their goals. More efficient data processing and analytics reduced reporting timelines, supporting faster insights and informed decision-making. The new framework also ensured compliance with key regulations and improved security and governance.  

This cohesive approach provided flexibility for dynamic scaling, simplified operations, and future-proofed the organization’s data ecosystem, positioning the Pilot Company for continued success in its data-driven journey.  

About Pilot Company

Pilot Travel Centers LLC (“Pilot Company”) keeps North America’s drivers moving as one of the leading suppliers of fuel and the largest operator of travel centers. Founded in 1958 and headquartered in Knoxville, Tennessee, Pilot Company has grown its network to more than 870 locations serving an average of 1.3 million guests per day and providing over 70,000 fleet customers with solutions for fuel, credit, factoring, services and rewards. Pilot Company has the third largest tanker fleet in North America with over 1,500 trucks that supply approximately 14 billion gallons of fuel per year and provide hauling and disposal services to the oil field sector. As a fuel and energy leader, Pilot Company is the largest provider of biodiesel and renewable fuels and is paving the way to the future by developing an EV charging network and low carbon fueling alternatives. Its Pilot and Flying J travel center network includes over 750 locations in 44 states and six Canadian provinces with more than 790 restaurants, 77,000 truck parking spaces, 5,500 deluxe showers, 6,300 diesel lanes and truck maintenance and tire service with Southern Tire Mart at Pilot Flying J. The One9 Fuel Network provides smaller fleets and independent professional drivers with a variety of fueling locations that offer everyday value, convenience, credit and perks. More information on locations and rewards is available in the myRewards Plus app.

Pilot Company is a majority owned subsidiary of Berkshire Hathaway. For additional information about Pilot Company, its 30,000 team members and commitment to giving back, visit pilotcompany.com. 

About Wavicle

Wavicle Data Solutions provides award-winning cloud data and analytics solutions that accelerate value, reduce risk, and empower our clients to make smart, data-driven decisions. We combine deep technical expertise and industry knowledge with proprietary automation tools to support the rapid shift to modern data architectures and real-time insights. Today’s executives need more visibility into their businesses, faster than ever, and we help them get it. Founded in 2013 and headquartered in the Chicago area, Wavicle has been recognized as an Inc. 5000 company five years running and is a multi-year recipient of the Chicago Tribune’s Top Workplace award. Wavicle is also an NMSDC-certified MBE. For more information, visit wavicledata.com.