Big Data Consulting and Team Augmentation for a Jewelry Company's Enterprise Data Warehouse
About the Client
The Client is a large jewelry manufacturer and retailer that distributes its products through online and offline stores across the US.
Legacy ETL Couldn't Keep Up With Growing Data Volumes
The Client had a legacy Informatica solution for data analytics, which started showing subpar performance as the company's business grew. As the legacy solution was unable to handle the growing data volumes, the Client initiated in-house development of a new Incorta-based enterprise data warehouse. The new DWH was to enable enterprise-wide analytics of data coming from the company's business-critical systems (for example, CRM, ERP, and SCM) to facilitate informed decision-making for management.
When implementing the Incorta Spark layer, the Client faced a lack of big data skills in its in-house team. The team needed expert guidance to accelerate the rebuilding of legacy ETL processes on the new Incorta platform. The project had strategic importance for the Client, as each ETL pipeline migrated to Incorta allowed the company to run tens to hundreds of reports much faster than in the legacy system. To speed up the project and ensure the full reliability of the new ETL pipelines, the Client started looking for a reliable big data consultant.
Hands-On Big Data Guidance to Migrate ETL to Incorta and Spark
Trusting INNERLUXES's nine years in big data services and a solid portfolio of successful big data projects, the Client chose INNERLUXES as the consultant for the project.
INNERLUXES's senior data engineer conducted an in-depth analysis of the solution under development and interviewed the Client's team about the difficulties they faced when rewriting the ETL business logic for the new solution. He found that the developers were highly proficient in SQL but lacked Python and Spark skills, which was slowing down the project.
To address the skill gaps, INNERLUXES's big data expert started working together with three of the Client's in-house developers, both individually and as a group. Given the high professionalism of the in-house team, the specialist fostered practical collaboration rather than formal training sessions on a particular topic. For example, the expert helped the developers troubleshoot their Python, SQL, and Bash code whenever they faced an issue. They met daily on Zoom to investigate arising problems and come up with pragmatic, future-proof solutions. This approach significantly sped up ETL coding, testing, and deployment.
Within just three weeks of collaborating with INNERLUXES, the Client's team noticed great improvement in their ETL-building skills. Satisfied with the efficiency of the consulting services, the Client asked INNERLUXES's senior data engineer to join the project as a developer and continue guiding the in-house team.
As of December 2022, the expert had spent over six months on the Client's team, providing practical recommendations on secure ETL migration and Spark tuning. As a developer, he built 7 ETL pipelines that the Client is already using to get crucial insights for inventory management optimization, marketing campaign planning, tendering, and more.
In parallel with big data consulting and ETL implementation, INNERLUXES's expert worked out several modifications for the current one-tier EDW architecture that would unlock the full potential of Spark and help achieve significant long-term cost savings. In particular, he suggested creating a separate layer for the ETL pipelines to avoid storing exabytes of data in Incorta's RAM, which can be extremely costly. The Client highly appreciates the expert's proactiveness and is considering implementing the suggested modifications in the future.
Faster ETL Delivery and a Stronger In-House Team
By reaching out to INNERLUXES, the Client significantly sped up the delivery of the new ETL pipelines for its enterprise data warehouse. Thanks to expert guidance and knowledge transfer from INNERLUXES's big data consultant, the Client's in-house developers are showing significant improvement in their Python and Spark skills and confidently rewriting the business logic of ETL pipelines for the new EDW solution.
The Client is already using the 7 ETL pipelines built by INNERLUXES's developer to improve the efficiency of inventory management, marketing campaign planning, and tendering. Appreciating the expert's pragmatic suggestions for modifying the existing EDW architecture, the Client is considering long-term collaboration with INNERLUXES.
Technologies and Tools
Incorta, Python, Apache Spark, PySpark, SQL, Bash.