Sr Azure Data Engineer (azure Databricks/adls)

Sr Azure Data Engineer Lead (Azure Databricks)

Job Title: Sr Azure Data Engineer (Azure Databricks/ADLS)

Location: Mississauga, ON (Hybrid)

Role: Full Time/CTH

Exp: 10-12+ years

Skills:

• Azure Data Lake Storage
• Azure Data Factory
• Azure Data Bricks
• Azure Synapse
• Azure Analysis Server
• Azure Devops
• Spark SQL
• SQL
• Data Modelling
• Data Warehousing
• Pyspark, Scala

Description:

Implemented end-to-end solution for migrating on-premises system to Azure Cloud.

Analysis of the requirements provided by the client and transforming them into functional and technical specifications.

Built the metadata driven Data Ingestion and Data Transformation pipelines using Azure Data factory.

Have done the performance optimization using delta lake house in databricks.

Built the complex business logic in Databricks using SparkSQL, SCALA and PySpark.

Built the Spark applications for data extraction, transformation and aggregation from multiple file formats for analyzing and transforming the data uncover insight into the customer usage patterns.

Creation of cubes on Visual Studio on top of Azure Synapse and ADLS which works as a consumption source for visualization on Power BI dashboards.

Created multiple complex and dynamic stored procedures using Azure Synapse and scheduled them Using ADF. Have done the performance optimization for Databricks and Stored procedures and reduced the execution time to some extent.

Deployed all the Components from one environment to another environment Using Azure Dev ops.

Managed the team and responsible for requirement gathering, data modelling, capacity planning, sprint planning, developing and on time delivery of the project.