MA - Ssr. Databricks Engineer - 189 - Remote Eligible

Databricks Engineer

We are seeking a highly skilled Databricks Engineer to join our dynamic data engineering team. This role is pivotal in driving the migration of existing Hadoop workloads to the modern Databricks Lakehouse platform, enabling our organization to leverage scalable, efficient, and innovative data solutions. The ideal candidate will be instrumental in designing, developing, optimizing, and monitoring data pipelines and architectures that support our business intelligence and analytics initiatives. This position offers an exciting opportunity to work with cutting-edge cloud technologies, collaborate across multiple teams, and contribute to a strategic phased migration roadmap that will transform our data infrastructure.

Responsibilities

• Identify and classify current Hadoop jobs and data sources to understand migration scope and complexity.
• Collaborate with stakeholders to prioritize and select use cases for Minimum Viable Product (MVP) migration initiatives.
• Set up and manage Databricks environments on Azure, ensuring optimal configuration for performance and security.
• Lead pilot projects to migrate Hadoop workloads to Databricks, validating data integrity and system performance post-migration.
• Design scalable, repeatable ETL processes using Apache Spark (Scala or PySpark) to transform and load data efficiently.
• Implement monitoring solutions to track pipeline health, query performance, and cost management.
• Utilize Databricks capabilities such as Delta Lake, Lakehouse Federation, Liquid Clustering, and Unity Catalog to enhance data governance and performance.
• Work closely with data scientists, analysts, DevOps, and business teams to ensure alignment and successful delivery of data solutions.
• Define and monitor KPIs related to migration success, data quality, system performance, and cost efficiency.
• Assist in developing a phased, strategic roadmap for the full migration from Hadoop to the Databricks Lakehouse platform.
• Implement and maintain data governance policies to ensure data security, privacy, and regulatory compliance throughout the migration process.