Lead Engineer, Big Data (ai/azure Data Services) - Remote - Remote Eligible
USA Jobs
Lead Data Engineer
A Lead Data Engineer collaborates with the Data Quality and Governance team to ensure data pipelines, data integrity, and compliance by defining data strategies, implementing Data Governance capabilities, creating self-service data assets, and integrating robust data quality and governance frameworks to support trustworthy AI solutions. Key responsibilities include designing scalable data architectures, establishing data quality standards and monitoring, managing data lineage, and ensuring adherence to regulatory requirements and privacy policies.
Knowledge, Skills & Abilities (Occupational knowledge and specific technical and professional skills and abilities required to perform the essential duties of this job):
Data Quality & Governance Leadership
Define and enforce data quality standards, validation rules, and monitoring systems tailored to healthcare data.
Collaborate with data stewards, compliance officers, and business stakeholders to resolve data integrity issues and ensure consistent data across systems.
Develop and maintain metadata management, data dictionaries, and stewardship workflows.
Regulatory Compliance & Security
Ensure all data engineering practices align with healthcare regulations such as HIPAA, HITECH, and other privacy laws.
Implement data governance policies that support secure, ethical, and compliant data usage across the organization.
Data Architecture & Lifecycle Management
Design scalable data architectures that support healthcare analytics and AI/ML workflows.
Automate data lineage tracking, governance documentation, and audit trails to support transparency and traceability.
Establish data lifecycle policies for retention, archiving, and disposal in accordance with regulatory and operational requirements.
AI/ML Enablement
Collaborate with data scientists and ML engineers to ensure data pipelines meet model training and inference requirements.
Architect and implement end-to-end AI pipelines using Agentic AI and GenAI frameworks.
Support the development of trustworthy AI by ensuring the use of reliable, consented, and well-governed data to ensure AI enabled solutions around Data Governance.
Cross-Functional Collaboration
Partner with clinical informatics, compliance, IT, and analytics teams to align data engineering efforts with healthcare delivery goals and governance strategies.
Lead by Example
Must be hands on Data Quality, Data Governance tools, Databricks, Power BI
Experience in Azure Data Services (like Azure Databricks, Unity Catalog, Purview, Azure Data Factory) and Power BI
Ability to lead, in solving technical issues while engaged with infrastructure and vendor support teams
Analyze current business practices, processes and procedures and identify opportunities for leveraging Microsoft Azure data & analytics PaaS services.
Leadership & Mentorship
Lead and mentor a team of data engineers, fostering a culture of quality, compliance, and innovation.
Oversee project delivery and ensure alignment with enterprise data governance objectives.
Job Function:
Responsible for all the aspects of architecture, design and implementation of Data Governance in Databricks.
Required Education:
Bachelor's Degree
Required Experience:
8+ years of data management experience
Prior experience leading projects or teams
Strong experience on Data Lake, Data Warehouse, Data Validation & Certification, Data Quality, Metadata Management and Data Governance
Experience in Azure Data Services (like Azure Databricks, Unity Catalog, Purview, Azure Data Factory) and Power BI, and in programming languages such as, PySpark/Python/SQL, etc.
Preferred experience in building stream-processing systems, using solutions such as Kafka, Storm or Spark-Streaming
Experience in implementing AI based solutions in Data Governance space
Preferred Education:
Master's Degree
Preferred Experience:
Experience in the healthcare industry is preferred