Data Engineering Consultant - Cloud, Databricks & AI

Senior Data Engineer

Optum is a global organization that delivers care, aided by technology to help millions of people live healthier lives. The work you do with our team will directly improve health outcomes by connecting people with the care, pharmacy benefits, data and resources they need to feel their best. Here, you will find a culture guided by inclusion, talented peers, comprehensive benefits and career development opportunities. Come make an impact on the communities we serve as you help us advance health optimization on a global scale. Join us to start Caring. Connecting. Growing together.

We are seeking a highly experienced and innovative Senior Data Engineer with deep expertise in cloud-native data engineering, particularly using Databricks, Python, and AI-driven data solutions. This role will be pivotal in designing and building scalable data platforms that support advanced analytics, machine learning, and business intelligence across the organization.

Primary Responsibilities:

• Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, and Python on Azure Cloud
• Architect and implement cloud-based data solutions leveraging Azure Data Lake, and Azure Data Factory
• Collaborate with data scientists and AI teams to operationalize machine learning models and integrate them into data workflows
• Build and maintain feature stores, model input pipelines, and real-time data processing frameworks
• Ensure data quality, governance, and security across all stages of the data lifecycle
• Develop reusable frameworks and automation scripts to accelerate data engineering and AI deployment
• Participate in Agile development processes, including sprint planning, code reviews, and CI/CD using GitHub or Azure DevOps
• Mentor junior engineers and contribute to knowledge sharing across teams

Required Qualifications:

• 7+ years of experience in data engineering, with a solid focus on cloud platforms and big data technologies
• Hands-on experience with Databricks, Apache Spark, and Delta Lake
• Experience with Azure Cloud Services including Data Factory, Blob Storage
• Experience with CI/CD tools and version control systems (e.g., Git, GitHub Actions, Azure DevOps)
• Expert-level proficiency in Python for data processing, automation, and integration
• Solid understanding of ETL/ELT, data modeling, and distributed computing
• Familiarity with AI/ML concepts, model deployment, and integration into data pipelines
• Proficiency in SQL and working with relational and NoSQL databases
• Proven excellent problem-solving, communication, and collaboration skills

Preferred Qualifications:

• Azure certifications (e.g., Azure Data Engineer Associate, Azure AI Engineer Associate) are a plus
• Experience with MLflow, Feature Store, or other MLOps tools
• Exposure to Snowflake, Power BI, or other BI platforms
• Knowledge of data governance, privacy, and compliance standards