Principal Data Engineer

Technical Leadership

Architect, design, and lead the development of enterprise-scale, production-grade data platforms and pipelines using Databricks and cloud-native technologies (AWS, Azure, or GCP). Champion the adoption of the Databricks Lakehouse architecture to unify data warehousing, data science, and machine learning workloads across the organization. Guide the design and deployment of AI-ready data pipelines to support predictive analytics, generative AI, and advanced decision intelligence use cases. Define and enforce data engineering standards, including performance optimization, scalability, data observability, and cost efficiency. Oversee code reviews, architecture reviews, and system design discussions to ensure technical excellence and maintainability across the engineering team. Lead the implementation of robust data quality, governance, and compliance frameworks, leveraging Databricks Unity Catalog and modern metadata management tools. Solve complex data architecture and integration challenges using advanced technologies such as Spark, Delta Live Tables, Airflow, and MLflow. Drive the development of automated, CI/CD-enabled data workflows and promote best practices in data infrastructure as code (IaC) and DevOps for data.

Project Management and Collaboration

Provide strategic technical leadership and mentorship to data engineering teams, fostering a collaborative environment that promotes innovation, accountability, and growth. Collaborate closely with data architects, AI/ML engineers, and analytics teams to align data solutions with organizational goals and research initiatives. Engage with cross-campus and cross-departmental technical groups to evangelize modern data practices and accelerate AI transformation initiatives. Lead knowledge-sharing sessions and architecture reviews on emerging data engineering trends, Databricks advancements, and AI integration techniques.

Communication

Effectively communicate technical strategies, project status, risks, and architecture decisions to both technical and non-technical stakeholders. Translate complex data engineering concepts into clear business impacts, helping decision-makers understand opportunities and trade-offs. Produce clear and detailed technical documentation, design specifications, and operational playbooks to support long-term scalability and training. Advocate for data engineering as a foundational enabler of AI, analytics, and digital transformation initiatives across the institution.

Innovation

Lead research and development efforts to evaluate and implement cutting-edge technologies within the Databricks ecosystem and broader AI/data landscape. Conduct feasibility studies and proofs of concept (POCs) for next-generation architectures involving AI model integration, real-time streaming, and intelligent automation. Partner with academic, administrative, and campus stakeholders to pilot AI-enabled data systems, such as model-assisted data validation and automated feature generation. Stay ahead of emerging trends in data engineering, AI readiness, and cloud infrastructure, continuously recommending and implementing innovative solutions.

Other

Contribute to recruitment, hiring, and onboarding of new data engineering team members. Represent the data engineering function in strategic planning discussions and cross-organizational technology initiatives. Perform other duties as assigned, aligned with the mission to build a secure, scalable, and AI-enabled data ecosystem.