Databricks Engineer

Join Astellas

Do you want to be part of an inclusive team that works to develop innovative therapies for patients? Every day, we are driven to develop and deliver innovative and effective new medicines to patients and physicians. If you want to be part of this exciting work, you belong to Astellas!

Astellas Pharma Inc. is a pharmaceutical company conducting business in more than 70 countries around the world. We are committed to turning innovative science into medical solutions that bring value and hope to patients and their families. Keeping our focus on addressing unmet medical needs and conducting our business with ethics and integrity enables us to improve the health of people throughout the world.

This position is based in Bengaluru and might require some on-site work.

Astellas' Global Capability Centres – Overview

Astellas' Global Capability Centres (GCCs) are strategically located sites that give Astellas the ability to access talent across various functions in the value chain and to co-locate core capabilities that are currently dispersed. Our three GCCs are located in India, Poland, and Mexico. The GCCs will enhance our operational efficiency, resilience, and innovation potential, enabling a timely response to changing business demands. Our GCCs are an integral part of Astellas, guided by our shared values and behaviors, and are critical enablers of the company's strategic priorities, sustainable growth, and commitment to turn innovative science into value for patients.

Purpose and Scope:

The Databricks Engineer is responsible for building and enhancing the data processing pipelines and distributed compute workloads that run on the Databricks Platform. This role focuses on writing scalable PySpark and SQL code, designing efficient Delta Lake data flows, and implementing reliable job orchestration patterns that support high volume, production grade data operations. You will work directly within Databricks notebooks and workflows to build ingestion and transformation logic, optimize cluster usage, and ensure pipelines meet performance, reliability, and cost expectations.

This position works closely with Data Engineers, Platform Engineering, and Data Science teams to translate technical requirements into well-structured data pipelines and automated jobs. The role involves debugging distributed compute issues, tuning Spark performance, enforcing coding and data quality standards, and integrating pipelines with CI/CD and monitoring tools. Your work ensures that downstream analytics, ML models, and business applications have access to accurate, timely, and well-organized data across the Astellas Data Platform.

Responsibilities and Accountabilities:

• Develop & Maintain Scalable Data Pipelines: Develop and maintain ETL/ELT pipelines using PySpark, Spark SQL, Auto Loader, and Delta Live Tables to support data ingestion and transformation needs.
• Implement Robust Lakehouse Architecture: Implement and enhance Medallion (Bronze/Silver/Gold) layers by applying Delta Lake features such as schema evolution, and optimization techniques.
• Integrate Data Across Cloud Platforms: Ingest and harmonize structured, semi structured, and unstructured data from multiple cloud environments including Azure, AWS, and enterprise object storage.
• Develop Reusable Engineering Frameworks: Create and maintain reusable Python, PySpark, and YAML based libraries and patterns to standardize ingestion, transformation, automation, and engineering workflows across teams.
• Implement Data Quality & Governance: Implement data validation checks and follow Unity Catalog governance standards for access control, lineage, external locations, and PII/PHI controls.
• CI/CD & Deployment Automation: Utilize Azure DevOps and Databricks Asset Bundles (DABs) to establish automated build, test, and deployment workflows; ensure source control discipline and promote engineering best practices.
• Optimize Performance & Cost Efficiency: Apply standard Spark performance techniques such as partitioning and query optimization to improve reliability and efficiency of data workloads.
• Collaborate with Data & Platform Teams: Work closely with Business, Analysts, SMEs, and Platform Engineering teams to translate requirements into scalable data solutions.
• Participate in Technical Reviews and Knowledge Sharing: Contribute to design discussions, share learnings with peers, and seek guidance from senior engineers to continuously improve engineering practices.