Data Engineering - Databricks Sr.

Data Engineering - Databricks Sr.

NEORIS, now part of EPAM, is a Digital Accelerator helping companies step into the future. With more than 20 years of experience as the trusted Digital Partner of some of the world's most recognized brands, we are a global team of 4,000+ professionals across 11 countries. Our multicultural and startup-driven culture fosters innovation, continuous learning, and the creation of high-impact solutions for our clients.

Main Responsibilities:

• Design, develop, and optimize end-to-end ETL/ELT data pipelines using Azure Databricks and Spark (SQL, PySpark).
• Build scalable ingestion processes for structured and semi-structured data from various sources including APIs, databases, and file systems.
• Implement complex data transformation logic and publish clean datasets to Azure Data Lake or other cloud storage for analytics consumption.
• Leverage Databricks Workflows and automation tools (e.g., Apache Airflow) for orchestrating and scheduling pipeline execution.
• Ensure best practices for performance tuning, including partitioning, caching, and efficient Spark configurations.
• Implement robust data governance and security practices using Unity Catalog and Azure-native capabilities, including encryption and RBAC.

Requirements:

Mandatory:

• +5 years of proven experience as a Data Engineer working with large-scale data processing in cloud environments.
• Expertise in building and optimizing data pipelines using Azure Databricks, Spark (SQL & PySpark), and Databricks Workflows.
• Strong hands-on experience with Azure Data Lake, Blob Storage, and Azure Synapse Analytics.
• Solid understanding of data governance practices and tools, including Databricks Unity Catalog.
• Demonstrated experience in performance tuning and optimization of Spark jobs.
• Working knowledge of infrastructure as code tools like Terraform and monitoring using Azure Monitor.
• English proficiency (B2 or above).

Nice to Have:

• Experience with Delta Lake for managing data versioning and incremental data loads.
• Familiarity with CI/CD pipelines and DevOps practices in Azure environments.
• Relevant certifications in Microsoft Azure or Databricks.
• Experience with Apache Airflow or similar orchestration tools.

We Offer:

• Permanent contract with a competitive salary.
• Flexible work model with remote work possibilities.
• Personalized career plan and continuous learning (certifications, English training, etc.).
• Participation in stable, technically advanced projects.
• Flexible schedule and work-life balance culture.
• Social benefits tailored to your needs.