Data Engineer

Data Engineer

Data Engineer in the Data & Analytics area designs, builds, and maintains cloud-based data pipelines, and data solutions to support data analysis and business intelligence initiatives. Key responsibilities include integrating cloud services, creating scalable data pipelines (ETL/ELT), ensuring data quality and security, and collaborating with data analysts to meet data needs using technologies like AWS Services, SQL, Python, PySpark and big data frameworks.

Key Responsibilities:

• Design, build, and manage scalable data pipelines (ETL/ELT) to ingest, transform, and load data from various sources into cloud data warehouses or data lakes.
• Integrate data from diverse systems, ensuring data is accessible, consistent, and reliable for downstream consumption.
• Implement data validation and cleansing procedures to ensure data integrity and work with security teams to protect data within the cloud environment.
• Work closely with data scientists, analysts, and other stakeholders to understand their data requirements and provide them with usable data.
• Monitor and optimize data pipelines for scalability, efficiency, and cost-effectiveness.

Required Skills & Qualifications:

• Strong proficiency in SQL with 6+ years exp and programming languages such as Python, PySpark, Shell scripting with 3+ years exp.
• 6+ years of hands-on data & analytics development experience.
• 4+ years of hands-on experience with AWS cloud services S3, EMR, Glue, Lambda.
• Expertise in big data technologies like Apache Spark, Kafka, and Spark Streaming.
• Experience with tools and processes for data extraction, transformation, and loading.
• Working experience with orchestration.
• Knowledge of data/dimensional modeling, data warehousing concepts, and database management.