Lead Data Engineer

Data Engineering Position

Job responsibilities:

• Generates data models for their team using firmwide tooling, linear algebra, statistics, and geometrical algorithms
• Delivers data collection, storage, access, and analytics data platform solutions in a secure, stable, and scalable way
• Implements database back-up, recovery, and archiving strategy
• Evaluates and reports on access control processes to determine effectiveness of data asset security with minimal supervision
• Adds to team culture of diversity, opportunity, inclusion, and respect

Required qualifications, capabilities, and skills:

• Formal training or certification on software engineering concepts and 5+ years applied experience
• 3+ years of experience using technologies such as Databricks, Pyspark, AWS, is essential and creating ETL Pipeline from scratch is a must.
• 3+ years of experience working with AWS (Lambda, Step Function, SQS, SNS, API Gateway, secrets manager and storage services) is a must.
• 3+ years of experience in software engineering and object-oriented programming skills with expertise in Python and Terraform
• Hands on experience with open-source frameworks/libraries, such as Apache NiFi, Apache Airflow and Autosys.
• Strong understanding of REST API development using FASTAPI or equivalent frameworks.
• Advanced at SQL (e.g., joins and aggregations)

Preferred qualifications, capabilities, and skills:

• Familiar with development tools such as Jenkins, Jira, Git/Stash, spinnaker
• Familiarity with unit testing frameworks such as pytest or unittest.
• Extensive experience in statistical data analysis, with the ability to select appropriate tools and identify data patterns for effective analysis, as well as experience throughout the data lifecycle