Data Engineer
Illinois Staffing
Data Engineer
Design, develop, and maintain efficient, reliable data pipelines and ETL processes using Microsoft Fabric or Databricks for large-scale data processing.
Collaborate with stakeholders to understand business requirements and create data solutions that align with technical and business strategies.
Optimize and improve workflows for data ingestion, transformation, and delivery to ensure high performance at scale.
Implement scalable architectures for big data processing, ensuring data quality, security, and governance best practices.
Develop and maintain reusable code to ensure consistency and reliability for future workflows.
Monitor and troubleshoot production environments, ensuring data systems perform as expected.
Stay current with emerging technologies, trends, and tools related to cloud-based data engineering platforms.
Required Skills:
- Experience working with Microsoft Fabric or Databricks for data engineering tasks such as data modeling, transformation, and integration.
- Proficiency in programming with one or more of the following: Python, SQL, Scala, or Java.
- Knowledge of relational and non-relational databases (e.g., SQL Server, MySQL, MongoDB).
- Experience with cloud platforms, such as Azure, AWS, or Google Cloud.
- Familiarity with big data tools and frameworks like Spark, Hadoop, or Kafka.
- Strong problem-solving skills and ability to work in dynamic, fast-changing environments.