Sr Data Engineer

Data Engineer

Job Description Responsibilities:

• Design and maintain scalable ETL/ELT pipelines to process time-series data from thermostats, weather sources, schedules, and device analytics.
• Develop and validate HVAC and energy datasets (e.g., Run & Drift, Thermal Coefficients, Efficiency Drift) as trusted sources for analytics, modeling, and automation.
• Refactor legacy Scala/Akka workflows into PySpark or Databricks jobs, enhancing observability, automated testing, and CI/CD coverage for upstream data feeds.
• Ensure reliable ingestion and consistency of data from MongoDB telemetry, Kafka/Pub/Sub streams, and cloud storage (GCS).
• Partner with data scientists to create and operationalize features for HVAC runtime prediction, anomaly detection, and demand response optimization.
• Implement best practices for data lineage, schema documentation, and change management to safeguard production systems against regressions.

Day to Day:

One of Insight Global's clients is looking for a Data Engineer to join their team. This individual will design and maintain scalable data pipelines that convert raw IoT telemetry into analytics-ready datasets, enabling energy and AI-driven applications. They will build and optimize data products that transform device telemetry into reliable, actionable intelligence. Additionally, they will manage integrations with streaming platforms and cloud storage. They will collaborate with data scientists to develop predictive models and demand response solutions, and enforce best practices for data governance. Additionally, the role may involve supporting AI/ML workflows for energy forecasting and other advanced applications. The ideal candidate is someone who has a startup mentality and isn't afraid of working on projects with little to no predefined requirements.

Skills and Requirements Must-Have:

• 5+ years of data engineering or backend data systems experience
• Strong proficiency in Python, SQL, and distributed data frameworks (PySpark, Databricks)
• Hands-on experience with GCP, Kafka/Pub-Sub, and data lake architecture
• Ability to read and modernize legacy Scala/Akka codebases
• Experience building production-grade pipelines that deliver analytics-ready datasets

Pluses:

• Skilled in ML platforms and feature engineering (e.g., Vertex AI)
• Hands-on experience with AI/ML applications: LLMs, computer vision, energy forecasting
• Knowledge of Demand Response and home energy automation
• Strong IoT background: protocols, telemetry
• Proficient with specialized databases:
• Graph (Neo4j)
• Vector (Pinecone)