Lead Data Engineer

Lead Data Engineer

As a Lead Data Engineer, you will be joining the newly formed AI, Data & Analytics team, whose mission is to drive increased value from the data InvestCloud captures to enable a smarter financial future for our clients, focused on "enhanced intelligence". Ensuring we have fit-for-purpose modern capabilities is a key goal for the team. This is a unique opportunity to shape the architecture and the tech execution patterns of a green field ecosystem to create a next-generation advisor and client experience. We're building for scale. As such, much of what we design and implement today will be the technology/infrastructure which will serve thousands of clients and petabyte-level volumes of data. The core stack we use and are building is: AWS as our cloud provider, Oracle as our legacy data warehouse, Snowflake as our next-gen data warehouse, Mage AI for data ingestion and processing, Kafka as our message bus, Terraform for building infrastructure. Key responsibilities include asking the right questions to deeply understand stakeholder needs, driving the technical strategy of the team, leading a team to build reliable and scalable pipelines and capabilities across the platform, inspiring, teaching and guiding team members, owning complex architectures tying multiple services, SaaS tooling and third-party data together, upholding high-quality standards across documentation, testing, resiliency, monitoring, and code quality, driving towards efficiency, lowering cloud spend, tackling tech debt, promoting data quality, governance and security, and complying with relevant regulations.

Required skills include at least 10 years of relevant professional experience in Data Engineering, shaping the architecture of a mature cloud data platform (AWS, GCP, Azure), serving as a leader in technology in the past, interacting with a large community of stakeholders, designing and implementing performant, reusable, and scalable data models in a cloud data warehouse and in a legacy ecosystem, building resilient batch and streaming data pipelines at scale, building and maintaining infrastructure through IaC, containerization, and CI/CD, advocating for operational procedures from data and infrastructure observability, bringing deep expertise in SQL and Python, and being AI-proficient or AI-curious.