Spectacular Data Engineering Lead (**mid Level And Above)

Black Cape

Job Title: Data Engineering Lead

Location: Arlington, Falls Church, Crystal City, Alexandria or Pentagon, VA

Employment Type: Full-time

ONSITE: Ability to go ONSITE up to 2-5 days per week (depending on customer needs)

Clearance: Active TS/SCI required - no CI Poly but need to be willing to get one

MUST HAVE an Active TS/SCI Clearance

Need to be OPEN to sitting for a CI Poly

Be able to flex in and out of SCIFs at Pentagon, Crystal City, Alexandria, and Falls Church

About the Project:

We are delivering a large-scale data platform initiative for a DoD client, involving a 50-person cross-functional team of data engineers, BI developers, product owners, and domain experts. Our infrastructure is built on Databricks for data processing and transformation, and Qlik for analytics and reporting.

We are seeking a Data Engineering Lead to lead the data engineering workstream—ensuring high-quality, scalable, and efficient data solutions that enable timely insights and business decision-making.

Role Overview:

The Data Engineering Lead will oversee the design, development, and maintenance of the project's data pipelines, data models, and integrations. This role involves both technical leadership and people management—guiding a team of engineers, setting priorities, and ensuring successful delivery in alignment with the project's goals and timelines.

We are expanding and need to hire motivated data engineers to help build innovative solutions for our customers. The ideal candidate needs limited direction and to be passionate about technology. A willingness and desire to pick up new technologies or languages quickly is a must.

Our team builds and deploys cutting-edge AI and Machine Learning applications that directly support national security missions. We leverage the Databricks platform for advanced data science and engineering, and transform complex model outputs into compelling, interactive dashboards using Qlik.

These solutions are built to empower 19 key Principal Staff members (Senior Executive Service members and General Officers) within the Office of the Undersecretary of Defense (OUSD) and the Intelligence community. Our products provide them with the critical, data-driven insights needed for strategic decision-making.

Our work is grounded in successful deployments on JWICS, NIPR, SIPR utilizing AWS, combining commercial best practices in an Agile environment to ensure we deliver impactful capabilities to senior defense leadership.

Key Responsibilities

Leadership & Management:

• Lead and manage a team of Data Engineers working on Databricks-based data processing workflows
• Coordinate closely with developers, Qlik specialists, and other project leads to align technical outputs with reporting requirements
• Oversee sprint planning, backlog management, and milestone tracking
• Mentor engineers and promote best practices for coding, testing, and deployment

Technical Oversight:

• Design and maintain scalable data pipelines using Databricks
• Ensure data quality, lineage, and governance across ingestion, transformation, and consumption layers
• Optimize Databricks clusters and jobs for performance and cost efficiency
• Integrate with Qlik data models to ensure seamless dashboard performance

Collaboration & Stakeholder Engagement:

• Work with project managers to define scope, deliverables, and timelines
• Partner with product owners and Qlik developers to ensure transformed data meets analytical needs
• Act as a technical point of contact for client stakeholders, communicating risks, trade-offs, and progress

Quality & Compliance:

• Implement robust testing and validation for all data pipelines
• Ensure compliance with data security and privacy requirements
• Maintain documentation for data flows, transformations, and system dependencies

Qualifications:

Required (MUST HAVES):

• 5+ years in data engineering, with at least 2 years in a team lead or manager role
• Hands-on experience with Databricks
• Strong knowledge of data modeling and transformation for BI tools, preferably Qlik. Tableau would be acceptable
• Proficiency in cloud platforms (AWS preferred) and related data services; if no AWS experience, we can consider Azure or GCP (Google Cloud Platform) experience
• Track record of delivering complex data projects on time and within budget
• Experience managing data engineering workstreams in a multi-team project (35+ people)
• Familiarity with CI/CD for data workflows and version control (e.g., Git)
• Working knowledge with JIRA and Confluence; some previous experience preferred
• Experience working in an Agile delivery in a data project context
• Customer-interfacing experience in the Federal sector preferred; working professionally across many cross-functional teams
• Team Player ALL THE WAY!

Preferred (NICE TO HAVES):

• Experience managing data engineering workstreams in a multi-team project (60+ people)
• Experience with Jupyter Notebooks or Familiarity with it

What We Offer:

• Opportunity to lead a high-impact project with modern data infrastructure
• Work alongside a skilled, multidisciplinary team(s)
• We help manage the mission; You manage your time and Team; Black Cape Offers a Lot of Autonomy to Our Leaders, because we hire well

About the Company:

Want to be a part of a LOYAL, Intelligent, Talented, Dynamic & FUN Team from the ground up and help build something GREAT?? Then, you should consider Us! We work with the Intelligence Community and Department of Defense to accomplish their goals! Come join us in creating cloud-native web applications, advanced data analysis tools, and new machine learning capabilities. Be a part of a team that is making an IMPACT on the world and its safety and HAVE FUN while doing so! Black Cape is a unique technology company with a focus on Machine Learning, AI, Cloud and BIG Data work and Services and Products positioned for Federal Government customers and some Commercial Customers. Black Cape is a company developed by Proven Success Stories. Former successful companies of these Founders supported many programs as both a prime and a subcontractor and created a strong reputation working with top tier partners, in every major Intelligence Agency and the Department of Defense. We are a COOL and FUN crew that works hard and plays maybe even a little harder!! Our Core Team is an impressive bunch with exceptional leadership at the helm! Black Cape provides full-stack software development and software integration capabilities for traditional, cloud, and hybrid-cloud environments. We are doing AI technology, Machine Learning and Cloud technology- that will IMPACT the greater good and the world's safety!! CONTACT: recruiting@blackcape.io to learn MORE! Great Benefits | 401K | Competitive Salary | Generous Time Off | Exceptional Team-Building & Fun Company Events | Mentorship & Professional Development Programs | Tuition Reimbursement