Lead Cloud Data Engineer - Manager

Lead Cloud Data Engineer

We are seeking an experienced Lead Cloud Data Engineer to design, build, and optimize scalable data solutions on cloud platforms. The candidate should have industry experience (preferably in Financial Services) in building cloud based database engineering platforms. The role requires strong technical expertise in Databricks and Spark.

What you will be responsible for:

• Design, develop, and optimize end-to-end data pipelines using Databricks
• Implement and manage Lakehouse architecture with Delta tables
• Develop high-performance PySpark and SQL transformations
• Ensure data quality, integrity, and performance across pipelines
• Lead troubleshooting, root cause analysis, and performance tuning
• Collaborate with Architects, Business Analysts, and downstream consumers
• Support CI/CD implementation for Databricks jobs and notebooks
• Mentor junior engineers and review code for quality and best practices
• Ensure adherence to security, governance, and compliance standards

What we value:

• Technical leadership and mentoring mindset
• Strong ownership and delivery focus
• Collaboration and teamwork across global teams
• Proactive identification of risks and improvement opportunities

Mandatory skills / experience:

• 10-14 years of hands on experience with AWS Cloud Data Engineering with Databricks platform
• Expertise in Data Modeling and DB design
• Hands on experience in setting up data pipelines for ETL jobs across multiple data sources.
• Hands-on experience with performance tuning, Git, Jenkins, and enterprise job scheduling tools (Autosys / Airflow)
• Basic understanding of Data warehousing and ETL concepts
• Knowledge of CI/CD tools (GitHub, Azure DevOps)
• Advanced SQL and Python skills
• Experience working in Agile delivery environments

Additional requirements / Good to have skills:

• Experience in regulatory or financial services domain
• Databricks or Cloud certifications
• Experience with visualization tools (PowerBI, Tableau)
• Exposure to Starburst, Spotfire, TDV, ServiceNow

Mode of work: Hybrid (Hyderabad or Bangalore)

Are you the right candidate? Yes!

We truly believe in the power that comes from the diverse backgrounds and experiences our employees bring with them. Although each vacancy details what we are looking for, we don't necessarily need you to fulfil all of them when applying. If you like change and innovation, seek to see the bigger picture, make data driven decisions and are a good team player, you could be a great fit.

Why this role is important to us:

Our technology function, Global Technology Services (GTS), is vital to State Street and is the key enabler for our business to deliver data and insights to our clients. We're driving the company's digital transformation and expanding business capabilities using industry best practices and advanced technologies such as cloud, artificial intelligence and robotics process automation.

We offer a collaborative environment where technology skills and innovation are valued in a global organization. We're looking for top technical talent to join our team and deliver creative technology solutions that help us become an end-to-end, next-generation financial services company.

Join us if you want to grow your technical skills, solve real problems and make your mark on our industry.