Senior Data Engineer - Manager

Senior Data Engineer (Officer)

State Street Investment Management's Data, Analytics & AI Services (DAAIS) team is seeking a Senior Data Engineer (Officer) for our Bangalore, India office. This role is accountable for building and operating modern data platform and data pipeline capabilities that enable the investment management business. The successful candidate will bring strong hands-on expertise in distributed data processing, data pipeline engineering, and cloud-based data platforms. The role will execute on solutions and technical direction defined by platform leadership/architecture, with a focus on reliable delivery, day-to-day operations, and continuous improvement of production data workloads.

Key Responsibilities

• Build and operate end-to-end batch and streaming data pipelines on lakehouse platforms (Snowflake/Databricks, Apache Spark), delivering curated, analytics-ready datasets aligned to established standards.
• Develop production-grade pipeline code using Python, Java/Scala, and SQL, applying best practices for reliability, maintainability, performance, and testability.
• Implement and maintain Apache Iceberg tables based on defined patterns, including partitioning, schema evolution, and metadata/catalog practices to support efficient analytical access.
• Deliver governed data integration and curation for reference and application domains; publish semantic/serving models via BI tools and other approved consumption interfaces.
• Implement automated data quality, validation, reconciliation, lineage, and observability to ensure trusted, auditable, production-grade outcomes.
• Partner with domain SMEs, architects, and application teams to understand data requirements and semantics, and implement agreed logical/physical models and consumption contracts.
• Use standard CI/CD pipelines and platform tooling to deploy and run data workloads; follow platform guardrails, security requirements, and operational best practices.
• Maintain data product interfaces (schemas, SLAs, documentation, versioning, and lifecycle processes) in alignment with the broader platform standards.
• Troubleshoot data and environment issues through impact analysis, root-cause analysis, and corrective/preventive actions; document fixes and lessons learned.
• Support incident management, execute backfills as needed, and maintain production stability for data pipelines.

Required Qualifications

• 10+ years of experience in software and/or data engineering; BS/MS in Computer Science, Engineering, or equivalent practical experience.
• Strong programming skills in Python and SQL; Java/Scala experience preferred; ability to write production-quality, well-tested, maintainable code.
• Hands-on experience with Apache Spark (batch and/or streaming), including troubleshooting, performance tuning, and cost-aware optimization.
• Hands-on experience with Snowflake and Databricks in an enterprise environment, including job orchestration, query optimization, and platform operations.
• Experience building production-grade data pipelines in cloud environments (AWS preferred; GCP or Azure acceptable).
• Solid understanding of data modeling concepts, ETL/ELT patterns, schema evolution, and distributed data processing principles across structured and unstructured data.
• Working knowledge of lakehouse/data platform components (e.g., Databricks, Snowflake, Trino, Iceberg) and interoperability, connectivity, and access patterns.
• Working knowledge of open table formats and metadata/catalog ecosystems (e.g., Apache Iceberg; Unity Catalog or equivalent).
• Experience in Capital Markets, with understanding of key reference/application domains (e.g., Securities, Benchmarking, Holdings, Sales, Accounts, Products).
• Excellent written and verbal communication skills, with demonstrated ability to collaborate with global engineering, operations, and business stakeholders.

Preferred Qualifications

• Experience with data quality frameworks, automated reconciliation patterns, and data lineage tooling.
• Professional Certificates on Snowflake and/or Databricks

Professional Certificates on AWS, GCP and/or Azure