Data Engineer
Virtusa
Lead Software Engineer
Design, develop, and maintain efficient and reliable data pipelines using Java, Scala, Apache Spark and Confluent Cloud (Kafka, KStreams, kSQLDB, Schema Registry).
Leverage Apache Spark (Java/Scala) for large-scale data processing and transformation.
Experience with building, maintaining and debugging applications and data pipelines using Confluent Cloud (Kafka, KStreams, kSQLDB, Schema Registry).
Build and optimize data storage solutions using NoSQL databases such as ScyllaDB and/or Cassandra.
Experienced with AWS services required for Data Engineering such as EMR, ServerlessEMR, AWS Glue, CodeCommit, EC2, S3 etc.
Familiarity with workflow orchestration tools such as Airflow.
Experience with building and deploying applications using Docker or AWS ECS or AWS EKS.
Well versed with code management using tools like GitHub and CI/CD pipelines and deployment of data pipelines on AWS cloud.
Implement and manage search and analytics capabilities using AWS OpenSearch and/or Elasticsearch.
Collaborate with data scientists, analysts, and other engineers to understand data requirements and deliver effective solutions.
Monitor and troubleshoot data pipelines to ensure data quality and performance.
Implement data governance and data quality best practices.
Automate data ingestion, processing, and deployment processes.
Stay up-to-date with the latest data engineering trends and technologies.
Contribute to the design and architecture of our data platform on AWS.