Performance Engineer

Performance Engineer

Diagnose issues that an in-house performance testing team has been unable to. There are five aspects to Performance Engineering: software development lifecycle and architecture, performance testing and validation, capacity planning, application performance management and problem detection and resolution. Must have skills: Databricks Unified Data Analytics Platform

Minimum 5 year(s) of experience is required Educational Qualification: 15 years full time education

The ideal candidate will have experience building: Reusable Python/PySpark frameworks for standardizing data engineering workflows Test frameworks to ensure pipeline reliability and correctness Data quality frameworks for monitoring and validation Additionally, hands-on experience with Datadog or similar observability tools is required to monitor pipeline performance, optimize resource usage, and ensure system reliability. You will work within a cross-functional team, building scalable, production-grade data pipelines on cloud platforms such as AWS, Azure, or GCP.

Roles & Responsibilities

• Data Engineering & Framework Development
• Develop and maintain ETL/ELT pipelines in Databricks using PySpark and Python.
• Build reusable, modular frameworks to accelerate development and enforce standards across pipelines.
• Implement test frameworks for automated unit, integration, and regression testing of pipelines.
• Design and maintain data quality frameworks to validate ingestion, transformation, and output.
• Optimize Spark jobs for performance, scalability, and cost-efficiency.
• Collaborate with data architects to define robust data models and design patterns.
• Cloud & Platform Integration
• Integrate Databricks pipelines with cloud-native storage services (e.g., S3, ADLS, Snowflake).
• Implement CI/CD pipelines for Databricks notebooks and jobs using Git, Jenkins, or Azure DevOps.
• Ensure pipelines follow best practices for modularity, reusability, and maintainability.
• Monitoring, Observability & Optimization
• Use Datadog to monitor pipeline performance, resource utilization, and system health.
• Build dashboards and alerts for proactive monitoring and troubleshooting.
• Analyze metrics and logs to identify bottlenecks and improve reliability.
• Collaboration & Delivery
• Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable solutions.
• Conduct code reviews, enforce best practices, and mentor junior engineers.
• Promote knowledge-sharing of reusable frameworks, testing practices, and data quality approaches.

Professional & Technical Skills:

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 5–8 years of experience in data engineering or software development.
• 3+ years hands-on experience with Databricks and PySpark.
• Strong Python programming skills, including writing reusable libraries and frameworks.
• Experience designing and implementing test frameworks for ETL/ELT pipelines.
• Experience building data quality frameworks for validation, monitoring, and anomaly detection.
• Proficiency in SQL and experience with cloud data warehouses (Snowflake, Redshift, BigQuery).
• Familiarity with Datadog or similar monitoring tools for metrics, dashboards, and alerts.
• Experience integrating Databricks with AWS, Azure, or GCP services.
• Working knowledge of CI/CD, Git, Docker/Kubernetes, and automated testing.
• Strong understanding of data architecture patterns — medallion/lakehouse architectures preferred.

Nice to Have:

• Experience with Airflow, Prefect, or Azure Data Factory for orchestration.
• Exposure to infrastructure-as-code tools (Terraform, CloudFormation).
• Familiarity with MLflow, Delta Live Tables, or Unity Catalog.
• Experience designing frameworks for logging, error handling, or observability.
• Knowledge of data security, access control, and compliance standards.

Soft Skills:

• Strong problem-solving and analytical skills.
• Excellent verbal and written communication.
• Ability to work in agile, cross-functional teams.
• Ownership mindset, proactive, and self-driven.

Additional Information:

• The candidate should have a minimum of 5 years of experience in Large Language Models.
• This position is based at our Bengaluru office.
• A 15 years full-time education is required.