ML Cluster Operations Engineer

ML Cluster Operations Engineer (Slurm / K8s)

At TensorWave, we're leading the charge in AI compute, building a versatile cloud platform that's driving the next generation of AI innovation. We're focused on creating a foundation that empowers cutting-edge advancements in intelligent computing, pushing the boundaries of what's possible in the AI landscape.

About the Role

We are seeking an exceptional Machine Learning Engineer who has made training and AI workload scheduling a specialty. This is a senior-level role for someone who has significant experience managing distributed machine learning workloads at scale using Slurm and/or Kubernetes.

As a technical visionary and hands-on expert, you will lead the evolution of our managed Slurm and Kubernetes offerings, as well as internal health checking and cluster automation.

Key Responsibilities

• Manage and iterate our containerized Slurm (Slurm-in-Kubernetes) solution, including customer configuration and deployment.
• Work closely with our engineering team to develop and maintain CI and automation for managed offerings.
• Ensure healthy cluster operations and uptime by implementing active and passive health checks, including automated node draining and triage.
• Help profile and debug distributed workloads, from small inference jobs to cluster-wide training.
• Establish best practices for running jobs at scale, including monitoring, checkpointing, etc.
• Mentor and upskill ML engineers in best practices.

Qualifications

Must-Have:

• 5+ years of experience in cloud infrastructure, HPC, or machine learning roles.
• Significant hands-on experience with Slurm in production HPC/ML environments, including understanding of setup/configuration, enroot (pyxis), modules, and MPI.
• Strong knowledge of distributed ML languages and frameworks, such as Python, PyTorch, Megatron, c10d, MPI, etc.
• Understanding of node lifecycle, including health checks, prolog / epilog scripts, and draining.
• Deep understanding of security, compliance, and resilience in containerized workloads.

Nice-to-Have:

• 3+ years of hands-on Kubernetes experience, including deep knowledge of the Kubernetes API, internals, networking, and storage.
• Proficiency in writing Kubernetes manifests, Helm charts, and managing releases.
• Experience with DAGs using K8s native tools such as Argo Workflows.
• Foundation in networking, especially as it pertains to RDMA, RoCE, and Infiniband.
• Experience with low level kernel libraries, such as CUDA and Composable Kernel.
• Contributions to open-source projects or ML/AI tooling.

What Success Looks Like

• A production-grade integrated Slurm platform that can support thousands of GPUs, with self-healing, scaling, and strong observability.
• Infrastructure is resilient, secure, resource-optimized, and compliant.
• Best practices and tooling are well-documented, standardized, and continuously improved across the company.
• Make GPUs go Brrrrrrr

What We Bring

• Stock Options
• 100% paid Medical, Dental, and Vision insurance
• Life and Voluntary Supplemental Insurance
• Short Term Disability Insurance
• Flexible Spending Account
• 401(k)
• Flexible PTO
• Paid Holidays
• Parental Leave
• Mental Health Benefits through Spring Health