Senior Cluster Site Reliability Engineer
Senior SRE scales and maintains high-performance research compute clusters for machine learning in finance, ensuring uptime, reliability, and observability across on-prem and cloud infrastructure. Requires 5+ years SRE/DevOps experience with HPC frameworks, IaC, and cloud services.
About the job
Responsibilities
- Be a first responder in the event of cluster outages or issues. Triage and resolve urgent issues as they arise
- Ensure a high degree of cluster uptime (measured in multiple nines), and define + track SLAs to quantify reliability
- Diagnose systemic/recurring patterns of problems, and engineer precision solutions to them in collaboration with engineering teams
- Develop robust metrics and observability for cluster health and use those metrics to inform your work. Build out custom observability mechanisms when off-the-shelf ones won't do
- Help software and research teams design policies around fair cluster usage, and help develop enforcement mechanisms for said policies
- Assist in forecasting cluster growth, and help select appropriate scale-up strategies. Help optimize operations across dimensions of cost and usability
Requirements
- 5+ years of experience in SRE or DevOps roles, preferably working as a senior engineer or tech lead
- Knowledge of HPC/batch compute frameworks (Slurm, Kueue, AWS/GCP Batch) and/or machine learning training systems (Kubeflow, MLflow, Horovod)
- Ability to develop scripts and utilities of moderate complexity in a common scripting language (Python, Ruby, etc.)
- Familiarity with infrastructure-as-code and configuration management tools (Terraform, Ansible)
- Experience with cloud infrastructure (AWS or GCP)
- Familiarity designing and implementing modern observability stacks (Prometheus, Grafana, Loki, ELK, OpenTelemetry)
- Experience with distributed storage technologies (Lustre, Ceph, S3)
- Embodies a "system engineer" rather than "system administrator" mindset, thinking systematically and leveraging automation
- Bachelor degree in computer science
Preferred Qualifications
- Hands-on experience with HPC frameworks (Slurm, Grid Engine) and Kubernetes-based job orchestrators (Airflow, Kueue, Kubeflow Pipelines), along with other distributed computing frameworks (Ray, Modin, Dask, Spark)
- Familiarity with ML frameworks (PyTorch/TensorFlow, JAX, Horovod, DeepSpeed)
- Familiarity with hybrid/on-prem environments
- Experience with containerization (Docker, Podman, Singularity), particularly for HPC/batch compute environments
- Experience with HPC networking (InfiniBand, RDMA)
- Solid security/IAM foundations (Identity management systems, AWS/GCP IAM, Zero Trust)
Skills
Slurm, Kubernetes, Terraform, Ansible, AWS, GCP, Prometheus, Grafana, Python, Kubeflow, Docker, Lustre, Ceph, S3, OpenTelemetry
Similar jobs
DevOps / SRE jobs
Skydio
San Mateo, CA
Build and improve cloud infrastructure, developer workflows, and internal tooling that make software development, testing, and releases more efficient and reliable. The role requires cloud architecture knowledge, CI/CD experience, Terraform and Bazel proficiency, and software development skills in Go, Python, or C++.
Anyscale
San Francisco, CA
Build and operate scalable control-plane and data-plane infrastructure for distributed AI workloads, including Ray cluster orchestration, scheduling, observability, and accelerator integration. Requires a bachelor's degree or equivalent experience, 3+ years of production coding, cloud-native expertise, Kubernetes, and Go/Python proficiency.
Onos Health
San Francisco, CA
Leads infrastructure and platform strategy for a production healthcare AI platform, owning AWS, reliability, disaster recovery, compliance, CI/CD, and secure AI-agent operations. Requires deep cloud and Terraform expertise, audit-cycle experience, and prior technical leadership.
Lightspark
Remote
The Production Engineer will build and operate secure, scalable, and reliable infrastructure and production services, while developing engineering frameworks and supporting on-call operations. The role requires 5+ years of production, site reliability, or DevOps experience and familiarity with AWS, Kubernetes, and Terraform.
Zoox
Foster City, CA
Senior engineer owning safety-critical software pipelines and infrastructure, from static and dynamic analysis through CI enforcement, dashboards, and reliability tooling. Requires an advanced technical degree, 7+ years working with large codebases, and expertise in Bazel, Python, backend infrastructure, and C++.