# Senior Cluster Site Reliability Engineer

**Company:** [The Voleon Group](https://hotfix.jobs/companies/voleon)
**Location:** Remote
**Role:** DevOps / SRE
**Salary:** $205k – $235k/yr
**Experience:** 5+ years
**Skills:** Slurm, Kubernetes, Terraform, Ansible, AWS, GCP, Prometheus, Grafana, Python, Kubeflow, Docker, Lustre, Ceph, S3, OpenTelemetry
**Posted:** 2026-04-17

> Senior SRE scales and maintains high-performance research compute clusters for machine learning in finance, ensuring uptime, reliability, and observability across on-prem and cloud infrastructure. Requires 5+ years SRE/DevOps experience with HPC frameworks, IaC, and cloud services.

## Job Description

## Responsibilities
- Be a first responder in the event of cluster outages or issues. Triage and resolve urgent issues as they arise
- Ensure a high degree of cluster uptime (measured in multiple nines), and define + track SLAs to quantify reliability
- Diagnose systemic/recurring patterns of problems, and engineer precision solutions to them in collaboration with engineering teams
- Develop robust metrics and observability for cluster health and use those metrics to inform your work. Build out custom observability mechanisms when off-the-shelf ones won't do
- Help software and research teams design policies around fair cluster usage, and help develop enforcement mechanisms for said policies
- Assist in forecasting cluster growth, and help select appropriate scale-up strategies. Help optimize operations across dimensions of cost and usability

## Requirements
- 5+ years of experience in SRE or DevOps roles, preferably working as a senior engineer or tech lead
- Knowledge of HPC/batch compute frameworks (**Slurm**, Kueue, AWS/GCP Batch) and/or machine learning training systems (**Kubeflow**, MLflow, Horovod)
- Ability to develop scripts and utilities of moderate complexity in a common scripting language (**Python**, Ruby, etc.)
- Familiarity with infrastructure-as-code and configuration management tools (**Terraform**, **Ansible**)
- Experience with cloud infrastructure (**AWS** or **GCP**)
- Familiarity designing and implementing modern observability stacks (**Prometheus**, **Grafana**, **Loki**, ELK, **OpenTelemetry**)
- Experience with distributed storage technologies (**Lustre**, Ceph, **S3**)
- Embodies a "system engineer" rather than "system administrator" mindset, thinking systematically and leveraging automation
- **Bachelor** degree in computer science

## Preferred Qualifications
- Hands-on experience with HPC frameworks (**Slurm**, Grid Engine) and Kubernetes-based job orchestrators (**Airflow**, Kueue, **Kubeflow** Pipelines), along with other distributed computing frameworks (**Ray**, Modin, **Dask**, **Spark**)
- Familiarity with ML frameworks (**PyTorch**/**TensorFlow**, JAX, Horovod, **DeepSpeed**)
- Familiarity with hybrid/on-prem environments
- Experience with containerization (**Docker**, Podman, Singularity), particularly for HPC/batch compute environments
- Experience with HPC networking (InfiniBand, RDMA)
- Solid security/IAM foundations (Identity management systems, AWS/GCP IAM, Zero Trust)

## Similar roles

- [Software Engineer, Network Infrastructure](https://hotfix.jobs/jobs/d0c09025-2101-4ac3-bd9c-a102ae63eabe) - Render - Remote - $204k – $250k/yr
- [Senior Manager, Site Reliability Engineering (Federal)](https://hotfix.jobs/jobs/b15c33da-b99a-40de-9a77-bdc9ca283703) - Okta - Washington, DC - $207k – $285k/yr
- [Senior Software Engineer, Enterprise Resilience](https://hotfix.jobs/jobs/b5f13b32-b4fa-49ce-a20b-623f8a73d7e3) - Vanta - Remote - $207k – $244k/yr
- [Product Lead, Infrastructure](https://hotfix.jobs/jobs/848506fe-1bb4-44a6-8a3e-1bf495a8ca32) - Render - Remote - $207k – $285k/yr
- [Senior Software Engineer, Build Infrastructure](https://hotfix.jobs/jobs/25a72da6-710d-48fb-bf7d-78fd625570f9) - Zoox - Foster City, CA - $208k – $263k/yr

**Apply:** https://hotfix.jobs/jobs/a701cccb-047d-4b39-933b-75462379ed40
**Canonical:** https://hotfix.jobs/jobs/a701cccb-047d-4b39-933b-75462379ed40