# Reliability Engineer, Supercomputing

**Company:** [Thinking Machines Lab](https://hotfix.jobs/companies/thinking-machines-lab)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $350k – $475k/yr
**Skills:** Python, Rust, Kubernetes, Slurm, Linux, Bmc, Idrac, Ipmi, Redfish, Dcgm, Nvlink, Nvswitch
**Posted:** 2026-06-24

> Ensure reliability of large GPU supercomputing clusters by diagnosing hardware/firmware/OS issues, automating monitoring, driving firmware rollouts, and working directly with vendors.

## Job Description

## What You’ll Do
- Investigate, reproduce, and remediate issues across large GPU clusters.
- Own the drivers, kernel surface, and diagnostics that span hardware, firmware, and OS.
- Automate the monitoring of fleet reliability and analyze error rates to validate whether a fix or firmware change measurably reduced failures rather than shifting them around.
- Drive the firmware lifecycle: tracking, qualification, staged rollout, and regression analysis.
- Engage vendors directly — GPUs, server OEMs, NIC vendors, and storage vendors — to get real fixes rather than ticket numbers. Manage RMA flows when hardware needs to come out.
- Monitor and improve GPU hardware health signals and turn them into actionable reliability improvements.
- Write clear postmortems and vendor cases that move issues forward.

## Skills and Qualifications
**Minimum qualifications:**
- Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
- Proficiency in at least one backend language (we use Python or Rust).
- Experience operating large‑scale clusters and container orchestration systems (e.g. Kubernetes or Slurm).
- Comfort operating across the stack and owning projects end-to-end.
- Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.
- A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

**Preferred qualifications:**
- Fluency with Linux systems and debugging tools.
- Proven statistical rigor in analyzing reliability.
- A track record of debugging a problem from application symptom to the root cause in hardware.
- Comfort reading vendor errata, firmware release notes, and kernel changelogs.
- Experience engaging hardware vendors directly — not just through escalation portals.
- Linux kernel literacy: the scheduler, memory management, IRQ paths, and the driver model.
- Out-of-band management experience: BMC / iDRAC / IPMI / Redfish.
- Depth in GPU hardware health: Xid error taxonomy, NVLink, NVSwitch, fabric manager, and DCGM.
- Proficiency in at least one backend language (we use Python and Rust).
- Significant ownership of the hardware reliability function at scale.
- Strong writing skills for vendor cases and postmortems.
- An instinct for telling apart a flaky machine, a flaky workload, and a flaky test.

## Similar roles

- [Network Engineer, Supercomputing](https://hotfix.jobs/jobs/2a952a8c-f34c-4fdd-a219-c8ac22399145) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Performance Engineer, Inference Systems](https://hotfix.jobs/jobs/c4e8b3e8-25ad-4a83-b11d-bebf411d63f1) - Anthropic - San Francisco, CA - $350k – $850k/yr
- [Site Reliability Engineer (SRE)](https://hotfix.jobs/jobs/893a897e-0421-40b8-a8ac-f61c0b30d5ff) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Research Engineer, Infrastructure, Training Systems](https://hotfix.jobs/jobs/9cc8be27-572e-4c19-97d0-8a7ae54e41a8) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Research Engineer, Infrastructure, RL Systems](https://hotfix.jobs/jobs/71d52d2b-ffa8-42f6-a7af-3a117bfd347b) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr

**Apply:** https://hotfix.jobs/jobs/e9263e4b-66f2-4203-b8da-b88ba1855dfe
**Canonical:** https://hotfix.jobs/jobs/e9263e4b-66f2-4203-b8da-b88ba1855dfe