# Network Engineer, Supercomputing

**Company:** [Thinking Machines Lab](https://hotfix.jobs/companies/thinking-machines-lab)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $350k – $475k/yr
**Skills:** Python, Rust, Kubernetes, Slurm, Rdma, Rocev2, Nccl, Nvlink, Nvswitch, CUDA, Linux
**Posted:** 2026-06-24

> Own and debug multi-thousand-GPU network fabric (RDMA/RoCE, NVLink/NVSwitch) for large-scale AI training and inference. Requires backend language proficiency, large-scale cluster experience, and cross-stack ownership.

## Job Description

## What You’ll Do
- Reason about and validate GPU network fabric design across our deployments.
- Debug RDMA / RoCEv2 across different NIC vendors. Diagnose collective failures of production NCCL, PFC/ECN tuning, and congestion control behavior.
- Own NVLink / NVSwitch interconnect — including fabric manager and IMEX health, link and lane errors, and how the GPU fabric interacts with collectives.
- Build host-level network instrumentation and use Linux tooling to build dashboards and alerts, not just the bug report.
- Navigate cross-cloud fabric quirks across providers and triage across the NIC, driver, kernel, switch, and workload boundaries.
- Drive escalations with cloud-provider networking teams, owning issues end-to-end until they're resolved.

## Skills and Qualifications
**Minimum qualifications:**
- Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
- Proficiency in at least one backend language (we use Python or Rust).
- Experience operating large‑scale clusters and container orchestration systems (e.g. Kubernetes or Slurm).
- Comfort operating across the stack and owning projects end-to-end.
- Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.
- A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

**Preferred qualifications:**
- Fluency with host-level debugging tools on Linux.
- Strong communication skills, internally and with cloud providers.
- Extensive experience with at least one of the following:
  - Familiarity with cloud network primitives across at least two cloud providers.
  - Hands-on experience with NVLink / NVSwitch, fabric manager, and IMEX.
  - Statistical rigor in reliability reasoning — comfort reasoning about failure and error rates, distributions, and base rates, and the judgment to separate signal from noise when characterizing a large fabric.
  - A track record of writing tooling that made the next debugging session meaningfully faster.
  - Familiarity with CUDA/NCCL and performance profiling for distributed training and inference.
  - Understanding of deep learning frameworks and their underlying system architectures.

## Compensation and Benefits
- Compensation: $350,000 - $475,000 USD annual salary range.
- Benefits: Generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.
- Visa sponsorship available.

## Similar roles

- [Reliability Engineer, Supercomputing](https://hotfix.jobs/jobs/e9263e4b-66f2-4203-b8da-b88ba1855dfe) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Performance Engineer, Inference Systems](https://hotfix.jobs/jobs/c4e8b3e8-25ad-4a83-b11d-bebf411d63f1) - Anthropic - San Francisco, CA - $350k – $850k/yr
- [Site Reliability Engineer (SRE)](https://hotfix.jobs/jobs/893a897e-0421-40b8-a8ac-f61c0b30d5ff) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Research Engineer, Infrastructure, Training Systems](https://hotfix.jobs/jobs/9cc8be27-572e-4c19-97d0-8a7ae54e41a8) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Research Engineer, Infrastructure, RL Systems](https://hotfix.jobs/jobs/71d52d2b-ffa8-42f6-a7af-3a117bfd347b) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr

**Apply:** https://hotfix.jobs/jobs/2a952a8c-f34c-4fdd-a219-c8ac22399145
**Canonical:** https://hotfix.jobs/jobs/2a952a8c-f34c-4fdd-a219-c8ac22399145