# Research Engineer, Infrastructure, RL Systems

**Company:** [Thinking Machines Lab](https://hotfix.jobs/companies/thinking-machines-lab)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $350k – $475k/yr
**Skills:** PyTorch, JAX, Kubernetes, Slurm, Prometheus, Grafana, OpenTelemetry, Ppo, Dpo, RLHF
**Posted:** 2026-05-04

> Designs and optimizes infrastructure for scalable reinforcement learning training of large models, improving reliability, observability, and throughput. Collaborates with researchers to productionize RL algorithms; requires strong engineering skills and deep learning framework knowledge.

## Job Description

## What You’ll Do
- Design, build, and optimize the infrastructure that powers large-scale reinforcement learning and post-training workloads.
- Improve the reliability and scalability of RL training pipeline, distributed RL workloads, and training throughput.
- Develop shared monitoring and observability tools to ensure high uptime, debuggability, and reproducibility for RL systems.
- Collaborate with researchers to translate algorithmic ideas into production-grade training pipelines.
- Build evaluation and benchmarking infrastructure that measures model progress on helpfulness, safety, and factuality.
- Publish and share learnings through internal documentation, open-source libraries, or technical reports that advance the field of scalable AI infrastructure.

## Skills and Qualifications

**Minimum qualifications:**
- Bachelor’s degree or equivalent experience in computer science, electrical engineering, statistics, machine learning, physics, robotics, or similar.
- Strong engineering skills, ability to contribute performant, maintainable code and debug in complex codebases.
- Understanding of deep learning frameworks (e.g., **PyTorch**, **JAX**) and their underlying system architectures.
- Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.
- A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

**Preferred qualifications:**
- Experience training or supporting large-scale language models with tens of billions of parameters or more.
- Experience working with reinforcement learning workloads (e.g., **PPO**, **DPO**, **RLHF**, or reward modeling).
- Background in high-performance or reliability engineering — distributed training frameworks and cluster orchestration (**Kubernetes**, **Slurm**).
- Familiarity with monitoring and observability tools (**Prometheus**, **Grafana**, **OpenTelemetry**).
- Contributions to large-scale ML research or infrastructure, open-source frameworks, or internal performance optimization efforts.

## Logistics
**Compensation:** Depending on background, skills and experience, the expected annual salary range for this position is **$350,000 - $475,000 USD**.

**Benefits:** Generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.

## Similar roles

- [Reliability Engineer, Supercomputing](https://hotfix.jobs/jobs/e9263e4b-66f2-4203-b8da-b88ba1855dfe) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Network Engineer, Supercomputing](https://hotfix.jobs/jobs/2a952a8c-f34c-4fdd-a219-c8ac22399145) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Performance Engineer, Inference Systems](https://hotfix.jobs/jobs/c4e8b3e8-25ad-4a83-b11d-bebf411d63f1) - Anthropic - San Francisco, CA - $350k – $850k/yr
- [Site Reliability Engineer (SRE)](https://hotfix.jobs/jobs/893a897e-0421-40b8-a8ac-f61c0b30d5ff) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Research Engineer, Infrastructure, Training Systems](https://hotfix.jobs/jobs/9cc8be27-572e-4c19-97d0-8a7ae54e41a8) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr

**Apply:** https://hotfix.jobs/jobs/71d52d2b-ffa8-42f6-a7af-3a117bfd347b
**Canonical:** https://hotfix.jobs/jobs/71d52d2b-ffa8-42f6-a7af-3a117bfd347b