# Research Engineer, RL Infrastructure and Reliability (Knowledge Work)

**Company:** [Anthropic](https://hotfix.jobs/companies/anthropic)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $350k – $850k/yr
**Skills:** Python, Machine Learning, Distributed Systems, SRE, Observability, Load Testing, Fault Injection, Chaos Engineering, Rl Environments, Llm Evaluation, Reward Modeling, Data Pipelines, Kubernetes, Tracing, Metrics
**Posted:** 2026-04-23

> Owns reliability, observability, and infrastructure for Knowledge Work team's RL training environments and evaluations. Ensures stability at scale through proactive hardening, SLOs, load testing, and incident response for ML systems.

## Job Description

## Key Responsibilities

- Serve as the dedicated reliability owner for the Knowledge Work training environments, providing continuity of context and reducing the operational overhead of rotating ownership
- Own a clean, canonical set of evaluation tools and processes for Knowledge Work capabilities, including the process used for model releases
- Build and automate observability, dashboards, and operational tooling for our training environments and evaluation systems, with an emphasis on high signal-to-noise: a small set of trusted metrics and alerts rather than sprawling instrumentation
- Proactively harden environments and evaluation systems through load testing, fault injection, and stress testing at realistic scale, so failures surface early rather than during critical training work
- Act as the primary point of contact for partner training and infrastructure teams when issues in our environments arise, and drive incidents to resolution
- Reduce the operational burden on researchers so they can stay focused on research

## Minimum Qualifications

- Highly experienced Python engineer who ships reliable, well-instrumented code that teammates trust in production
- Demonstrated experience operating ML or distributed systems at scale, including significant on-call and incident-response experience
- Strong SRE or production-engineering mindset — reaching for SLOs, load tests, and failure injection before reaching for more dashboards
- Foundational ML knowledge sufficient to understand what a training environment or evaluation is actually measuring, and recognize when an evaluation has become stale or gameable
- Able to read research code and reason evaluation integrity

## Preferred Qualifications

- 5+ years of experience operating ML or distributed systems at scale
- Experience building or operating RL environments, agent harnesses, or LLM evaluation frameworks
- Familiarity with reward modeling, evaluation design, or detecting and mitigating reward hacking
- Experience with observability stacks (metrics, tracing, structured logging) and operational dashboard tooling
- Background in chaos engineering, fault injection, or large-scale load testing
- Experience with data quality pipelines, drift detection, or evaluation-set curation and versioning
- Familiarity with large-scale training or inference infrastructure (schedulers, multi-agent orchestration, sandboxed execution)
- Prior experience as a dedicated reliability or operations owner embedded within a research team

## Similar roles

- [Reliability Engineer, Supercomputing](https://hotfix.jobs/jobs/e9263e4b-66f2-4203-b8da-b88ba1855dfe) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Network Engineer, Supercomputing](https://hotfix.jobs/jobs/2a952a8c-f34c-4fdd-a219-c8ac22399145) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Performance Engineer, Inference Systems](https://hotfix.jobs/jobs/c4e8b3e8-25ad-4a83-b11d-bebf411d63f1) - Anthropic - San Francisco, CA - $350k – $850k/yr
- [Site Reliability Engineer (SRE)](https://hotfix.jobs/jobs/893a897e-0421-40b8-a8ac-f61c0b30d5ff) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr
- [Research Engineer, Infrastructure, Training Systems](https://hotfix.jobs/jobs/9cc8be27-572e-4c19-97d0-8a7ae54e41a8) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr

**Apply:** https://hotfix.jobs/jobs/66d55393-cd19-4b15-8e06-c3ae182793d0
**Canonical:** https://hotfix.jobs/jobs/66d55393-cd19-4b15-8e06-c3ae182793d0