Research Engineer, RL Infrastructure and Reliability (Knowledge Work)

Key Responsibilities

• Serve as the dedicated reliability owner for the Knowledge Work training environments, providing continuity of context and reducing the operational overhead of rotating ownership
• Own a clean, canonical set of evaluation tools and processes for Knowledge Work capabilities, including the process used for model releases
• Build and automate observability, dashboards, and operational tooling for our training environments and evaluation systems, with an emphasis on high signal-to-noise: a small set of trusted metrics and alerts rather than sprawling instrumentation
• Proactively harden environments and evaluation systems through load testing, fault injection, and stress testing at realistic scale, so failures surface early rather than during critical training work
• Act as the primary point of contact for partner training and infrastructure teams when issues in our environments arise, and drive incidents to resolution
• Reduce the operational burden on researchers so they can stay focused on research

Minimum Qualifications

• Highly experienced Python engineer who ships reliable, well-instrumented code that teammates trust in production
• Demonstrated experience operating ML or distributed systems at scale, including significant on-call and incident-response experience
• Strong SRE or production-engineering mindset — reaching for SLOs, load tests, and failure injection before reaching for more dashboards
• Foundational ML knowledge sufficient to understand what a training environment or evaluation is actually measuring, and recognize when an evaluation has become stale or gameable
• Able to read research code and reason evaluation integrity

Preferred Qualifications

• 5+ years of experience operating ML or distributed systems at scale
• Experience building or operating RL environments, agent harnesses, or LLM evaluation frameworks
• Familiarity with reward modeling, evaluation design, or detecting and mitigating reward hacking
• Experience with observability stacks (metrics, tracing, structured logging) and operational dashboard tooling
• Background in chaos engineering, fault injection, or large-scale load testing
• Experience with data quality pipelines, drift detection, or evaluation-set curation and versioning
• Familiarity with large-scale training or inference infrastructure (schedulers, multi-agent orchestration, sandboxed execution)
• Prior experience as a dedicated reliability or operations owner embedded within a research team