Evals Infrastructure Tech Lead / Manager

Responsibilities

• Lead the team building the distributed systems that schedule, orchestrate, and execute evals for our frontier model training
• Own eval throughput and cost: compute allocation across suites, queueing against constrained accelerator pools, caching and reuse of eval work
• Build and scale the harnesses researchers use to define, run, and iterate on evals
• Make eval results trustworthy — determinism, reproducibility, and honest uncertainty quantification on reported metrics
• Ensure eval signal reaches the dashboards and reviews where launch decisions actually get made
• Contribute directly as an engineer while managing and growing the team, prioritizing its work, and coaching your reports

Requirements

• Led technical projects end-to-end on large-scale distributed systems, and have 1+ years managing engineers (or tech-lead-with-reports experience)
• Strong in Python and Rust
• Built high-throughput, fault-tolerant systems on cloud or on-prem accelerator fleets
• Care about measurement quality, not just pipeline uptime — you'd notice if a metric moved for the wrong reason
• Communicate well with researchers and can translate research needs into infrastructure
• Deeply interested in the transformative effects of advanced AI and committed to safe development
• Bachelor’s degree or an equivalent combination of education, training, and/or experience in a field relevant to the role

Nice-to-Haves

• Worked on LLM inference or training infrastructure
• Experience with eval or benchmarking systems, especially agentic evals requiring sandboxed execution
• Working statistical literacy — variance, confidence intervals, sample-size sufficiency for noisy metrics
• Experience with observability and regression detection over time-series metrics