Key Responsibilities
- Design and run new evaluations of Claude's capabilities (reasoning, agentic behavior, knowledge, safety properties) and produce visualizations that make results legible to researchers and decision-makers.
- Build and harden the distributed eval execution platform for reliable execution of hundreds of evals against checkpoints during production RL training runs.
- Own dashboards used by researchers and leadership to monitor model health during training; improve signal-to-noise, reduce latency, and prevent missed regressions.
- Debug anomalous eval results mid-training-run to determine if caused by model change or infrastructure issue; communicate clearly under time pressure.
- Improve tooling, libraries, and workflows for researchers to implement and iterate on evaluations.
- Partner with research teams across the full lifecycle of a new capability, from defining metrics to interpreting results during training.
- Run experiments to characterize effects of prompting, sampling, and scaffolding on internal and industry benchmarks.
- Communicate evaluations and results to internal stakeholders and, where appropriate, external audiences.
Minimum Qualifications
- Strong Python programming skills, including production or research infrastructure.
- Experience building or operating distributed systems, data pipelines, or other infrastructure that needs to be reliable at scale.
- Clear written and verbal communication, especially explaining technical results to non-specialists.
- Comfort operating in an on-call or production-support capacity during live training runs.
- Care about societal impacts of work and interest in steering powerful AI to be safe and beneficial.
Preferred Qualifications
- Hands-on experience using large language models such as Claude, including prompting, sampling, and scaffolding.
- Background in data visualization and track record of building trusted dashboards.
- Experience developing robust evaluation metrics for language models.
- Experience with observability, monitoring, or experiment-tracking systems.
- Background in statistics and experimental design.
- Experience with large-scale dataset sourcing, curation, and processing.
- Experience running or supporting ML training infrastructure.
- Bias toward picking up slack and operating flexibly across team boundaries.
- Enjoy pair programming.
Representative Projects
- Stand up a new eval for a specific reasoning capability from scratch: define task, build dataset, implement scoring, validate against known signals, and ship a dashboard.
- Diagnose a mid-training regression in an eval suite within hours to determine if caused by model, harness, data, or infrastructure.
- Stabilize a flaky distributed eval pipeline with better retries, observability, and faster feedback.
- Partner with a research team on a new capability to define "good" and translate into measurable artifacts.
Compensation
Annual Salary: $500,000—$850,000 USD
Minimum education: Bachelor’s degree or equivalent. Required field of study relevant to the role.