Research Lead, Training Insights

Responsibilities:

• Build new novel and long-horizon evaluations
• Develop novel measurement approaches for understanding how model capabilities emerge and evolve during RL training
• Lead strategic evaluation coverage across the company
• Shape the evaluation narrative for model releases
• Lead and mentor a small team of researchers and research engineers, setting research direction and fostering a culture of rigorous, creative research
• Design evaluation frameworks that balance scientific rigor with the practical demands of production training schedules
• Build and maintain relationships across Anthropic's research organization to ensure evaluation insights inform training and deployment decisions
• Contribute to the broader research community through publications, open-source contributions, or external engagement on evaluation best practices

You may be a good fit if you:

• Have significant experience designing and running evaluations for large language models or similar complex ML systems
• Have led technical projects or teams, either formally or through sustained ownership of critical research directions
• Are equally comfortable designing experiments and writing code—you can move between research and implementation fluidly
• Think strategically about what to measure and why, not just how to measure it
• Can synthesize information across multiple teams and workstreams to form a coherent picture of model capabilities
• Communicate complex technical findings clearly to both technical and non-technical audiences
• Are results-oriented and thrive in fast-paced environments where priorities shift based on research findings
• Care deeply about AI safety and want your work to directly influence how capable AI systems are developed and deployed

Strong candidates may also have:

• Experience building evaluations for long-horizon or agentic tasks
• Deep familiarity with Reinforcement Learning training dynamics and how model behavior changes during training
• Published research in machine learning evaluation, benchmarking, or related areas
• Experience with safety evaluation frameworks and red teaming methodologies
• Background in psychometrics, experimental psychology, or other measurement-focused disciplines
• A track record of communicating evaluation results to inform high-stakes decisions about model development or deployment
• Experience managing or mentoring researchers and engineers

Representative projects:

• Designing and implementing a suite of long-horizon evaluations that test model capabilities on tasks requiring sustained reasoning, planning, and tool use over extended interactions
• Building systems to track capability development across RL training checkpoints, surfacing insights about when and how specific capabilities emerge
• Conducting a cross-org audit of evaluation coverage, identifying blind spots, and prioritizing new evaluations to fill critical gaps across Pretraining, RL, Inference, and Product
• Developing the evaluation methodology and narrative for a major model release, working with research leads and communications to clearly characterize model capabilities and limitations
• Researching and prototyping novel evaluation approaches for capabilities that are difficult to measure with existing benchmarks
• Leading a team effort to build reusable evaluation infrastructure that serves multiple teams across the research organization