Skip to content
AnthropicAnthropicSan Francisco, CA

Research Engineer, Model Evaluations

Build and run evaluations to measure Claude's capabilities, safety, and performance. Design metrics, implement scalable distributed eval infrastructure and dashboards, debug training runs, and partner with researchers to characterize and improve AI systems.

500k – 850k/yr
Hybrid5+ YOEML Engineering

About the role

Key Responsibilities

  • Design and run new evaluations of Claude's capabilities (reasoning, agentic behavior, knowledge, safety properties) and produce visualizations that make results legible to researchers and decision-makers.
  • Build and harden the distributed eval execution platform for reliable execution of hundreds of evals against checkpoints during production RL training runs.
  • Own dashboards used by researchers and leadership to monitor model health during training; improve signal-to-noise, reduce latency, and prevent missed regressions.
  • Debug anomalous eval results mid-training-run to determine if caused by model change or infrastructure issue; communicate clearly under time pressure.
  • Improve tooling, libraries, and workflows for researchers to implement and iterate on evaluations.
  • Partner with research teams across the full lifecycle of a new capability, from defining metrics to interpreting results during training.
  • Run experiments to characterize effects of prompting, sampling, and scaffolding on internal and industry benchmarks.
  • Communicate evaluations and results to internal stakeholders and, where appropriate, external audiences.

Minimum Qualifications

  • Strong Python programming skills, including production or research infrastructure.
  • Experience building or operating distributed systems, data pipelines, or other infrastructure that needs to be reliable at scale.
  • Clear written and verbal communication, especially explaining technical results to non-specialists.
  • Comfort operating in an on-call or production-support capacity during live training runs.
  • Care about societal impacts of work and interest in steering powerful AI to be safe and beneficial.

Preferred Qualifications

  • Hands-on experience using large language models such as Claude, including prompting, sampling, and scaffolding.
  • Background in data visualization and track record of building trusted dashboards.
  • Experience developing robust evaluation metrics for language models.
  • Experience with observability, monitoring, or experiment-tracking systems.
  • Background in statistics and experimental design.
  • Experience with large-scale dataset sourcing, curation, and processing.
  • Experience running or supporting ML training infrastructure.
  • Bias toward picking up slack and operating flexibly across team boundaries.
  • Enjoy pair programming.

Representative Projects

  • Stand up a new eval for a specific reasoning capability from scratch: define task, build dataset, implement scoring, validate against known signals, and ship a dashboard.
  • Diagnose a mid-training regression in an eval suite within hours to determine if caused by model, harness, data, or infrastructure.
  • Stabilize a flaky distributed eval pipeline with better retries, observability, and faster feedback.
  • Partner with a research team on a new capability to define "good" and translate into measurable artifacts.

Compensation

Annual Salary: $500,000—$850,000 USD

Minimum education: Bachelor’s degree or equivalent. Required field of study relevant to the role.

Skills

PythonDistributed SystemsData PipelinesLLMsData VisualizationObservabilityMonitoringStatisticsExperimental DesignMl Training InfrastructurePromptingSamplingScaffolding

Similar roles

ML Engineering jobs
Anthropic

Research Engineer, Computer Use

AnthropicSan Francisco, CA +2

Research Engineer advancing Claude's computer use capabilities through experiments, RL environments, evaluations, and infrastructure for perception and agentic tasks. Requires Python, ML training/evaluation experience, and a focus on safe AI.

500k – 850k/yr
Hybrid5+ YOEML Engineering
Anthropic

Research Engineer, Machine Learning (RL Velocity)

AnthropicSan Francisco, CA +1

Builds and optimizes RL training infrastructure, removes bottlenecks in the RL stack, and partners with researchers to accelerate model development at scale. Requires strong software engineering, ML infra experience, and comfort across the stack.

500k – 850k/yr
HybridML Engineering
OpenAI

Research Engineer, Privacy

OpenAISan Francisco, CA

Research Engineer on OpenAI's Privacy team designing and prototyping privacy-preserving ML algorithms like differential privacy and federated learning at scale. Requires hands-on PETs experience, fluency in PyTorch/JAX, and a track record implementing or publishing novel privacy work.

380k – 445k/yr
HybridML Engineering
OpenAI

TL, Research Inference

OpenAISan Francisco, CA

Leads development of high-performance inference systems for large-scale AI models, optimizing execution paths, distributed GPU inference, and operators. Partners with research teams to enable efficient exploration of new architectures grounded in real scalability constraints.

380k – 555k/yr
On-siteML Engineering
Thinking Machines Lab

Research Infrastructure Engineer

Thinking Machines LabSan Francisco, CA

Build and operate research infrastructure like evaluation frameworks, RL training systems, and experiment tracking platforms. Partner directly with ML researchers to identify bottlenecks, ensure high adoption of tools, and accelerate research velocity.

350k – 475k/yr
On-siteML Engineering