# Research Engineer, Model Evaluations

**Company:** [Anthropic](https://hotfix.jobs/companies/anthropic)
**Location:** San Francisco, CA, New York, NY
**Role:** ML Engineering
**Salary:** $500k – $850k/yr
**Experience:** 5+ years
**Skills:** Python, Distributed Systems, Data Pipelines, LLMs, Data Visualization, Observability, Monitoring, Statistics, Experimental Design, Ml Training Infrastructure, Prompting, Sampling, Scaffolding
**Posted:** 2026-07-09

> Build and run evaluations to measure Claude's capabilities, safety, and performance. Design metrics, implement scalable distributed eval infrastructure and dashboards, debug training runs, and partner with researchers to characterize and improve AI systems.

## Job Description

## Key Responsibilities
- Design and run new evaluations of Claude's capabilities (reasoning, agentic behavior, knowledge, safety properties) and produce visualizations that make results legible to researchers and decision-makers.
- Build and harden the distributed eval execution platform for reliable execution of hundreds of evals against checkpoints during production RL training runs.
- Own dashboards used by researchers and leadership to monitor model health during training; improve signal-to-noise, reduce latency, and prevent missed regressions.
- Debug anomalous eval results mid-training-run to determine if caused by model change or infrastructure issue; communicate clearly under time pressure.
- Improve tooling, libraries, and workflows for researchers to implement and iterate on evaluations.
- Partner with research teams across the full lifecycle of a new capability, from defining metrics to interpreting results during training.
- Run experiments to characterize effects of prompting, sampling, and scaffolding on internal and industry benchmarks.
- Communicate evaluations and results to internal stakeholders and, where appropriate, external audiences.

## Minimum Qualifications
- Strong Python programming skills, including production or research infrastructure.
- Experience building or operating distributed systems, data pipelines, or other infrastructure that needs to be reliable at scale.
- Clear written and verbal communication, especially explaining technical results to non-specialists.
- Comfort operating in an on-call or production-support capacity during live training runs.
- Care about societal impacts of work and interest in steering powerful AI to be safe and beneficial.

## Preferred Qualifications
- Hands-on experience using large language models such as Claude, including prompting, sampling, and scaffolding.
- Background in data visualization and track record of building trusted dashboards.
- Experience developing robust evaluation metrics for language models.
- Experience with observability, monitoring, or experiment-tracking systems.
- Background in statistics and experimental design.
- Experience with large-scale dataset sourcing, curation, and processing.
- Experience running or supporting ML training infrastructure.
- Bias toward picking up slack and operating flexibly across team boundaries.
- Enjoy pair programming.

## Representative Projects
- Stand up a new eval for a specific reasoning capability from scratch: define task, build dataset, implement scoring, validate against known signals, and ship a dashboard.
- Diagnose a mid-training regression in an eval suite within hours to determine if caused by model, harness, data, or infrastructure.
- Stabilize a flaky distributed eval pipeline with better retries, observability, and faster feedback.
- Partner with a research team on a new capability to define "good" and translate into measurable artifacts.

## Compensation
Annual Salary: $500,000—$850,000 USD

Minimum education: Bachelor’s degree or equivalent. Required field of study relevant to the role.

## Similar roles

- [Research Engineer, Computer Use](https://hotfix.jobs/jobs/c890b68c-75a2-4f4e-bd91-5effe3e64ac7) - Anthropic - San Francisco, CA - $500k – $850k/yr
- [Research Engineer, Machine Learning (RL Velocity)](https://hotfix.jobs/jobs/031b6eea-f217-47c1-9cd0-5da28939ee7f) - Anthropic - San Francisco, CA - $500k – $850k/yr
- [Research Engineer, Privacy](https://hotfix.jobs/jobs/3220c5d0-0e5f-4869-b133-1ed126b4208a) - OpenAI - San Francisco, CA - $380k – $445k/yr
- [TL, Research Inference](https://hotfix.jobs/jobs/7e46ddd9-df9f-4337-bcd7-b681febdc04b) - OpenAI - San Francisco, CA - $380k – $555k/yr
- [Research Infrastructure Engineer](https://hotfix.jobs/jobs/ea3efdd7-00d7-492b-a405-ceec9a86fd52) - Thinking Machines Lab - San Francisco, CA - $350k – $475k/yr

**Apply:** https://hotfix.jobs/jobs/4f2b9b5a-da09-4541-b6ab-851b56ee2013
**Canonical:** https://hotfix.jobs/jobs/4f2b9b5a-da09-4541-b6ab-851b56ee2013