# Research Scientist, Benchmarks & Evaluations

**Company:** [Protege](https://hotfix.jobs/companies/protege)
**Location:** Remote
**Role:** AI Research
**Skills:** Llm Evaluation, Agent Evaluation, Prompting, Scaffolding, Inter-Rater Reliability, Item Response Theory, Contamination Analysis, Statistical Modeling, Annotator Quality Control, Scientific Writing
**Posted:** 2026-05-21

> Lead the design and validation of trustworthy benchmarks and evaluations for frontier AI models, including agentic and domain-specific tasks. Own the science of evals and annotator quality, publishing research and translating findings into production datasets.

## Job Description

## What you’ll do
- Design tasks and benchmarks that distinguish capability levels across frontier models — including agentic, reasoning-heavy, and domain-specific (healthcare, finance, scientific) settings.
- Validate evaluations rigorously: run human baselines, analyze inter-rater reliability, study how elicitation and scaffolding shift results, and quantify what’s signal versus noise.
- Develop the “science of evals” at Protege — including item response theory, contamination analysis, predictive validity studies, and statistical frameworks for comparing models with appropriate uncertainty.
- Run evaluations on current frontier models, sometimes in collaboration with partners at AI labs, enterprises, and government.
- Publish research that establishes Protege as the standard-setter for evaluation data, and contribute to the broader AI community’s understanding of what good evals look like.
- Translate findings into product, working closely with the data and engineering teams to turn research into evaluation datasets customers can deploy.
- Partner with outsourced annotation vendors to own the statistical machinery that determines which annotators are trusted, on which tasks, and by how much — translating that into trustworthiness scores customers can rely on.

## What we’re looking for
- Advanced degree (PhD preferred, or MS/BS plus equivalent industry experience) in a quantitative field — applied econometrics with AI experience, quantitative finance, computer science, engineering, statistics/mathematics or any applied research discipline.
- Hands-on experience evaluating LLMs, agents, or other ML systems — including prompting, scaffolding, and fluency with the tooling researchers use to run evals at scale.
- Experience with annotator quality and inter-rater reliability — designing labeling protocols, computing agreement statistics, and reasoning about annotator bias and calibration.
- Excellent scientific writing and communication — synthesize technical findings into narratives that frontier labs, enterprise customers, and policymakers can act on.
- A bias toward velocity — know which pipelines need to be production-grade and which can be scrappy, and get reliable results fast.

## Bonus
- Experience with RL evaluation techniques — reward modeling, off-policy evaluation, evals for RLHF/RLAIF or agentic RL pipelines.
- Ability to navigate new customer architectures, data systems, and requirements quickly.
- Experience with latent-variable models of annotator skill (Dawid-Skene, MACE, IRT-style approaches) or with running large expert-annotator panels in regulated domains.
- Track record of published benchmarks or evaluation papers the field has adopted.

## Similar roles

- [ML Scientist](https://hotfix.jobs/jobs/401a3fda-7a0b-4af9-aa9c-134e4ae4b19c) - Sesame - San Francisco, CA - $190k – $320k/yr
- [Research Scientist](https://hotfix.jobs/jobs/34963160-7769-44e3-8e41-872372a811ed) - OpenRouter - Remote
- [Research Engineer, Benchmarks](https://hotfix.jobs/jobs/71636332-204e-4abc-b0b5-c2b5c2ca3726) - hud - San Francisco, CA
- [Research Scientist](https://hotfix.jobs/jobs/5c6acbfc-713c-4c38-bb2c-7a8d9dbc74d1) - Spotify - Remote
- [Research Scientist, Life Sciences](https://hotfix.jobs/jobs/3612b705-0fb1-4f66-a4a1-0b28ba382b5c) - Anthropic - San Francisco, CA - $300k – $320k/yr

**Apply:** https://hotfix.jobs/jobs/5c2a882a-7f55-40af-9868-449a7f3d3aad
**Canonical:** https://hotfix.jobs/jobs/5c2a882a-7f55-40af-9868-449a7f3d3aad