# Senior Research Scientist, Model Evaluation

**Company:** [Cohere](https://hotfix.jobs/companies/cohere)
**Location:** San Francisco, CA, Seattle, WA, New York, NY, Toronto, Canada, London, United Kingdom
**Role:** AI Research
**Experience:** 5+ years
**Skills:** LLMs, Llm Evaluation, Evaluation Benchmarks, Llm Judges, Data Synthesis, Software Engineering, Prototyping, Data Quality, Ai Capabilities, Model Performance
**Posted:** 2025-10-22

> Conducts research and builds benchmarks, methods, and infrastructure to evaluate the capabilities and progress of large language models. The role requires strong software engineering, prototyping, data-quality review, and rigorous measurement skills.

## Job Description

## Responsibilities
- Create ambitious new evaluation benchmarks that push the limits of what models can accomplish.
- Work on highly cross-functional teams to translate model feedback into trustworthy, repeatable evaluations.
- Conduct research to advance the state of the art in LLM evaluation methods, including:
  - Training LLM judges.
  - Refining LLM-based data synthesis pipelines.
  - Improving evaluation efficiency.
- Build scalable and reusable tools for analyzing model performance.

## Requirements and Qualifications
- Experience rapidly building prototypes that demonstrate the boundaries of LLM capabilities.
- Experience developing resources to measure model capabilities.
- Extensive experience reviewing complex data and LLM outputs to ensure high data quality.
- Strong focus on rigorously measuring AI capabilities and aligning measurements with the capabilities being evaluated.
- Strong software engineering skills.

## Compensation and Benefits
- Weekly lunch stipend of $75, £75, or equivalent in local currency.
- Full health and dental benefits, including a separate mental health budget.
- RRSP matching, 401(k), or pension scheme.
- 100% parental leave top-up for up to six months for either parent.
- Annual enrichment benefits for arts and culture, fitness and wellness, quality time, and workspace improvements.
- Education and learning stipend for conferences, courses, and coaching.
- Six weeks of paid vacation.
- Travel budget for remote employees visiting other offices and an annual company offsite.
- Coworking benefit for employees who are not near an office.
- $500 home office stipend.

## Similar jobs

- [Senior Research Engineer, Safety](https://hotfix.jobs/jobs/895361c1-4855-4f5a-8be7-da7e4af996f6) - Decagon - San Francisco, CA - $200k – $400k/yr
- [Senior Clinical Specialist, AI Systems](https://hotfix.jobs/jobs/5c4c71bf-2abf-42c7-bb48-8e763cabead7) - Function Health - Remote
- [AI Solutions Architect](https://hotfix.jobs/jobs/e4b5119d-ef06-4f6a-9eed-e53f14f7406a) - Gusto - Denver, CO - $168k – $247k/yr
- [Applied AI, Research Engineer](https://hotfix.jobs/jobs/50836d6b-5367-4c06-a6bf-5ef143d5366c) - Anthropic - San Francisco, CA - $300k – $400k/yr
- [Senior Research Scientist, Open Ecosystem](https://hotfix.jobs/jobs/c6703bf2-8e4d-44a2-b14a-da5e3e6b4dc0) - Ai2 - Seattle, WA - $170k – $270k/yr

**Apply:** https://hotfix.jobs/jobs/bb833bec-88a0-4cb2-bb45-fb8452293796
**Canonical:** https://hotfix.jobs/jobs/bb833bec-88a0-4cb2-bb45-fb8452293796