# Human Evaluation Researcher

**Company:** [Nuance Labs](https://hotfix.jobs/companies/nuance-labs)
**Location:** Seattle, WA
**Role:** UX Research
**Salary:** $160k – $190k/yr
**Experience:** 5+ years
**Skills:** Qualitative Research, Quantitative Research, Psychometrics, Experimental Design, Statistics, Cohen'S Kappa, Krippendorff'S Alpha, Ethnography, Contextual Inquiry, Crowdsourcing, Python, R, Llm-As-Judge
**Posted:** 2026-08-31

> Designs and operates human-evaluation studies for real-time AI avatars, translating subjective qualities such as naturalness, emotion, trust, and presence into reliable signals for model development and release decisions. Requires 5+ years of qualitative and quantitative human-subjects research experience.

## Job Description

## Responsibilities
- Design and run qualitative and quantitative human-subjects studies of AI avatars, including side-by-side comparisons, controlled rating experiments, interviews, think-alouds, diary studies, and longitudinal panels.
- Convert ambiguous judgments such as naturalness, emotional resonance, trust, and presence into reliable instruments, including rubrics, anchored scales, and annotation guidelines.
- Measure and improve inter-rater agreement while preserving meaningful signal.
- Use ethnographic methods such as observation of live conversations, contextual inquiry, and fieldwork to understand user experiences.
- Build the human-evaluation pipeline, including participant panels, rater training and calibration, tooling, and evaluation cadence tied to model releases.
- Calibrate automated and model-based metrics, including LLM-as-judge systems, against human judgment.
- Work directly with founders and modeling researchers to determine which models ship and what to train next.

## Requirements
- 5+ years designing and running human-subjects research in industry or academia, such as UX research, HCI, experimental psychology, behavioral science, or a related field.
- Demonstrated experience designing studies that produce aligned judgments about ambiguous qualities such as tone, emotion, quality, or trust.
- Strong qualitative research skills, including interviews, ethnography, and contextual inquiry.
- Strong quantitative research skills, including survey and psychometric design, experimental design, and statistics for rating and pairwise-comparison data.
- Expertise in agreement and reliability measures, including Cohen’s kappa and Krippendorff’s alpha.
- Ability to run rigorous, practical studies quickly and communicate findings clearly to ML researchers.

## Nice-to-haves
- Experience evaluating generative AI, including avatars, digital humans, speech or video generation, conversational agents, or emotion expression and recognition.
- Background in perceptual science or psychophysics.
- MS or PhD in a related field.
- Experience with human evaluation at scale, including crowdsourcing platforms, annotation tooling, and golden datasets.
- Ability to analyze data with Python or R.

## Compensation and Benefits
- $160,000–$190,000 base salary plus meaningful equity.
- Health plans, including an HDHP with approximately $2,000 in annual company HSA contributions.
- 15 days of PTO, 10 public holidays, and a full office closure at year-end.
- Weekday lunch, drinks, and snacks.
- Commuter benefits of up to $340 per month.
- 401(k) match.

## Similar jobs

- [Research Manager](https://hotfix.jobs/jobs/ad736e22-28a3-4b71-ab90-dfb936e5bbac) - Outset - San Francisco, CA - $150k – $190k/yr
- [Senior UX Researcher - Patient](https://hotfix.jobs/jobs/df554a58-34f7-4cd5-8e05-369b080fb915) - Rula - Remote - $166k – $203k/yr
- [Senior UX Researcher, Consumer Product](https://hotfix.jobs/jobs/a9675a00-d508-40d2-b087-060e3e094b34) - Reddit - Remote - $168k – $235k/yr
- [Senior User Researcher II](https://hotfix.jobs/jobs/0a0f9501-a664-45ca-aab1-ab154596f394) - Wrapbook - Remote - CA$149k – CA$205k/yr
- [UX Researcher, Quantitative](https://hotfix.jobs/jobs/fd3d7840-ff21-47a0-956c-805a40866e87) - Gusto - Denver, CO - $172k – $253k/yr

**Apply:** https://hotfix.jobs/jobs/246c4d95-0f54-4683-b26d-02c16fd2cab2
**Canonical:** https://hotfix.jobs/jobs/246c4d95-0f54-4683-b26d-02c16fd2cab2