# Senior Product Operations Manager, Evaluation Quality

**Company:** [Harvey](https://hotfix.jobs/companies/harvey)
**Location:** San Francisco, CA
**Role:** Product Operations
**Salary:** $155k – $233k/yr
**Experience:** 5+ years
**Skills:** product operations, quality assurance, evaluation operations, rubric design, inter-rater reliability, calibration, sampling, benchmarking, Data Analysis, operational dashboards, llm-as-judge, human-in-the-loop, Vendor Management, legal research, process documentation
**Posted:** 2026-08-04

> Owns the quality standards, methodology, and operational systems for human evaluations, including rater calibration, QA, benchmarks, and error taxonomies. The role requires 6+ years of product, research, evaluation, or quality operations experience and strong measurement and cross-functional collaboration skills.

## Job Description

## Responsibilities
- Own the quality bar for human evaluations, defining standards that make evaluation data rigorous, calibrated, reproducible, and decision-grade.
- Author and maintain evaluation guidelines, instructions, benchmarks, and gold references for contract attorneys; arbitrate ambiguous or disputed judgments.
- Standardize rubric and evaluation design into repeatable templates and a documented methodology.
- Own contract-attorney quality, including onboarding, calibration training, inter-rater reliability, and ongoing feedback loops.
- Define and maintain failure-mode and error taxonomies that turn evaluation results into structured, prioritized signals.
- Run QA on vendor and contract-attorney deliverables before results inform launch decisions.
- Maintain quality standards across jurisdictions and non-English geographies.
- Establish a standard approach to analyzing evaluation results and build lightweight dashboards tracking rater capability and evaluation-program health.
- Support evaluation certification before work scales to contract attorneys.
- Partner with Analytics to determine how human evaluations align with online signals and provide expanded insights.

## Requirements
- 6+ years of experience in product operations, research operations, evaluation/QA operations, or quality program management.
- Experience owning quality inputs such as guidelines, instructions, benchmarks, and QA procedures for complex, expert-driven, or human-in-the-loop work.
- Experience onboarding, training, and calibrating distributed expert raters, annotators, or reviewers.
- Grounding in calibration, inter-rater reliability, sampling, and rubric design.
- Comfort interpreting evaluation data, natively or with AI tool support.
- Experience scaling and streamlining quality processes under shipping pressure, with an emphasis on documentation and reproducibility.
- Ability to work with domain experts and translate nuanced judgment into repeatable, documented standards.
- Strong cross-functional coordination across Product, Engineering, Research, legal research, and vendors.
- Clear communication, stakeholder credibility, bias toward action, and high ownership.

## Nice-to-haves
- Experience in legal technology or regulated industries.
- Experience owning quality across multiple markets, languages, or jurisdictions.
- Experience building calibration, inter-rater reliability, or capability-tracking systems for annotation or evaluation pipelines.
- Experience transitioning evaluation work in-house or improving evaluation ROI.
- Familiarity with LLM-as-judge and automated evaluation alongside human evaluation.
- Experience at an early-stage, hyper-growth startup or a world-class product/platform operations organization.

## Compensation
- $155,400 - $233,200 USD, depending on location.

## Similar roles

- [Senior Product Operations Manager](https://hotfix.jobs/jobs/62502584-4d39-49e4-bb94-bbe22ea1824c) - Bestow - Remote - $160k – $185k/yr
- [Senior Product Operations Manager, Evaluation](https://hotfix.jobs/jobs/8ee6af14-ca82-4595-9c83-a389984364fc) - Harvey - San Francisco, CA - $150k – $210k/yr
- [Product Operations Manager](https://hotfix.jobs/jobs/64d4b578-15a6-4806-b599-0c191b43f5d0) - Decagon - San Francisco, CA - $150k – $230k/yr
- [Senior Product Operations Manager](https://hotfix.jobs/jobs/712e4d4e-300e-4222-a4e1-360b176c0609) - Calendly - San Francisco, CA - $163k – $247k/yr
- [Enterprise Context Architect](https://hotfix.jobs/jobs/52d18a4e-bee5-4f60-a1e7-a949a9ab25d1) - Dropbox - Remote - $141k – $215k/yr

**Apply:** https://hotfix.jobs/jobs/50dbb5dc-9325-4f90-ac60-b7519f821652
**Canonical:** https://hotfix.jobs/jobs/50dbb5dc-9325-4f90-ac60-b7519f821652