Senior Product Operations Manager, Evaluation Quality

Responsibilities

• Own the quality bar for human evaluations, defining standards that make evaluation data rigorous, calibrated, reproducible, and decision-grade.
• Author and maintain evaluation guidelines, instructions, benchmarks, and gold references for contract attorneys; arbitrate ambiguous or disputed judgments.
• Standardize rubric and evaluation design into repeatable templates and a documented methodology.
• Own contract-attorney quality, including onboarding, calibration training, inter-rater reliability, and ongoing feedback loops.
• Define and maintain failure-mode and error taxonomies that turn evaluation results into structured, prioritized signals.
• Run QA on vendor and contract-attorney deliverables before results inform launch decisions.
• Maintain quality standards across jurisdictions and non-English geographies.
• Establish a standard approach to analyzing evaluation results and build lightweight dashboards tracking rater capability and evaluation-program health.
• Support evaluation certification before work scales to contract attorneys.
• Partner with Analytics to determine how human evaluations align with online signals and provide expanded insights.

Requirements

• 6+ years of experience in product operations, research operations, evaluation/QA operations, or quality program management.
• Experience owning quality inputs such as guidelines, instructions, benchmarks, and QA procedures for complex, expert-driven, or human-in-the-loop work.
• Experience onboarding, training, and calibrating distributed expert raters, annotators, or reviewers.
• Grounding in calibration, inter-rater reliability, sampling, and rubric design.
• Comfort interpreting evaluation data, natively or with AI tool support.
• Experience scaling and streamlining quality processes under shipping pressure, with an emphasis on documentation and reproducibility.
• Ability to work with domain experts and translate nuanced judgment into repeatable, documented standards.
• Strong cross-functional coordination across Product, Engineering, Research, legal research, and vendors.
• Clear communication, stakeholder credibility, bias toward action, and high ownership.

Nice-to-haves

• Experience in legal technology or regulated industries.
• Experience owning quality across multiple markets, languages, or jurisdictions.
• Experience building calibration, inter-rater reliability, or capability-tracking systems for annotation or evaluation pipelines.
• Experience transitioning evaluation work in-house or improving evaluation ROI.
• Familiarity with LLM-as-judge and automated evaluation alongside human evaluation.
• Experience at an early-stage, hyper-growth startup or a world-class product/platform operations organization.

Compensation

• $155,400 - $233,200 USD, depending on location.