Product Manager, APEX
Own the strategy, integrity, infrastructure, and adoption of Mercor’s APEX AI evaluation products and public leaderboards. The role requires 5+ years in product management or a customer-facing technical role, strong evaluation literacy, technical fluency, and the ability to align research, engineering, operations, and go-to-market teams.
About the job
Responsibilities
- Own the roadmap and portfolio strategy for APEX benchmarks, evaluation products, infrastructure, and public and private leaderboards.
- Evaluate and prioritize new benchmark proposals against customer demand and company strategy.
- Enforce evaluation integrity through contamination policies, holdout strategies, versioning, auditability, and release cadence.
- Own the end-to-end evaluation pipeline, including harness execution, grading, model onboarding, hyperparameter scaffolds, cost and latency reporting, and leaderboard publication.
- Work with frontier AI labs and enterprise customers to understand evaluation needs, present results, and inform future benchmarks.
- Support go-to-market launches, partnerships, and thought leadership.
- Connect leaderboard demand signals to loss-analysis investments and dataset production; track adoption, usage, and revenue impact.
- Write product specifications and PRDs, inspect trajectories, spot-check failures, and make small code changes to improve the system.
Requirements
- 5+ years of experience in product management, technical program management, or a customer-facing technical role.
- Prior experience in software engineering, machine learning, or data science is strongly preferred.
- Fluency in rubric design, inter-rater reliability, agentic harnesses, contamination, overfitting, and evaluation uncertainty.
- Ability to distinguish genuine capability gaps from measurement noise.
- Strong public judgment and the ability to publish neutral, precise, and defensible model evaluations.
- Strong instincts for identifying valuable capabilities and meaningful results.
- Track record of building a product, program, or business line from the ground up.
- High ownership and accountability for outcomes.
- Ability to work independently in ambiguous contexts.
- Ability to align research, engineering, operations, and go-to-market stakeholders around a shared methodology.
Benefits
- Bi-annual performance bonus structure
- Generous equity grant vested over 4 years
- Up to $15,000 relocation bonus
- $10,000 housing bonus for employees living within 0.5 miles of the office
- $1,500 monthly meal stipend
- Free Equinox membership
- $200 monthly laundry reimbursement
- $200 monthly personal wellness reimbursement
- Health, dental, and vision insurance
Skills
Product Management, Technical Program Management, Machine Learning, Data Science, Rubric Design, Inter-Rater Reliability, Agentic Harnesses, Contamination Testing, Overfitting, Evaluation Pipelines, Leaderboard Design, Python
Similar jobs
Product Management jobs
SpotOn
Chicago, ILOwn the strategy and execution of SpotOn’s end-to-end payments product, covering acceptance, processing, settlement, and merchant funding. The role requires 5+ years of product management experience, including substantial payments or fintech expertise, and strong cross-functional leadership.
Tessera Labs
San Jose, CAOwns strategy and execution for AI-driven enterprise products, translating customer workflows and model capabilities into scalable features. The role partners across engineering, data science, GTM, and customer teams and requires B2B product experience, enterprise-platform expertise, and strong analytical judgment.
Mercor
San Francisco, CA
Owns Mercor’s Trust and Safety product, including fraud detection, risk signals, enforcement, and operations tooling. The role requires 5+ years in product or a related technical risk function, strong data and AI fluency, and the judgment to balance false positives against missed fraud.
Coinbase
United States
Owns strategy and execution for enterprise identity, organization management, and multi-user experiences across Coinbase business products. The role requires at least five years of product management experience and expertise in identity protocols, security, and cross-functional platform launches.
Fivetran
Oakland, CAOwn the vision, strategy, roadmap, and growth of a portfolio of data connectors and analytical use cases. The role requires product management experience, strong SQL and analytical skills, customer empathy, and close collaboration with engineering, operations, analytics, and design teams.