# Member of Technical Staff, Enterprise Evals Platform

**Company:** [Mercor](https://hotfix.jobs/companies/mercor)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Salary:** $220k – $425k/yr
**Skills:** Agent Engineering, Llm Evaluation, Evaluation Suites, Software Engineering, Agent Runtimes, Agent Harnesses, Benchmarking, Terminal-Bench, Tau-Bench, Apex, Harbor, Reinforcement Learning, Regression Testing, Prompt Optimization, Python
**Posted:** 2026-08-15

> Builds the platform, verifiers, environments, and grading infrastructure used to evaluate enterprise AI agents at scale. The role combines strong software engineering with expertise in agent runtimes, evaluation design, benchmarks, and production failure analysis.

## Job Description

## Responsibilities
- Define golden sets by decomposing real tasks and encoding expert quality standards.
- Build verifiers over agent trajectories and outputs that are calibrated and difficult to game.
- Build evaluation platforms that run offline environments, task suites, and grading infrastructure at scale.
- Run loss analysis over production trajectories and turn failure modes into regression tests.
- Run optimization loops across models, prompts, skills, and harnesses.
- Own rollout gates that determine whether an agent change ships.
- Partner with the Enterprise Platform team and Applied AI engineers embedded with customers.

## Requirements
- Professional, academic, or research experience in agent engineering and evaluation, including agent runtimes, harnesses, trajectories, and failure modes.
- Experience building evaluation suites for LLM or agent systems.
- Familiarity with benchmarks such as terminal-bench, tau-bench, and APEX, including how they are constructed and where they can be gamed.
- Strong judgment about task and rubric design, translating fuzzy notions of quality into measurable criteria.
- Strong software engineering fundamentals.
- Ability to work independently on ambiguous, loosely specified problems.

## Nice-to-haves
- Experience with Harbor environments and reinforcement learning environments.

## Compensation and Benefits
- Salary range: $220,000–$425,000.
- Up to $15,000 relocation bonus.
- $10,000 housing bonus for employees living within 0.5 miles of the office.
- $1,500 monthly meal stipend.
- Generous equity grant vested over four years.
- Free Equinox membership.
- $200 monthly laundry reimbursement.
- $200 monthly personal wellness reimbursement.
- Health, dental, and vision insurance.

## Similar jobs

- [AI/ML Engineer](https://hotfix.jobs/jobs/c4ec2979-ed97-4e7b-9084-fb3f6179add5) - Cinder - New York, NY - $220k – $260k/yr
- [Member of Technical Staff](https://hotfix.jobs/jobs/7f36e0f1-d01f-4eeb-bb83-4a2056256a5e) - Perplexity - San Francisco, CA - $220k – $405k/yr
- [Machine Learning Performance Engineer - Offboard Training & Inference](https://hotfix.jobs/jobs/b2f397d3-cd2d-4eeb-bc02-1dfe0c90a5a7) - Applied Intuition - Sunnyvale, CA - $215k – $285k/yr
- [Machine Learning Engineer](https://hotfix.jobs/jobs/7da1481e-3c05-4f34-92f6-af12f1cc55d0) - Firecrawl - San Francisco, CA - $210k – $240k/yr
- [Applied Scientist III](https://hotfix.jobs/jobs/00c9850b-9666-4032-aa7c-5fd2253c5ef7) - Garner Health - New York, NY - $236k – $260k/yr

**Apply:** https://hotfix.jobs/jobs/b31d2b6c-47ed-427d-a6f7-eb0678b91514
**Canonical:** https://hotfix.jobs/jobs/b31d2b6c-47ed-427d-a6f7-eb0678b91514