# Software Engineer, Agent Evaluation and Quality

**Company:** [Cursor](https://hotfix.jobs/companies/cursor)
**Location:** San Francisco, CA, New York, NY
**Role:** Data Engineering
**Skills:** Ai Evaluation, Datasets, Scorers, Dashboards, Pipelines, Data Analysis, Machine Learning, Python, SQL, Experimentation
**Posted:** 2026-04-13

> Builds measurement, evaluation, and feedback infrastructure for AI agent quality. Designs datasets, pipelines, dashboards, and analysis tools to improve agent reliability, partnering with research and product teams. Requires strong data acumen and software engineering skills.

## Job Description

## Responsibilities
- Design and build best-in-class AI evaluation system: curated datasets, offline replay, scorers / judges, regression alerts, and dashboards.
- Design feedback loops from real usage: collecting, cleaning, and interpreting user signals to inform model and harness changes.
- Develop analysis tooling and workflows for debugging agent behavior: deep dives on failure modes, clustering themes, and surfacing actionable insights.
- Improve reliability and guardrails by making quality measurable and operational: defining “good/bad/degraded” sessions, alerting, and triage primitives.

## Requirements
- Built and operated evaluation or measurement systems, such as AI evals, experimentation, ranking/relevance, or search quality. Can turn ambiguous “quality” questions into concrete metrics, pipelines, and decisions.
- Strong data acumen, and can collaborate effectively with data scientists and researchers.
- Taste and strong opinions on model and agent behaviors. Stay up-to-date on emerging research and industry trends.
- Strong software engineering fundamentals and enjoy shipping production systems.

## Similar roles

- [Data Analytics](https://hotfix.jobs/jobs/b8dd4634-fd75-4c54-8f56-f1c6d2ca6b99) - OnePay - Remote - $130k – $170k/yr
- [Forward Deployed Data Engineer (Integration)](https://hotfix.jobs/jobs/2c2de93d-e128-430e-a6c1-c51b1b338a8a) - Hilbert - San Francisco, CA
- [Software Engineer, Data Infrastructure](https://hotfix.jobs/jobs/a8009f18-5785-47c4-a091-169beaee14fd) - The Voleon Group - Remote - $235k – $300k/yr
- [Client Delivery Specialist](https://hotfix.jobs/jobs/6fa1ca24-291c-4451-862b-4e20168ad45b) - Hinge Health - San Francisco, CA - $80k – $120k/yr
- [Software Engineer](https://hotfix.jobs/jobs/8bf697ae-8d3c-4693-b178-f1f9771d729a) - xAI - Palo Alto, CA - $125k – $400k/yr

**Apply:** https://hotfix.jobs/jobs/703e6ba9-f5a2-4567-9bef-de67303f88db
**Canonical:** https://hotfix.jobs/jobs/703e6ba9-f5a2-4567-9bef-de67303f88db