# Senior Software Engineer, AI Evals

**Company:** [Sentry](https://hotfix.jobs/companies/sentry)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Salary:** $240k – $280k/yr
**Experience:** 5+ years
**Skills:** Python, TypeScript, Machine Learning, Ai Evaluation, LLMs, Datasets, Metrics Pipelines, Evaluation Frameworks, Agentic Systems, Data Infrastructure
**Posted:** 2026-01-28

> Build evaluation infrastructure for AI systems at Sentry, designing datasets, benchmarks, and test harnesses to measure accuracy and reliability of debugging agents. Requires 5+ years experience, Python/TypeScript proficiency, and AI/ML background.

## Job Description

## In this role you will
- Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
- Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
- Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
- Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
- Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring

## You’ll love this job if you
- Care deeply about correctness, rigor, and measurement in AI systems
- Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
- Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
- Thrive in cross-functional environments and enjoy influencing model design through better evaluation

## Qualifications
- Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
- Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
- Comfort writing production-quality code (**Python** and **TypeScript**)
- Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
- Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
- **Bonus**: experience evaluating LLMs, agentic systems, or AI-assisted developer tools

**Base salary range: $240,000 to $280,000 USD.**

## Similar roles

- [Senior AI/ML Engineer](https://hotfix.jobs/jobs/feac02ca-e4ed-4189-9ee5-c17968056cf5) - Sigma - New York, NY - $240k – $270k/yr
- [Senior Machine Learning Engineer - 3D Sensor Simulation](https://hotfix.jobs/jobs/9e1a9534-3a8c-428c-b646-42cd8c9f250c) - Zoox - Foster City, CA - $239k – $333k/yr
- [Senior Machine Learning Engineer - ML Agents and Planning](https://hotfix.jobs/jobs/482ee8a9-f2f2-4c5d-9ea7-f556ded7d518) - Zoox - Foster City, CA - $242k – $290k/yr
- [Senior Machine Learning Engineer - Perception Mapping](https://hotfix.jobs/jobs/49bb2893-5822-4d07-aa97-260a23af0156) - Zoox - Foster City, CA - $242k – $290k/yr
- [Senior Machine Learning Engineer - Perception](https://hotfix.jobs/jobs/82e55013-d9de-49ab-a7e2-37ae4e97c0c8) - Zoox - Foster City, CA - $242k – $290k/yr

**Apply:** https://hotfix.jobs/jobs/58e693b0-5569-4b9f-8a33-52bac16caeab
**Canonical:** https://hotfix.jobs/jobs/58e693b0-5569-4b9f-8a33-52bac16caeab