# AI Engineer, Evaluation

**Company:** [Distyl AI](https://hotfix.jobs/companies/distyl-ai)
**Location:** San Francisco, CA, New York, NY
**Role:** ML Engineering
**Salary:** $150k – $250k/yr
**Experience:** 2+ years
**Skills:** Python, Evaluation Frameworks, Experimentation, Llm-Based Graders, Prompt Engineering, Ai Systems, Test Case Development, Regression Testing, Production Pipelines, Model Evaluation
**Posted:** 2026-06-18

> Design and implement evaluation frameworks and pipelines for AI systems using Evaluation-Driven Development. Build Python-based test suites, LLM graders, and measurement systems that guide prompt iteration and production deployment decisions.

## Job Description

## Key Responsibilities
- Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments
- Define how system quality is measured in each domain, ensuring that evaluation signals reflect real user needs, domain constraints, and business objectives
- Build and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture critical behaviors and edge cases
- Develop and maintain evaluation pipelines—offline and online—that integrate directly into system iteration loops
- Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments
- Work closely with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts

## What We Require
- 2+ years of software engineering experience
- Strong Python Engineering Skills: Write clean, maintainable Python and are comfortable building evaluation and experimentation pipelines that run in production environments
- Experience with Evaluation-Driven or Experiment-Driven Development: Experience using structured evaluation or experimentation frameworks to drive system iteration
- Ability to Translate Human Judgment into Code: Work with subject matter experts to elicit high-quality judgments and encode them into test cases, scoring functions, and graders
- Systems-Oriented Mindset: Understand how evaluation interacts with prompts, agents, data, and deployment
- AI-Native Working Style: Use AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging and iteration
- Travel: Travel between 10-50% of the time, depending on the project

## What We Offer
- Base salary range: $150K – $250K
- Meaningful equity
- 100% covered medical, dental, and vision for employees and dependents
- 401(k) with additional perks
- Access to state-of-the-art models and modern AI tools
- Offices in San Francisco and New York with hybrid collaboration model (3+ days per week Tuesday–Thursday in-office)

## Similar roles

- [Quantitative Researcher](https://hotfix.jobs/jobs/9a7f5564-4b13-4cf3-85ac-a96dd78fc052) - Trexquant - New York, NY - $150k – $200k/yr
- [AI Engineer - Data Intelligence](https://hotfix.jobs/jobs/557feeb9-a096-4b62-a589-0c755e4ebf60) - Clarium - Remote - $150k – $180k/yr
- [Forward Deployed Engineer](https://hotfix.jobs/jobs/155cd05a-a4ab-456a-aa78-7ea9fe033920) - Bretton AI - San Francisco, CA - $150k – $200k/yr
- [Early Career Research Engineer](https://hotfix.jobs/jobs/5187da29-bf50-40b5-bb8d-9f99015c7c87) - Parallel - Palo Alto, CA - $150k – $300k/yr
- [Machine Learning Engineer](https://hotfix.jobs/jobs/e7b20b1f-7eb6-4634-9b05-e7e7675f233b) - Reducto - San Francisco, CA - $150k – $300k/yr

**Apply:** https://hotfix.jobs/jobs/f6b322a2-816f-43c9-8487-e15edbac454c
**Canonical:** https://hotfix.jobs/jobs/f6b322a2-816f-43c9-8487-e15edbac454c