# AI Engineer, Quality (Evals)

**Company:** [Fieldguide](https://hotfix.jobs/companies/fieldguide)
**Location:** Remote
**Role:** ML Engineering
**Salary:** $170k – $220k/yr
**Experience:** 3+ years
**Skills:** Python, TypeScript, React, Postgres, Langsmith, LangGraph, LLMs, RAG, Vector Databases, Observability, Ml Engineering, Prompt Engineering, Agentic Systems
**Posted:** 2026-04-20

> Owns evaluation infrastructure for AI agents in audit workflows, building unified platforms, automated pipelines, observability, and feedback loops to ensure enterprise-scale reliability. Requires experience with LLMs, TypeScript/Python, and production AI systems.

## Job Description

## What You'll Own

### Measurable AI Agents
- Design and build a unified evaluation platform that serves as the single source of truth for all of our agentic systems and audit workflows
- Build observability systems that surface agent behavior, trace execution, and failure modes in production, and feedback loops that turn production failures into first-class evaluation cases
- Own the evaluation infrastructure stack including integration with **LangSmith** and **LangGraph**
- Translate customer problems into concrete agent behaviors and workflows
- Integrate and orchestrate **LLMs**, tools, retrieval systems, and logic into cohesive, reliable agent experiences

### Rapid Model Evaluation
- Build automated pipelines that evaluate new models against all critical workflows within hours of release
- Design evaluation harnesses for our most complex Agentic systems and workflows
- Implement comparison frameworks that measure effectiveness, consistency, latency, and cost across model versions
- Design guardrails and monitoring systems that catch quality regressions before they reach customers

### AI-native engineering execution
- Use AI as core leverage in how you design, build, test, and iterate
- Prototype quickly to resolve uncertainty, then harden systems for enterprise-grade reliability
- Build evaluations, feedback mechanisms, and guardrails so agents improve over time
- Work with SMEs and ML Engineers to create evaluation datasets by curating production traces
- Design prompts, retrieval pipelines, and agent orchestration systems that perform reliably at scale

### Ownership of Quality and Large Product Areas
- Define and document evaluation standards, best practices, and processes for the engineering organization
- Advocate for evaluation-driven development and make it easy for the team to write and run evals
- Partner with product and ML engineers to integrate evaluation requirements into agent development from day one
- Take full ownership of large product areas rather than executing on narrow tasks

## Who You Are
You are an engineer who believes that evaluations are foundational to building reliable AI systems.

## Experience
- Multiple years of experience shipping production software in complex, real-world systems
- Experience with **TypeScript**, **React**, **Python**, and **Postgres**
- Built and deployed **LLM**-powered features serving production traffic
- Implemented evaluation frameworks for model outputs and agent behaviors
- Designed observability or tracing infrastructure for **AI/ML** systems
- Worked with vector databases, embedding models, and **RAG** architectures
- Experience with evaluation platforms (**LangSmith**, **Langfuse**, or similar)

## Benefits
- Competitive compensation packages with meaningful ownership
- Flexible PTO
- 401k
- Wellness benefits, including a bundle of free therapy sessions
- Technology & Work from Home reimbursement
- Flexible work schedules

## Similar roles

- [Machine Learning Engineer](https://hotfix.jobs/jobs/a3359f08-ea84-4500-b684-678e1b789991) - Airbnb - San Francisco, CA - $170k – $180k/yr
- [AI Platform Engineer](https://hotfix.jobs/jobs/e5a08a02-c346-4778-8598-c6f2b3ff1189) - Notable - San Mateo, CA - $170k – $205k/yr
- [Autonomy Engineer - ML & DL Infrastructure](https://hotfix.jobs/jobs/64e727de-a200-4cdd-81c1-9ed94150308b) - Skydio - San Mateo, CA - $170k – $278k/yr
- [Research Engineer](https://hotfix.jobs/jobs/b91e0912-d75f-418f-97a6-1b0a51108f23) - Forus - New York, NY - $170k – $300k/yr
- [Applied Machine Learning Engineer](https://hotfix.jobs/jobs/03161b4c-017e-42c8-b206-8f437cdbebfa) - Fireworks AI - New York, NY - $170k – $240k/yr

**Apply:** https://hotfix.jobs/jobs/54a01151-d1f8-4597-be26-69ae94139fb2
**Canonical:** https://hotfix.jobs/jobs/54a01151-d1f8-4597-be26-69ae94139fb2