# Senior Automation Specialist, AI Operations

**Company:** [Fetch](https://hotfix.jobs/companies/fetch)
**Location:** Remote
**Role:** ML Engineering
**Salary:** $75k – $85k/yr
**Experience:** 3+ years
**Skills:** Llm-As-Judge, SQL, Python, AI Tools, Llm Workflows, Automation Platforms, Prompt Evaluation, Evaluation Frameworks, Gold Datasets, Model Evaluation
**Posted:** 2026-04-14

> Owns evaluation frameworks, datasets, and automations to enhance AI system performance in operations. Requires 3+ years experience with AI tools, LLM workflows, scripting (SQL/Python), and driving ambiguous tasks to execution.

## Job Description

## Role Responsibilities
- Own evaluation & datasets: Define evaluation approaches, design gold datasets (GDS), and ensure coverage of real-world scenarios and edge cases
- Build evaluation systems: Develop manual and automated evals, including LLM-as-judge patterns, to measure model quality and performance
- Translate ambiguity into structure: Turn open-ended questions into clear evaluation frameworks and execution plans
- Build automations: Create automations that improve workflows, including dataset creation, evaluation pipelines, and lightweight operational processes
- Measure and iterate: Define and track performance metrics; refine datasets, evaluations, and workflows based on results
- Drive execution forward: Operate with urgency and ownership; identify next steps, unblock progress, and move work forward with minimal oversight
- Collaborate cross-functionally: Partner with other Automation Specialists, engineering, cross-functional stakeholders, and project leads to ensure high-quality, timely project deliverables
- Improve systems: Identify gaps and implement scalable improvements to evaluation and data workflows

## Minimum Requirements
- 3+ years of experience designing or working with evaluation frameworks, datasets, or quality measurement systems
- Experience building or managing datasets (labeling, QA, iteration)
- Ability to independently drive tasks from problem definition to execution
- Hands-on experience with AI tools, LLM workflows, or automation platforms
- Experience in defining and tracking model performance
- Basic scripting or data skills (SQL, Python, etc.)

## Preferred Requirements
- Experience with LLM-as-judge or model evaluation techniques
- Familiarity with prompt evaluation or benchmarking approaches
- Experience productionizing evaluation workflows with engineering teams

## Compensation
Base salary range: $75,000 - $85,000

## Similar roles

- [Automation Lead, AI Operations](https://hotfix.jobs/jobs/7060e745-ee31-4b28-9ebe-8c503703bfd4) - Fetch - Remote - $97k – $114k/yr
- [Senior Software Engineer](https://hotfix.jobs/jobs/4c99a2b5-fd8f-4ff5-8406-933272a4dd44) - Astera - Emeryville, CA - $100k – $400k/yr
- [Sr. Applied AI Engineer](https://hotfix.jobs/jobs/7a6b5cc5-dc8a-485b-b536-5bfc45b42dcd) - Zapier - Remote - $102k – $287k/yr
- [Senior Software Engineer, AI](https://hotfix.jobs/jobs/f5373a65-e352-47d7-87cb-80e609ab7454) - Navan - New York, NY - $113k – $252k/yr
- [Machine Learning Operations (MLOps) Architect - Generative Al Focus](https://hotfix.jobs/jobs/9e70326f-737f-4aac-bc22-bc9b9e1645cb) - Kapitus - Arlington, VA - $118k – $189k/yr

**Apply:** https://hotfix.jobs/jobs/c96197f5-b0bd-4490-9530-ac8c381d9c8e
**Canonical:** https://hotfix.jobs/jobs/c96197f5-b0bd-4490-9530-ac8c381d9c8e