Senior Automation Specialist, AI Operations

Role Responsibilities

• Own evaluation & datasets: Define evaluation approaches, design gold datasets (GDS), and ensure coverage of real-world scenarios and edge cases
• Build evaluation systems: Develop manual and automated evals, including LLM-as-judge patterns, to measure model quality and performance
• Translate ambiguity into structure: Turn open-ended questions into clear evaluation frameworks and execution plans
• Build automations: Create automations that improve workflows, including dataset creation, evaluation pipelines, and lightweight operational processes
• Measure and iterate: Define and track performance metrics; refine datasets, evaluations, and workflows based on results
• Drive execution forward: Operate with urgency and ownership; identify next steps, unblock progress, and move work forward with minimal oversight
• Collaborate cross-functionally: Partner with other Automation Specialists, engineering, cross-functional stakeholders, and project leads to ensure high-quality, timely project deliverables
• Improve systems: Identify gaps and implement scalable improvements to evaluation and data workflows

Minimum Requirements

• 3+ years of experience designing or working with evaluation frameworks, datasets, or quality measurement systems
• Experience building or managing datasets (labeling, QA, iteration)
• Ability to independently drive tasks from problem definition to execution
• Hands-on experience with AI tools, LLM workflows, or automation platforms
• Experience in defining and tracking model performance
• Basic scripting or data skills (SQL, Python, etc.)

Preferred Requirements

• Experience with LLM-as-judge or model evaluation techniques
• Familiarity with prompt evaluation or benchmarking approaches
• Experience productionizing evaluation workflows with engineering teams

Compensation

Base salary range: $75,000 - $85,000