Senior Software Engineer, AI Evals

In this role you will

• Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems
• Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data
• Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows
• Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria
• Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring

You’ll love this job if you

• Care deeply about correctness, rigor, and measurement in AI systems
• Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics
• Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team
• Thrive in cross-functional environments and enjoy influencing model design through better evaluation

Qualifications

• Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learning, or a related field
• Experience building testing, evaluation, or data infrastructure for complex systems (AI/ML experience strongly preferred)
• Comfort writing production-quality code (Python and TypeScript)
• Experience working with structured and unstructured datasets, labeling workflows, or data quality pipelines
• Familiarity with modern ML systems and evaluation techniques (e.g., offline metrics, online evaluation, regression testing for models or prompts)
• Bonus: experience evaluating LLMs, agentic systems, or AI-assisted developer tools

Base salary range: $240,000 to $280,000 USD.