AI Engineer, Evaluation

Key Responsibilities

• Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments
• Define how system quality is measured in each domain, ensuring that evaluation signals reflect real user needs, domain constraints, and business objectives
• Build and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture critical behaviors and edge cases
• Develop and maintain evaluation pipelines—offline and online—that integrate directly into system iteration loops
• Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments
• Work closely with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts

What We Require

• 2+ years of software engineering experience
• Strong Python Engineering Skills: Write clean, maintainable Python and are comfortable building evaluation and experimentation pipelines that run in production environments
• Experience with Evaluation-Driven or Experiment-Driven Development: Experience using structured evaluation or experimentation frameworks to drive system iteration
• Ability to Translate Human Judgment into Code: Work with subject matter experts to elicit high-quality judgments and encode them into test cases, scoring functions, and graders
• Systems-Oriented Mindset: Understand how evaluation interacts with prompts, agents, data, and deployment
• AI-Native Working Style: Use AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging and iteration
• Travel: Travel between 10-50% of the time, depending on the project

What We Offer

• Base salary range: $150K – $250K
• Meaningful equity
• 100% covered medical, dental, and vision for employees and dependents
• 401(k) with additional perks
• Access to state-of-the-art models and modern AI tools
• Offices in San Francisco and New York with hybrid collaboration model (3+ days per week Tuesday–Thursday in-office)