Skip to content
Distyl AIDistyl AISan Francisco, CA

AI Engineer, Evaluation

Design and implement evaluation frameworks and pipelines for AI systems using Evaluation-Driven Development. Build Python-based test suites, LLM graders, and measurement systems that guide prompt iteration and production deployment decisions.

150k – 250k
Hybrid2+ YOEML Engineering

About the role

Key Responsibilities

  • Design and implement evaluation frameworks that enable Evaluation-Driven Development for AI systems deployed in customer environments
  • Define how system quality is measured in each domain, ensuring that evaluation signals reflect real user needs, domain constraints, and business objectives
  • Build and maintain golden test cases and regression suites in Python, using both human-authored and AI-assisted test generation to capture critical behaviors and edge cases
  • Develop and maintain evaluation pipelines—offline and online—that integrate directly into system iteration loops
  • Define, calibrate, and operate LLM-based graders, aligning automated judgments with expert human assessments
  • Work closely with Forward Deployed AI Engineers, Architects, Product Engineers, AI Strategists, and domain experts

What We Require

  • 2+ years of software engineering experience
  • Strong Python Engineering Skills: Write clean, maintainable Python and are comfortable building evaluation and experimentation pipelines that run in production environments
  • Experience with Evaluation-Driven or Experiment-Driven Development: Experience using structured evaluation or experimentation frameworks to drive system iteration
  • Ability to Translate Human Judgment into Code: Work with subject matter experts to elicit high-quality judgments and encode them into test cases, scoring functions, and graders
  • Systems-Oriented Mindset: Understand how evaluation interacts with prompts, agents, data, and deployment
  • AI-Native Working Style: Use AI tools to generate tests, analyze failures, explore edge cases, and accelerate debugging and iteration
  • Travel: Travel between 10-50% of the time, depending on the project

What We Offer

  • Base salary range: $150K – $250K
  • Meaningful equity
  • 100% covered medical, dental, and vision for employees and dependents
  • 401(k) with additional perks
  • Access to state-of-the-art models and modern AI tools
  • Offices in San Francisco and New York with hybrid collaboration model (3+ days per week Tuesday–Thursday in-office)

Skills

PythonEvaluation FrameworksExperimentationLlm-Based GradersPrompt EngineeringAi SystemsTest Case DevelopmentRegression TestingProduction PipelinesModel Evaluation

Similar roles

ML Engineering jobs
Trexquant

Quantitative Researcher

TrexquantNew York, NY

Quantitative Researcher developing cutting-edge ML and statistical models to identify predictive signals in global financial markets. Requires PhD in a STEM field, strong Python skills, and interest in ML/AI; no finance experience needed. Matched to Alpha Research, Data Science, or Strategy Research teams.

150k – 200k
On-siteEntry levelML Engineering
Clarium

AI Engineer - Data Intelligence

ClariumUnited States

As an AI Engineer, you will build and maintain Clarium's master data enrichment pipeline, focusing on classification and entity resolution workflows. You will write production Python and SQL code, analyze complex datasets, and proactively audit data for quality issues.

150k – 180k
RemoteEntry levelML Engineering
Bretton AI

Forward Deployed Engineer

Bretton AISan Francisco, CA

Forward Deployed Engineer deploys AI agents for financial services customers, handles onsite integrations in regulated environments, builds connectors and frameworks, and translates customer needs into product insights. Requires 2+ years software engineering with Python, APIs, data pipelines, cloud, and AI experience.

150k – 200k
On-site2+ YOEML Engineering
Parallel

Early Career Research Engineer

ParallelPalo Alto, CA +1

Designs and trains embedding and retrieval models for AI agents to access web data at hyperscale, balancing research innovation with production efficiency for sub-second latency and fresh indexes.

150k – 300k
On-siteEntry levelML Engineering
Reducto

Machine Learning Engineer

ReductoSan Francisco, CA

Develops and deploys ML models for parsing unstructured enterprise data like PDFs, focusing on training vision models, experimenting with LLMs, building data pipelines, and integrating into products. Requires 2+ years in production ML, Python proficiency, and computer vision expertise.

150k – 300k
On-site2+ YOEML Engineering