Skip to content
CantinaCantinaSan Francisco, CA

Machine Learning Enginer, Core Evaluations

Designs evaluation pipelines, metrics, and user studies for speech generation and recognition models (ASR/TTS). Trains evaluation models, builds dashboards, and collaborates with ML, data, and product teams to improve performance on large-scale systems.

Salary not listed
Remote3+ YOEML Engineering

About the role

Responsibilities

  • Design model evaluation pipelines for models in development and production.
  • Design user studies for subjective model evaluations.
  • Convert requirements into measurable metrics.
  • Design and develop automated evaluation dashboards to monitor model performances and compare results.
  • Train new models to capture new and different evaluation metrics.
  • Communicate with model team to design better models based on evaluation results.
  • Communicate with data team to decide data needed to improve model performance.
  • Communicate with product manager to ensure product requirements are correctly measured.
  • Help grow the evaluation team as founding member and lead it in the future.

Requirements

  • Strong experience designing metrics that capture model performance.
  • Strong experience designing user studies on Mechanical Turk or similar platforms.
  • Strong experience with model training and fine-tuning for evaluation.
  • Strong statistical knowledge to compare evaluation results and make decisions.
  • Very strong engineering and programming skills.
  • Experience training ASR and TTS models.
  • Experience at ML teams working on large-scale problems (>3B models with >1m hours of data).

Skills

AsrTtsModel EvaluationUser StudiesMechanical TurkModel TrainingFine-TuningStatisticsPythonDashboards

Similar roles

ML Engineering jobs
Cerebras Systems

CoDesign & NextGen Performance Engineer

Cerebras SystemsSunnyvale, CA

Characterize, analyze, and optimize performance of state-of-the-art AI models on Cerebras' wafer-scale hardware. Build performance models, optimize kernels and compilers, debug runtime behavior, and develop visualization tools to influence next-gen AI architecture.

Salary not listed
On-site3+ YOEML Engineering
OpenAI

Research Engineer, Privacy

OpenAISan Francisco, CA

Research Engineer on OpenAI's Privacy team designing and prototyping privacy-preserving ML algorithms like differential privacy and federated learning at scale. Requires hands-on PETs experience, fluency in PyTorch/JAX, and a track record implementing or publishing novel privacy work.

380k – 445k/yr
HybridML Engineering
Console

Research Engineer

ConsoleSan Francisco, CA

Research Engineer building self-improving AI agent systems at Console. Develop eval/optimization loops, fine-tune specialist models, and improve agent reasoning over enterprise context using production data to drive measurable gains in quality, latency, and reliability.

200k – 350k/yr
On-siteML Engineering
Notion

Software Engineer, AI Platform

NotionSan Francisco, CA +1

Build and scale the shared AI platform foundations at Notion, enabling fast and safe shipping of AI products. Requires experience with LLM/ML platforms, strong ownership, and comfort across backend, infrastructure, and product code.

180k – 201k/yr
Hybrid5+ YOEML Engineering
Liftoff

Machine Learning Engineer

LiftoffCalifornia

Machine Learning Engineer building statistical models, optimization systems, and experiments for mobile ad tech economics on the Revenue Engine team. Requires PhD in CS/ML/Economics and industry experience applying ML or economics at scale.

215k – 275k/yr
RemoteML Engineering