Skip to content

Research Engineer, Evals

Build benchmarks, datasets, and evaluation systems to measure and improve AI model quality for fraud, identity, and risk judgment tasks. Collaborate across research, engineering, and product to drive rigorous experimentation and iteration in high-stakes environments.

250k – 400kSan Francisco, CAML EngineeringOnsite

About the role

Responsibilities

  • Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows
  • Design and run offline and online evals that measure model performance on real customer tasks
  • Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling
  • Study where models and agents break, and turn those failures into better evals, datasets, and training loops
  • Build reusable evaluation tools and quality building blocks
  • Partner closely with research, engineering, product, and design to improve system quality
  • Help create a strong culture of scientific experimentation, clear measurement, and continuous iteration

Requirements

  • Care deeply about craftsmanship and have strong opinions about model quality, measurement, and experimental rigor
  • Want to work on core model and agent behavior
  • Excited by defining what “good” looks like in messy, high-stakes environments
  • Think in tight loops: hypothesis, benchmark design, evaluation, failure analysis, iteration
  • Strong engineering fundamentals and like building robust systems around ambiguous research problems
  • Thrive in environments where success criteria are initially underspecified
  • Willing to review outputs, grade edge cases, curate datasets, and refine tasks

Preferred

  • Experience training, evaluating, or improving modern ML systems
  • Strong programming skills and comfort working in research-heavy codebases
  • Experience building benchmarks, datasets, evaluation pipelines, or quality systems
  • Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas
  • Ability to design clean experiments and draw reliable conclusions from noisy results
  • Strong engineering judgment and a bias toward building
  • Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains

Compensation & Benefits

  • Competitive salary and meaningful equity
  • Platinum-level medical, dental, and vision insurance
  • Unlimited PTO, sick leave, and parental leave
  • Up to $100 per month in reimbursement for personal health and wellness expenses
  • 401(k) plan

Skills

LLMsMachine LearningBenchmarksDatasetsEvaluation PipelinesAgent SystemsRetrievalPost-TrainingPythonExperimentationFraud DetectionRisk Assessment

Similar roles

ML Engineering jobs

Researcher, Alignment Oversight

Designs and runs experiments to improve oversight of increasingly capable AI models, including model training, evaluation, and deployment of practical systems. Analyzes failures and develops techniques to train more aligned models using oversight signals.

250k – 445kSan Francisco, CAML EngineeringHybridLLMsPyTorch

Research Scientist / Engineer — Multimodal Agent

Builds and trains large-scale multimodal agentic models involving reasoning, planning, coding, and tool calling. Requires strong ML foundations, PyTorch expertise, and experience with distributed training on massive datasets.

250k – 450kPalo Alto, CAML EngineeringHybridVlmLLMs

Research Engineer, Judgment Systems

Research Engineer designs evaluations, studies model failures, and builds research loops to improve AI agents for high-stakes fraud detection and judgment tasks. Requires ML training experience, experimental rigor, and strong engineering skills in adversarial environments.

250k – 400kSan Francisco, CAML EngineeringOn-siteLLMsPython

AI Engineer

Builds and deploys AI primitives and agents to automate workflows and enhance user experiences in investment management platform. Requires AI agent experience, distributed systems knowledge, and product-minded engineering across tech stacks.

250k – 325kNew York, NYML EngineeringOn-siteGoPython

Lead AI Engineer

Leads development of proprietary AI reasoning model TRAM for interpreting global trade law, building data pipelines, fine-tuning LLMs, and evaluation frameworks for high-speed, accurate compliance determinations. Requires AI product experience, especially RAG systems and model fine-tuning.

250k – 280kSan Francisco, CAML EngineeringOn-siteRAGLLMs