Skip to content
VarianceVarianceSan Francisco, CA

Research Engineer, Judgment Systems

Research Engineer designs evaluations, studies model failures, and builds research loops to improve AI agents for high-stakes fraud detection and judgment tasks. Requires ML training experience, experimental rigor, and strong engineering skills in adversarial environments.

250k – 400k/yr
On-siteML Engineering

About the role

Responsibilities

  • Train, fine-tune, and improve models for fraud, scams, abuse, and other high-stakes judgment workflows
  • Own research threads focused on improving agent capability, reliability, and decision quality
  • Build proprietary benchmarks, datasets, and evals that reflect real customer workflows, regulatory constraints, and real failure modes
  • Design and run experiments across post-training, retrieval, tool use, planning, memory, and long-horizon agent behavior
  • Study where models break, why they break, and how to make them more robust
  • Prototype new training strategies, agent architectures, and evaluation methods, then turn the best ideas into production systems
  • Work closely with founders and engineering to translate research advances into deployed product capabilities
  • Push the boundary of what AI agents can do in regulated industries

Requirements

  • Care deeply about protecting people from fraud, scams, and abuse
  • Have strong opinions about model quality, evaluation, and experimental rigor
  • Want to work on core model and agent behavior
  • Excited to train, fine-tune, and improve models for hard real-world judgment tasks
  • Think in tight research loops: hypothesis, experiment, evaluation, failure analysis, iteration
  • Thrive in ambiguous, fast-moving environments where the path is not obvious and the feedback loop is short
  • Motivated by the challenge of making AI systems work in adversarial, regulated, and high-consequence settings
  • Want to help define what trustworthy AI means in real-world use cases

Preferred Background

  • Experience training, fine-tuning, or evaluating modern ML systems
  • Strong programming skills and comfort working in research-heavy codebases
  • Familiarity with LLMs, agent systems, post-training, reinforcement learning, retrieval, or adjacent areas
  • Ability to design clean experiments and draw reliable conclusions from noisy results
  • Strong engineering judgment and a bias toward building
  • Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains

Compensation & Benefits

  • Competitive salary and meaningful equity ($250,000 - $400,000)
  • Platinum-level medical, dental, and vision insurance
  • Unlimited PTO, sick leave, and parental leave
  • Up to $100 per month in reimbursement for personal health and wellness expenses
  • 401(k) plan

Skills

Machine LearningLLMsFine-TuningReinforcement LearningRetrieval Augmented GenerationAgent SystemsPost-TrainingEvaluationBenchmarkingPython

Similar roles

ML Engineering jobs
OpenAI

Researcher, Alignment Oversight

OpenAISan Francisco, CA

Designs and runs experiments to improve oversight of increasingly capable AI models, including model training, evaluation, and deployment of practical systems. Analyzes failures and develops techniques to train more aligned models using oversight signals.

250k – 445k/yr
HybridML Engineering
Luma AI

Research Scientist / Engineer — Multimodal Agent

Luma AIPalo Alto, CA

Builds and trains large-scale multimodal agentic models involving reasoning, planning, coding, and tool calling. Requires strong ML foundations, PyTorch expertise, and experience with distributed training on massive datasets.

250k – 450k/yr
HybridML Engineering
Variance

Research Engineer, Evals

VarianceSan Francisco, CA

Build benchmarks, datasets, and evaluation systems to measure and improve AI model quality for fraud, identity, and risk judgment tasks. Collaborate across research, engineering, and product to drive rigorous experimentation and iteration in high-stakes environments.

250k – 400k/yr
On-siteML Engineering
Moment

AI Engineer

MomentNew York, NY

Builds and deploys AI primitives and agents to automate workflows and enhance user experiences in investment management platform. Requires AI agent experience, distributed systems knowledge, and product-minded engineering across tech stacks.

250k – 325k/yr
On-siteML Engineering
Zoox

AI Inference Engineer - Model Optimization & Deployment

ZooxFoster City, CA +2

Optimizes and deploys large-scale AI models (LLMs, VLMs) for real-time inference on power-constrained vehicle hardware. Requires expertise in quantization, TensorRT compilation, custom CUDA kernels, and production C++/Python for edge devices.

242k – 290k/yr
HybridML Engineering