Research Engineer, Evals
Build benchmarks, datasets, and evaluation systems to measure and improve AI model quality for fraud, identity, and risk judgment tasks. Collaborate across research, engineering, and product to drive rigorous experimentation and iteration in high-stakes environments.
About the job
Responsibilities
- Build proprietary benchmarks and datasets to evaluate models and model systems on fraud, identity, and risk workflows
- Design and run offline and online evals that measure model performance on real customer tasks
- Define quality metrics for judgment systems, including precision, calibration, consistency, abstention, and failure handling
- Study where models and agents break, and turn those failures into better evals, datasets, and training loops
- Build reusable evaluation tools and quality building blocks
- Partner closely with research, engineering, product, and design to improve system quality
- Help create a strong culture of scientific experimentation, clear measurement, and continuous iteration
Requirements
- Care deeply about craftsmanship and have strong opinions about model quality, measurement, and experimental rigor
- Want to work on core model and agent behavior
- Excited by defining what “good” looks like in messy, high-stakes environments
- Think in tight loops: hypothesis, benchmark design, evaluation, failure analysis, iteration
- Strong engineering fundamentals and like building robust systems around ambiguous research problems
- Thrive in environments where success criteria are initially underspecified
- Willing to review outputs, grade edge cases, curate datasets, and refine tasks
Preferred
- Experience training, evaluating, or improving modern ML systems
- Strong programming skills and comfort working in research-heavy codebases
- Experience building benchmarks, datasets, evaluation pipelines, or quality systems
- Familiarity with LLMs, agent systems, retrieval, post-training, or adjacent areas
- Ability to design clean experiments and draw reliable conclusions from noisy results
- Strong engineering judgment and a bias toward building
- Interest in fraud, risk, trust and safety, compliance, or other regulated and adversarial domains
Compensation & Benefits
- Competitive salary and meaningful equity
- Platinum-level medical, dental, and vision insurance
- Unlimited PTO, sick leave, and parental leave
- Up to $100 per month in reimbursement for personal health and wellness expenses
- 401(k) plan
Skills
LLMs, Machine Learning, Benchmarks, Datasets, Evaluation Pipelines, Agent Systems, Retrieval, Post-Training, Python, Experimentation, Fraud Detection, Risk Assessment
Similar jobs
ML Engineering jobs
Hyperbound
San Francisco, CA
Build and operate machine learning models for sales roleplay, scoring, and coaching products, owning the lifecycle from fine-tuning and evaluation through production and on-device deployment. The role emphasizes open-source models, latency and privacy optimization, and rigorous model testing.
Garner Health
New York, NY
Build and deploy algorithmic systems for high-impact healthcare problems, choosing among machine learning, optimization, heuristics, and hybrid approaches. The role requires 4+ years of relevant industry experience, strong applied problem-solving and evaluation skills, and fluency in modern ML tooling.
OpenAI
San Francisco, CA
Build research infrastructure and tooling that enables AI models to design silicon, including reinforcement learning environments, EDA integrations, evaluations, and experiment workflows. The role requires strong software engineering fundamentals and comfort working across research, tooling, and chip-design systems.
OpenAI
San Francisco, CA
Build and optimize the production LLM inference runtime for frontier models on OpenAI’s custom silicon. The role spans scheduling, distributed execution, memory and KV-cache management, performance tooling, and hardware-software co-design.
Cinder
New York, NY
Build and operate production machine-learning systems for content safety, from messy customer data through classification, evaluation, and inference. The role requires 5+ years of ML engineering experience, strong Python and MLOps skills, and sound judgment across classical models and LLMs.