Skip to content
hudhudSan Francisco, CA

Research Engineer, Benchmarks

Build high-quality, domain-specific benchmarks and infrastructure to rigorously evaluate frontier AI agents on realistic workflows. Requires strong Python/Docker/Linux skills, experience with evals or benchmarks, and a deep understanding of what makes a benchmark reliable and useful.

Salary not listed
On-siteAI Research

About the role

Responsibilities

  • Own the design, implementation, and quality of HUD’s internal agent benchmarks
  • Work with subject-matter experts to define tasks and create domain-specific benchmarks that evaluate agents on realistic workflows
  • Build infrastructure to reliably run models and agents against benchmark tasks
  • Develop metrics and analyses to understand benchmark difficulty, reliability, and failure modes
  • Validate whether benchmark performance correlates with real-world evals, customer needs, and lab expectations
  • Write clear documentation and benchmark reports that make results legible and credible to technical audiences

Requirements

  • Proficiency in Python, Docker, and Linux environments
  • Published papers or written technical blogs on relevant topics such as public benchmarks and their limitations, model failure modes, etc.
  • Strong understanding of what a “good benchmark” means and what makes one realistic, reliable, and useful
  • Experience working on environments and evals
  • Curiosity and ability to truly understand how workflows in various domains work

Nice-to-Haves

  • Detail-oriented and able to spot subtle inconsistencies or edge cases in tasks
  • Able to reason from first principles about task design, scoring, and failure modes
  • Thrive in unstructured problem spaces
  • Early-stage startup experience with ability to work independently in fast-paced environments
  • Strong communication skills for remote collaboration across time zones

Skills

PythonDockerLinuxBenchmarksEvalsAgent Evaluation

Similar roles

AI Research jobs
Sesame

ML Scientist

SesameSan Francisco, CA +2

Research-oriented Machine Learning Scientist developing multimodal ML models (NLP, Speech, Computer Vision) for lifelike voice agents. Requires published papers in large-scale deep learning and familiarity with SOTA AI.

190k – 320k
On-siteAI Research
OpenRouter

Research Scientist

OpenRouterUnited States

Conduct original research on LLM evaluation, routing optimization, and model behavior using billions of real-world generations. Design novel benchmarks, run large-scale empirical studies, and develop statistical foundations for intelligent routing. Requires MS/PhD, publication track record, deep stats/ML expertise, and Python/SQL skills.

Salary not listed
RemoteAI Research
Spotify

Research Scientist

SpotifyNew York, NY

Research Scientist advancing generative audio models (diffusion/flow matching) for music, focusing on vocal synthesis, post-training alignment (DPO/RLHF), or audio editing. Requires PhD, top publications, and PyTorch expertise to turn research into artist-first Spotify products.

Salary not listed
RemoteAI Research
Anthropic

Research Scientist, Life Sciences

AnthropicSan Francisco, CA

Founding Research Scientist on Anthropic's Life Sciences team, designing and executing wet-lab experiments in molecular biology and biochemistry. Partners with computational biologists and uses Claude AI for hypothesis generation, experimental planning, and rapid iteration to drive AI-accelerated biological discoveries.

300k – 320k
HybridAI Research
Anthropic

Researcher, Education Labs

AnthropicSan Francisco, CA +1

As the first dedicated education researcher at Anthropic's Education Labs, design and run mixed-methods studies on AI skill development, build/validate learning measures and instruments, and translate insights into product, curriculum, and model improvements. Requires research background in learning sciences or related fields, strong mixed-methods expertise, Python/LLM technical skills, and comfort with ambiguity.

300k – 405k
HybridAI Research