Skip to content
Distyl AIDistyl AISan Francisco, CA

Applied AI Researcher, Benchmarking

Designs and constructs AI benchmarks and evaluation frameworks to measure reasoning, reliability, and real-world impact of intelligent systems. Requires experience with model evaluations, statistical rigor, building with AI models, and strong programming for prototypes.

150k – 250k/yr
HybridAI Research

About the role

Key Responsibilities

  • Design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact.
  • Construct benchmarks that reflect real-world complexity to judge new architectures, techniques, and releases.
  • Explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment.
  • Investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability.

Who You Are (Requirements)

  • Experience designing and running evaluations: built or maintained benchmarks, test suites, or experimental frameworks.
  • Statistical and analytical rigor: design fair, reproducible experiments and extract signal from noisy results.
  • Experience building with models (compound AI systems, agentic collaboration, ensembling, ReAct, graph-of-thoughts, etc.).
  • Proven track record of research results (publications, public work).
  • Uses AI every day (ChatGPT, Cursor, Perplexity).
  • Strong programming and data analysis skills for prototypes and experiments.
  • Biases towards showing vs telling.

Compensation & Benefits

  • Base salary: $150K – $250K (depending on experience, location, level).
  • Meaningful equity.
  • 100% covered medical, dental, vision for employees/dependents.
  • 401(k), commuter benefits, in-office lunch.
  • Access to state-of-the-art models and AI tools.

Skills

Ai BenchmarksEvaluation FrameworksLlm EvaluationReactGraph-Of-ThoughtsEnsemblingPythonData AnalysisCompound Ai SystemsAgentic Systems

Similar roles

AI Research jobs
Distyl AI

Applied AI Researcher, Multi-Agent Systems

Distyl AISan Francisco, CA +1

Develop multi-agent AI architectures for enterprise coordination and collaborative reasoning. Requires research experience in MARL/GNNs, strong prototyping skills, and daily AI tool usage.

150k – 250k/yr
HybridAI Research
Basis

Intelligence Architect

BasisNew York, NY

Designs and authors context, procedures, skills, and system prompts for AI agents performing autonomous accounting tasks. Ensures consistency, monitors performance, and shapes future capabilities through precise language and systems thinking.

150k – 225k/yr
On-siteAI Research
AfterQuery

Research Scientist - Frontier Data

AfterQuerySan Francisco, CA

Designs datasets and evaluation frameworks for frontier AI models, collaborating with top labs to expose failure modes, refine RLHF/RLVR pipelines, and measure data impact on capabilities. Requires strong quantitative skills, familiarity with LLM training, and research experience up to master's level.

150k – 250k/yr
On-siteAI Research
AfterQuery

Research Scientist - Post Training

AfterQuerySan Francisco, CA

Design and run SFT/RL experiments to measure dataset impact on LLM performance, capabilities, and alignment. Collaborate with labs to provide evidence of improvements; requires strong LLM training knowledge and fast experimentation, ideally undergrad/master's research background.

150k – 250k/yr
On-siteAI Research
Astera

Machine Learning Researcher - Springtail

AsteraEmeryville, CA

Conducts fundamental research on data-efficient ML architectures, including bootstrapped program synthesis and self-synthesizing learning systems. Requires Master's in ML/math, PyTorch fluency, and research experience.

150k – 300k/yr
HybridAI Research