# Applied AI Researcher, Benchmarking

**Company:** [Distyl AI](https://hotfix.jobs/companies/distyl-ai)
**Location:** San Francisco, CA, New York, NY
**Role:** AI Research
**Salary:** $150k – $250k/yr
**Skills:** Ai Benchmarks, Evaluation Frameworks, Llm Evaluation, React, Graph-Of-Thoughts, Ensembling, Python, Data Analysis, Compound Ai Systems, Agentic Systems
**Posted:** 2025-10-16

> Designs and constructs AI benchmarks and evaluation frameworks to measure reasoning, reliability, and real-world impact of intelligent systems. Requires experience with model evaluations, statistical rigor, building with AI models, and strong programming for prototypes.

## Job Description

## Key Responsibilities
- Design evaluation frameworks that capture reasoning depth, interaction quality, reliability, and operational impact.
- Construct benchmarks that reflect real-world complexity to judge new architectures, techniques, and releases.
- Explore new paradigms for evaluating intelligent systems: adversarial robustness testing, longitudinal performance tracking, and human-in-the-loop assessment.
- Investigate how metrics shape model behavior and establish rigorous methodologies for quantifying emergent capability.

## Who You Are (Requirements)
- Experience designing and running evaluations: built or maintained benchmarks, test suites, or experimental frameworks.
- Statistical and analytical rigor: design fair, reproducible experiments and extract signal from noisy results.
- Experience building with models (compound AI systems, agentic collaboration, ensembling, ReAct, graph-of-thoughts, etc.).
- Proven track record of research results (publications, public work).
- Uses AI every day (ChatGPT, Cursor, Perplexity).
- Strong programming and data analysis skills for prototypes and experiments.
- Biases towards showing vs telling.

## Compensation & Benefits
- Base salary: $150K – $250K (depending on experience, location, level).
- Meaningful equity.
- 100% covered medical, dental, vision for employees/dependents.
- 401(k), commuter benefits, in-office lunch.
- Access to state-of-the-art models and AI tools.

## Similar roles

- [Applied AI Researcher, Multi-Agent Systems](https://hotfix.jobs/jobs/6aa59b8a-03f0-4837-bb6e-dd10d59ce3c7) - Distyl AI - San Francisco, CA - $150k – $250k/yr
- [Intelligence Architect](https://hotfix.jobs/jobs/86d5c45a-0c28-428a-9710-53f4bbe379d4) - Basis - New York, NY - $150k – $225k/yr
- [Research Scientist - Frontier Data](https://hotfix.jobs/jobs/838be600-594a-404f-aa4b-5bdb573721c4) - AfterQuery - San Francisco, CA - $150k – $250k/yr
- [Research Scientist - Post Training](https://hotfix.jobs/jobs/69d58c6c-4776-40c5-8cf8-6da6cd0650fe) - AfterQuery - San Francisco, CA - $150k – $250k/yr
- [Machine Learning Researcher - Springtail](https://hotfix.jobs/jobs/7beb0953-43ae-4c56-8de8-5a505d2538c8) - Astera - Emeryville, CA - $150k – $300k/yr

**Apply:** https://hotfix.jobs/jobs/a49ba776-93b9-4c97-a0a0-a771100775e7
**Canonical:** https://hotfix.jobs/jobs/a49ba776-93b9-4c97-a0a0-a771100775e7