# Machine Learning Scientist

**Company:** [Arena](https://hotfix.jobs/companies/arena)
**Location:** San Francisco, CA, Berkeley, CA
**Role:** AI Research
**Skills:** PyTorch, JAX, TensorFlow, Python, LLMs, Transformers, RLHF, Dpo, Reinforcement Learning, Statistics
**Posted:** 2025-12-18

> Designs and conducts experiments to evaluate AI models using human preference data, develops new metrics and methodologies, and analyzes large-scale interaction data to advance model reliability and alignment.

## Job Description

## Responsibilities
- Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions
- Develop new metrics, methodologies, and evaluation protocols that go beyond traditional benchmarks
- Analyze large-scale human voting and interaction data to uncover insights into model performance and user preferences
- Collaborate with engineers to implement and scale research findings into production systems
- Prototype and test research ideas rapidly, balancing rigor with iteration speed
- Author internal reports and external publications that contribute to the broader ML research community
- Partner with model providers to shape evaluation questions and support responsible model testing
- Contribute to the scientific integrity and transparency of the Arena Intelligence leaderboard and tools

## Requirements
- Hands-on experience training large-scale models, including reward models, preference models, and fine-tuning LLMs with methods like **RLHF**, **DPO**, and contrastive learning
- Strong foundation in ML and statistics, with a track record of designing novel training objectives, evaluation schemes, or statistical frameworks to improve model reliability and alignment
- Fluent in the full experimental stack, from dataset design and large-batch training to rigorous evaluation and ablation, with an eye for what scales to production
- Deeply collaborative mindset, working closely with engineers to productionize research insights and iterating with product teams to align modeling goals with user needs
- **PhD** or equivalent research experience in Machine Learning, Natural Language Processing, Statistics, or a related field
- Strong understanding of **LLMs** and modern deep learning architectures (e.g., **Transformers**, diffusion models, **reinforcement learning with human feedback**)
- Proficiency in **Python** and ML research libraries such as **PyTorch**, **JAX**, or **TensorFlow**
- Demonstrated ability to design and analyze experiments with statistical rigor
- Experience publishing research or working on open-source projects in ML, NLP, or AI evaluation
- Comfortable working with real-world usage data and designing metrics beyond standard benchmarks
- Ability to translate research questions into practical systems and collaborate across engineering and product teams

## Similar roles

- [ML Scientist](https://hotfix.jobs/jobs/401a3fda-7a0b-4af9-aa9c-134e4ae4b19c) - Sesame - San Francisco, CA - $190k – $320k/yr
- [Research Scientist](https://hotfix.jobs/jobs/34963160-7769-44e3-8e41-872372a811ed) - OpenRouter - Remote
- [Research Engineer, Benchmarks](https://hotfix.jobs/jobs/71636332-204e-4abc-b0b5-c2b5c2ca3726) - hud - San Francisco, CA
- [Research Scientist](https://hotfix.jobs/jobs/5c6acbfc-713c-4c38-bb2c-7a8d9dbc74d1) - Spotify - Remote
- [Research Scientist, Life Sciences](https://hotfix.jobs/jobs/3612b705-0fb1-4f66-a4a1-0b28ba382b5c) - Anthropic - San Francisco, CA - $300k – $320k/yr

**Apply:** https://hotfix.jobs/jobs/7bf601e3-6e8f-4500-9386-87b21d5efe9b
**Canonical:** https://hotfix.jobs/jobs/7bf601e3-6e8f-4500-9386-87b21d5efe9b