# Machine Learning Scientist - Open Source Lead

**Company:** [Arena](https://hotfix.jobs/companies/arena)
**Location:** San Francisco, CA, California
**Role:** AI Research
**Skills:** PyTorch, JAX, TensorFlow, Python, LLMs, Transformers, RLHF, Dpo, Machine Learning, Statistics
**Posted:** 2025-12-18

> Leads open-source ML research by designing experiments, developing evaluation methodologies, analyzing preference data, and releasing datasets/code to advance AI model transparency. Requires PhD-level expertise in ML/LLMs and hands-on experience with RLHF/DPO fine-tuning.

## Job Description

## Responsibilities

- Design and conduct experiments to evaluate AI model behavior across reasoning, style, robustness, and user preference dimensions
- Develop new metrics, methodologies, and evaluation protocols that go beyond traditional benchmarks
- Analyze large-scale human voting and interaction data to uncover insights into model performance and user preferences
- Communicate results with the broader research community via academic papers, educational content, conference talks
- Collaborate with engineers to implement and scale research findings into production systems
- Prototype and test research ideas rapidly, balancing rigor with iteration speed
- Partner with model providers to shape evaluation questions and support responsible model testing
- Contribute to the scientific integrity and transparency of the LMArena leaderboard and tools

## Requirements

- Hands-on experience training large-scale models, including reward models, preference models, and fine-tuning LLMs with methods like **RLHF**, **DPO**, and contrastive learning
- Strong foundation in ML and statistics, with a track record of designing novel training objectives, evaluation schemes, or statistical frameworks to improve model reliability and alignment
- Fluent in the full experimental stack, from dataset design and large-batch training to rigorous evaluation and ablation, with an eye for what scales to production
- Deeply collaborative mindset, working closely with engineers to productionize research insights and iterating with product teams to align research with user needs
- Comfortable being a visible representative of Arena Intelligence, engaging openly with the research community, and building a strong personal brand to help shape AI research culture
- **PhD** or equivalent research experience in Machine Learning, Natural Language Processing, Statistics, or a related field
- Strong understanding of **LLMs** and modern deep learning architectures (e.g., **Transformers**, diffusion models, reinforcement learning with human feedback)
- Proficiency in **Python** and ML research libraries such as **PyTorch**, **JAX**, or **TensorFlow**
- Demonstrated ability to design and analyze experiments with statistical rigor
- Experience publishing research or working on open-source projects in ML, NLP, or AI evaluation
- Comfortable working with real-world usage data and designing metrics beyond standard benchmarks
- Ability to translate research questions into practical systems and collaborate across engineering and product teams
- Passion for open science, reproducibility, and community-driven research

## Nice-to-Haves

- Skilled at public speaking, writing, and presenting research work to diverse audiences
- Actively participates in conferences, panels, and online forums to foster relationships and thought leadership
- Builds trust through transparent communication and consistent community engagement
- Serves as a go-to contact for external researchers, journalists, and partners

## Similar roles

- [Agent Post-Training, Frontier Evals and Environments Research](https://hotfix.jobs/jobs/ff0a15ea-be79-471d-bed3-8b433b4d1f9c) - OpenAI - San Francisco, CA - $295k – $445k/yr
- [Research Scientist](https://hotfix.jobs/jobs/c1fb9422-45e8-41a0-82e0-bb88fb30a754) - Parallel - California - $150k – $300k/yr
- [Applied Research Scientist / Engineer](https://hotfix.jobs/jobs/adbea7ef-cd6b-435e-826e-ccffeeb8cf8d) - Luma AI - New York, NY - $200k – $450k/yr
- [Senior Research Scientist](https://hotfix.jobs/jobs/4e9d747c-3f11-455f-bf20-0d015ffbdd3a) - Pindrop - Remote - $185k – $215k/yr
- [Senior Research Scientist](https://hotfix.jobs/jobs/534ccd27-e6dd-46ee-a0f4-fb0d8975943c) - Adaption Labs - Remote

**Apply:** https://hotfix.jobs/jobs/80b7e357-1063-4d31-8614-9f800d5b327e
**Canonical:** https://hotfix.jobs/jobs/80b7e357-1063-4d31-8614-9f800d5b327e