Data Scientist, Codex
Data Scientist measures and accelerates product-market fit for AI developer tools by defining developer productivity metrics, running A/B tests on coding models and UX, and building dashboards. Requires 5+ years quantitative experience, SQL/Python fluency, and experiment design skills.
About the job
Responsibilities
- Embed with the Codex product team to discover opportunities that improve developer outcomes and growth
- Design and interpret A/B tests and staged rollouts of new coding models and product features
- Define and operationalize metrics such as suggestion acceptance, edit distance, compile/test pass rates, task completion, latency, and session productivity
- Build dashboards and analyses that help the team self-serve answers to product questions (by language, framework, repo size, task type)
- Diagnose failure modes and partner with Research on targeted improvements (model quality signals, user feedback, evals)
Requirements
- 5+ years in a quantitative role at a developer-facing or high-growth product
- Fluency in SQL and Python; comfort with experiment design and causal inference
- Experience defining product metrics tied to user value
- Ability to communicate clearly with PM, Eng, and Design—and to influence product direction
Nice-to-Haves
- Strong programming background; ability to prototype, run simulations, and reason about code quality
- Familiarity with IDE/extension telemetry or developer tooling analytics
- Prior experience with NLP/LLMs, code models, or evaluations for generative coding
Skills
SQL, Python, A/B Testing, Causal Inference, Product Metrics, Dashboards, NLP, LLMs, Code Models
Similar jobs
Data Science jobs
OpenAI
San Francisco, CAThe Data Scientist will shape analytics strategy and deliver forecasting, experimentation, optimization, dashboards, models, and decision tools for Real Estate & Workplace operations. The role requires strong applied statistics, causal inference, SQL, Python, stakeholder communication, and comfort working with ambiguous operational data.
OpenAI
San Francisco, CAPeople Data Scientist focused on AI fairness and bias testing for People systems. Designs algorithmic audits, validation studies, and fairness infrastructure across the employee lifecycle. Requires deep expertise in fairness metrics, statistical modeling, and Python/R/SQL.
OpenAI
New York, NYThis senior data scientist will define measurement frameworks for AI-agent security, evaluate controls and security findings, and improve detection and response outcomes. The role requires 5+ years of quantitative experience, strong SQL and Python skills, and experience with cybersecurity or other adversarial-risk domains.
Watershed
San Francisco, CA
Develops scientifically rigorous sustainability methodologies that become product capabilities for corporate climate and ESG data. The role combines climate expertise, GHG accounting, standards interpretation, data reasoning, and hands-on collaboration with engineers and product teams.
Plaid
San Francisco, CASenior Data Scientist supporting Plaid’s Credit product area, translating ambiguous product questions into analytics, metrics, experiments, and strategic insights. The role partners closely with product and engineering teams and requires 5+ years of data science or related analytics experience.