Senior Product Quality Analyst - AI Voice

Evaluates and defines quality standards for Spotify’s AI voice models across products, languages, and use cases. The role designs qualitative and quantitative evaluation programs, benchmarks competing technologies, and translates findings into product and research recommendations.

About the job

Responsibilities

Own the quality bar for Spotify’s AI voice models, defining what “good enough” means for different products, audiences, languages, and use cases.
Design evaluation rubrics and build approaches to measure naturalness, tone, pacing, pronunciation, accent, audio quality, and overall experience.
Plan and conduct evaluation campaigns, translating qualitative and quantitative findings into recommendations for Voice Production, Speak, and product teams.
Establish an ongoing evaluation rhythm and determine when and how to reassess model quality as technology and products evolve.
Benchmark Spotify’s AI voices against third-party models and competitors.
Help determine quality trade-offs as Spotify expands into additional languages and regions.
Partner with voice experts, machine learning teams, product partners, and vendors.
Identify meaningful gaps in model performance and use evaluation insights to influence research and technical priorities.

Requirements

Experience in product quality, evaluation, data curation, or a related field involving human judgment to assess model or product performance.
Ability to independently define evaluation criteria and convert subjective qualities into clear, repeatable rubrics.
Experience designing and running qualitative and MOS-style evaluations and synthesizing results into actionable insights.
Understanding of voice or audio quality, including naturalness, pacing, tone, accent, and pronunciation, or the ability to develop this expertise quickly.
Strong judgment and ability to maintain a consistent quality bar while balancing product requirements, markets, and stakeholder perspectives.
Ability to work autonomously and establish useful evaluation processes and cadences.
Clear communication across creative, technical, and product teams.
Interest in generative AI, speech technology, and the evolving AI voice landscape.

Work Arrangement

Based in New York or Los Angeles.
Flexible work arrangement with some in-person meetings and the ability to work from home.

Skills

Generative AI, Speech Technology, Voice Quality Evaluation, Audio Quality, Qualitative Evaluation, Mos Evaluation, Evaluation Rubrics, Data Curation, Machine Learning, Benchmarking