Research Engineer, Audio and Speech
Research Engineer focused on building and deploying real-time audio and speech models for conversational voice agents. The role requires experience with speech or multimodal machine learning, production inference, Python, and PyTorch, with emphasis on taking research from prototype to measurable production impact.
About the job
Responsibilities
- Design and build agent harnesses optimized for streaming speech, turn-taking, interruptions, overlapping speech, and continuous interaction.
- Research and train multimodal and full-duplex models that jointly understand audio, reason, and generate speech.
- Improve speech recognition, voice activity detection, endpointing, and speech generation across diverse speakers, environments, domains, and languages.
- Build evaluations and use production calls to deliver measurable improvements in accuracy, latency, naturalness, and task outcomes.
- Optimize end-to-end inference for responsiveness, throughput, stability, and cost, partnering with platform and infrastructure teams for deployment at scale.
Requirements
- 2+ years of experience in speech, audio machine learning, multimodal machine learning, or production machine learning.
- Experience developing or adapting autoregressive, diffusion, flow-matching, or codec-based speech models.
- Hands-on experience with streaming agent systems, low-latency inference, production model serving, and evaluation on real-world audio.
- Fluency in Python and a modern deep-learning framework such as PyTorch, with strong foundations in machine learning and signal processing.
- Track record of taking research ideas from prototype to reliable, measurable production impact.
Nice-to-haves
- Familiarity with speech-to-speech or full-duplex models.
- Experience with telephony, multilingual speech, noisy-channel robustness, speaker adaptation, or expressive speech generation.
Compensation and Benefits
- $200,000–$400,000 plus equity.
- Medical, dental, and vision benefits for employees and families.
- Life insurance and disability benefits.
- Retirement plan.
- Parental leave.
- Fertility and family-building benefits.
- Monthly wellness and lifestyle stipend.
- Daily office lunches and snacks.
- Flexible vacation policy.
Skills
Python, PyTorch, Speech Recognition, Voice Activity Detection, Speech Generation, Multimodal Machine Learning, Deep Learning, Signal Processing, Low-Latency Inference, Model Serving, Autoregressive Models, Diffusion Models, Flow Matching, Full-Duplex Models, Telephony
Similar jobs
ML Engineering jobs
Decagon
San Francisco, CAResearch Engineer focused on making conversational AI agents safe, reliable, and controllable in production. The role develops evaluations, safeguards, post-training methods, and monitoring systems, requiring 2+ years of AI/ML or safety experience and strong Python and production engineering skills.
AfterQuery
San Francisco, CA
Research Engineer designing post-training infrastructure and running controlled experiments to measure how datasets affect foundation-model behavior. Requires at least 2 years of ML or research engineering experience, strong Python, and hands-on experience with PyTorch, JAX, Ray, Slurm, and LLM post-training.
Stripe
South San Francisco, CA
Build and productionize scalable machine learning models and systems for underwriting and portfolio management. The role requires a bachelor's degree and at least two years of experience shipping ML systems, plus expertise in model development, deployment, data pipelines, and deep learning.
Earnin
Mountain View, CA
Machine learning engineer who trains, evaluates, and productionizes models and LLM-powered applications for financial products. Requires 2+ years of ML systems experience, strong Python and PyTorch skills, production data pipelines, model evaluation, and API development.
SentiLink
United States
Build and deploy production machine-learning models for fraud detection, identity verification, and financial risk products. The role suits new PhD graduates or early-career researchers with strong quantitative foundations, Python experience, and interest in owning the full ML lifecycle.