ML Model Serving Engineer
Optimizes and extends ML model serving infrastructure for LLMs, speech, and vision models, focusing on high-throughput, low-latency inference using frameworks like VLLM and SGLang. Requires deep PyTorch expertise, systems programming, and performance engineering for reliable production deployment.
About the job
Responsibilities
- Turbocharge our serving layer, consisting of a variety of LLM, speech, and vision models.
- Partner with ML infrastructure and training engineers to build a fast, cost-effective, accurate, and reliable serving layer to power a new consumer product category.
- Modify and extend LLM serving frameworks like VLLM and SGLang to take advantage of the latest techniques in high-performance model serving.
- Work with the training team to identify opportunities to produce faster models without sacrificing quality.
- Use techniques like in-flight batching, caching, and custom kernels to speed up inference.
- Find ways to reduce model initialization times without sacrificing quality.
Required Qualifications
- Expert in some differentiable array computing framework, preferably PyTorch.
- Expert in optimizing machine learning models for serving reliably at high throughput, with low latency.
- Significant systems programming experience (e.g., working on high-performance server systems—comfortable with the internals of VLLM as with a complex PyTorch codebase).
- Significant performance engineering experience (e.g., bottleneck analysis in high-scale server systems or profiling low-level systems code).
- Always up to date on the latest techniques for model serving optimization.
Preferred Qualifications
- Familiarity with high-performance LLM serving (e.g., experience with VLLM, SGlang deployment, and internals).
- Experience with a public cloud platform such as GCP, AWS, or Azure.
- Experience deploying and scaling inference workloads in the cloud using Kubernetes, Ray, etc.
- Track record of leading complex multi-month projects without assistance.
Benefits
- 401(k) max employer match: 3.5% of compensation
- 100% employer-paid health, vision, and dental benefits for you and your dependents
- Unlimited PTO and sick time
- Flexible spending account with employer matching up to $1,650/year (medical FSA)
- Guardian Employee Assistance Program (EAP)
- Competitive stock options
Skills
PyTorch, vLLM, Sglang, Kubernetes, Ray, GCP, AWS, Azure, Llm Serving, Performance Optimization
Similar jobs
ML Engineering jobs
Mirage
New York, NY
Build and deploy agentic systems that power AI-driven creative video workflows. The role requires 5+ years of experience, production ML or agentic pipeline development, context engineering, and expertise in evaluation and agent infrastructure.
Mirage
New York, NY
Build and advance agentic machine-learning systems for multimodal creative tasks, with a focus on video understanding, reasoning, control, and tool use. The role requires strong production ML or agent-pipeline experience and deep knowledge of modern LLM techniques.
Taste Labs
San Francisco, CA
Build evaluation methods, RL environments, agent tooling, and scalable infrastructure that make subjective qualities such as design and taste measurable for frontier AI models. The role requires experience with evaluations, RL environments, ML or post-training, plus strong backend engineering skills.
Mirage
New York, NY
Build and scale generative video and multimodal models, optimizing training and inference for efficiency, throughput, and ultra-low latency. The role requires deep learning systems expertise, strong PyTorch/CUDA experience, and the ability to move research models into production.
Clay
New York, NY
Build and ship production AI agents and the platform infrastructure that makes them reliable, steerable, and measurable. The role requires strong backend fundamentals, production LLM or agent experience, and expertise in evaluations, retrieval, orchestration, or tool-use design.