Member of Technical Staff — Model Optimization and Inference
Early-career engineer optimizing inference for real-time multimodal AI avatars. Focus on KV cache strategies, serving frameworks, quantization, and latency reduction for LLMs and diffusion models.
About the job
What You’ll Do
- Contribute to end-to-end inference optimization across our model stack — LLMs, audio models, and diffusion-based components
- Implement and tune KV cache strategies for long-context conversations, including eviction policies, compression, and memory-efficient attention
- Work with inference serving frameworks (vLLM, SGLang, TensorRT-LLM, etc.) and extend them for our specific workloads
- Profile and benchmark end-to-end latency and throughput; identify and systematically eliminate bottlenecks
- Build internal tooling that makes optimization work faster and more rigorous — profiling viewers, end-to-end inference test harnesses, and other infrastructure that helps the team move quickly
- Accelerate diffusion model inference — consistency models, step distillation, caching strategies, and custom kernel optimizations
- Apply quantization techniques (INT8, INT4, GPTQ, AWQ, and beyond) to reduce memory footprint and increase throughput without meaningfully degrading quality
- Work closely with research and infrastructure to ensure new models ship with optimized serving from day one
What We’re Looking For
- BS, MS, or PhD in CS, ML, or a related field — completed or in the final stretch
- Strong fundamentals in LLM inference or ML systems — KV caching, memory layout, attention kernels, batching, or serving — picked up through coursework, research, internships, or open-source
- Exposure to inference serving frameworks (vLLM, SGLang, TensorRT-LLM, or similar) — even at a research or hobby level
- Strong Python and PyTorch skills; familiarity with CUDA or Triton is a significant plus
- A systematic approach to profiling and optimization — you measure first, then optimize
- Curiosity about diffusion inference, speculative decoding, quantization, or other inference-time acceleration techniques
Bonus Points
- Internship or research experience with LLM inference, ML systems, or model serving
- Contributions to open-source inference frameworks (vLLM, SGLang, TensorRT-LLM, etc.)
- CUDA / Triton kernel work, even at a research or hobby scale
- Publications or research projects in MLSys, model compression, or inference optimization
- Familiarity with multimodal or streaming inference architectures
- Experience with hard latency SLAs in any real-time system
Compensation
$200,000 – $300,000 base salary, plus meaningful equity.
Benefits
- Health: HSA plan with ~$2,000 in annual company contributions
- Time off: 15 days of PTO plus public holidays, and we close the office for a full week at year-end
- Food: Lunch, drinks, and snacks on us every workday
- Commuter benefits
- 401(k): In the works
Skills
Python, PyTorch, vLLM, Sglang, Tensorrt-Llm, CUDA, Triton, Kv Caching, Quantization, Model Optimization
Similar jobs
ML Engineering jobs
Decagon
San Francisco, CAResearch Engineer focused on building and deploying real-time audio and speech models for conversational voice agents. The role requires experience with speech or multimodal machine learning, production inference, Python, and PyTorch, with emphasis on taking research from prototype to measurable production impact.
Decagon
San Francisco, CAResearch Engineer focused on making conversational AI agents safe, reliable, and controllable in production. The role develops evaluations, safeguards, post-training methods, and monitoring systems, requiring 2+ years of AI/ML or safety experience and strong Python and production engineering skills.
AfterQuery
San Francisco, CA
Research Engineer designing post-training infrastructure and running controlled experiments to measure how datasets affect foundation-model behavior. Requires at least 2 years of ML or research engineering experience, strong Python, and hands-on experience with PyTorch, JAX, Ray, Slurm, and LLM post-training.
Stripe
South San Francisco, CA
Build and productionize scalable machine learning models and systems for underwriting and portfolio management. The role requires a bachelor's degree and at least two years of experience shipping ML systems, plus expertise in model development, deployment, data pipelines, and deep learning.
Earnin
Mountain View, CA
Machine learning engineer who trains, evaluates, and productionizes models and LLM-powered applications for financial products. Requires 2+ years of ML systems experience, strong Python and PyTorch skills, production data pipelines, model evaluation, and API development.