Software Engineer, Model Inference
Optimizes large AI models for high-volume, low-latency production and research environments. Collaborates with researchers and engineers on inference stack performance, requiring 5+ years experience with PyTorch, GPUs, CUDA, and distributed systems.
About the job
Responsibilities
- Work alongside machine learning researchers, engineers, and product managers to bring latest technologies into production.
- Enable advanced research through engineering.
- Introduce new techniques, tools, and architecture to improve performance, latency, throughput, and efficiency of model inference stack.
- Build tools for visibility into bottlenecks and instability, then design and implement solutions.
- Optimize code and Azure VMs to maximize GPU utilization.
Requirements
- Understanding of modern ML architectures and optimization for inference.
- Own problems end-to-end and learn as needed.
- At least 5 years of professional software engineering experience.
- Familiarity with PyTorch, NVIDIA GPUs, NCCL, CUDA, HPC technologies (InfiniBand, MPI, NVLink).
- Experience architecting, building, observing, and debugging production distributed systems (bonus for performance-critical).
- Experience rebuilding/refactoring production systems at scale.
- Self-directed, humble, eager to help team.
Nice-to-Haves
- Performance-critical distributed systems experience.
Skills
PyTorch, Nvidia Gpus, CUDA, Nccl, InfiniBand, Mpi, Nvlink, Azure, Distributed Systems, Ml Inference
Similar jobs
ML Engineering jobs
OpenAI
San Francisco, CA
Build and optimize OpenAI’s inference stack for AWS Trainium across high-performance kernels, compilers, runtimes, and model execution. The role requires systems programming and accelerator experience, with opportunities to solve end-to-end performance problems for frontier-scale AI models.
Anthropic
San Francisco, CABuild and deploy LLM-powered tools, agents, and ecosystem infrastructure with life sciences research institutions. The role requires deep scientific or biomedical research experience, production software development expertise, and the ability to translate partner workflows into scalable AI systems.
OpenAI
San Francisco, CA
Build research infrastructure and tooling that enables AI models to design silicon, including reinforcement learning environments, EDA integrations, evaluations, and experiment workflows. The role requires strong software engineering fundamentals and comfort working across research, tooling, and chip-design systems.
OpenAI
San Francisco, CA
Build and optimize the production LLM inference runtime for frontier models on OpenAI’s custom silicon. The role spans scheduling, distributed execution, memory and KV-cache management, performance tooling, and hardware-software co-design.
Hyperbound
San Francisco, CA
Build and operate machine learning models for sales roleplay, scoring, and coaching products, owning the lifecycle from fine-tuning and evaluation through production and on-device deployment. The role emphasizes open-source models, latency and privacy optimization, and rigorous model testing.