Skip to content
OpenAIOpenAISan Francisco, CA

Software Engineer, Model Inference

Optimizes large AI models for high-volume, low-latency production and research environments. Collaborates with researchers and engineers on inference stack performance, requiring 5+ years experience with PyTorch, GPUs, CUDA, and distributed systems.

295k – 555k/yr
On-site5+ YOEML Engineering

About the role

Responsibilities

  • Work alongside machine learning researchers, engineers, and product managers to bring latest technologies into production.
  • Enable advanced research through engineering.
  • Introduce new techniques, tools, and architecture to improve performance, latency, throughput, and efficiency of model inference stack.
  • Build tools for visibility into bottlenecks and instability, then design and implement solutions.
  • Optimize code and Azure VMs to maximize GPU utilization.

Requirements

  • Understanding of modern ML architectures and optimization for inference.
  • Own problems end-to-end and learn as needed.
  • At least 5 years of professional software engineering experience.
  • Familiarity with PyTorch, NVIDIA GPUs, NCCL, CUDA, HPC technologies (InfiniBand, MPI, NVLink).
  • Experience architecting, building, observing, and debugging production distributed systems (bonus for performance-critical).
  • Experience rebuilding/refactoring production systems at scale.
  • Self-directed, humble, eager to help team.

Nice-to-Haves

  • Performance-critical distributed systems experience.

Skills

PyTorchNvidia GpusCUDANcclInfiniBandMpiNvlinkAzureDistributed SystemsMl Inference

Similar roles

ML Engineering jobs
OpenAI

Agent Post-Training, API & Power Users

OpenAISan Francisco, CA

Improve agentic model capabilities, reliability, and product fit for power users and API developers through evals, training data, and post-training interventions.

295k – 445k/yr
On-siteML Engineering
OpenAI

Software Engineer, RL Training Infra

OpenAISan Francisco, CA

Build and maintain infrastructure for large-scale RL training runs of frontier OpenAI models. Debug across training, inference, and distributed systems while supporting research integrations.

295k – 445k/yr
HybridML Engineering
OpenAI

Software Engineer, ML Systems & Training Architecture

OpenAISan Francisco, CA

Hands-on senior software engineer focused on maintaining and improving ML training infrastructure, debugging training systems, and unblocking researchers on the robotics team.

295k – 380k/yr
On-site3+ YOEML Engineering
OpenAI

Software Engineer, Inference - Performance Optimization

OpenAISan Francisco, CA

Models inference performance across application, model, and fleet layers using microbenchmarks to build cost-to-serve estimates. Analyzes workloads end-to-end, enhances bottleneck detection tools, and collaborates on optimizations for latency, throughput, and cost.

295k – 555k/yr
On-siteML Engineering
OpenAI

Machine Learning Engineer, Distributed Data Systems

OpenAISan Francisco, CA

Designs and scales distributed data infrastructure for large-scale multimodal AI training and evaluation. Collaborates with researchers to build reliable, high-performance systems in a fast-paced environment.

295k – 445k/yr
HybridML Engineering