# Software Engineer, Workload Enablement

**Company:** [OpenAI](https://hotfix.jobs/companies/openai)
**Location:** San Francisco, CA, Seattle, WA
**Role:** ML Engineering
**Salary:** $293k – $455k/yr
**Experience:** 5+ years
**Skills:** PyTorch, Nccl, Rccl, Rdma, Kubernetes, Python, C++, CUDA, Hip, Nsight, Perf, Distributed Systems, Ml Systems, Hpc, Llm Training
**Posted:** 2026-03-28

> Software Engineer enabling production AI workloads on new hardware platforms through porting, benchmarking, stress testing, and performance optimization. Requires 5+ years in ML systems, distributed training, PyTorch, and RDMA/NCCL expertise.

## Job Description

## Key Responsibilities
- Port and validate key inference and training workloads on new platforms/SKUs as they arrive; drive correctness, performance, and stability to an internal readiness bar.
- Build a suite of benchmarks and stress tests that capture real E2E behavior of our workloads by exercising all aspects of a system, including CPU, GPU, memory subsystem, frontend, scale-up, and scale-out networking (including WAN traffic, NVlink and RDMA collectives), storage, thermals, and any other relevant parts.
- Deep-dive performance on distributed training/inference:
  - Collective performance and tuning (across NCCL/RCCL and internal libraries)
  - Overlap of compute/communication, kernel-level bottlenecks, memory bandwidth and scheduling effects
- Create repeatable test harnesses that run in CI / lab environments and produce actionable outputs (pass/fail, performance score, regression detection).
- Partner with systems + fleet bring-up engineers to ensure the platform is not only stable and performant, but also operationally usable and scalable (containerization, K8s integration, telemetry hooks, failure triage loops).
- Work cross-functionally with vendors and internal stakeholders by producing clear bug reports, minimal repros, and prioritized issue lists.

## Qualifications
- BS in CS/EE (or equivalent practical experience).
- 5+ years in one or more of: ML systems, performance engineering, distributed systems, or HPC.
- Strong hands-on experience with:
  - PyTorch and modern LLM training/inference stacks
  - Large-scale distributed training concepts (data/model/pipeline parallel, collective comms)
  - Experience with RDMA and debugging/optimizing comms libraries (NCCL or RCCL) and their interaction with hardware/network
- Proficiency in Python plus comfort reading/writing performance-critical code (**C++/CUDA/HIP** is a plus).
- Strong profiling/debugging skills (e.g., Nsight, rocprof, perf, flamegraphs; ability to reason from traces/counters).

## Preferred Skills
- Experience building workload-shaped benchmarks and stress/fault tests that correlate to production behavior (not just synthetic loops or microbenchmarks).
- Familiarity with RDMA networking and transport tuning; understanding of how network topology and congestion impact collectives.
- Experience running and validating workloads in Kubernetes, and bridging “research code” into robust, repeatable infrastructure.
- Hands-on lab experience with early hardware (new NICs, new GPUs/accelerators, early racks).

## Similar roles

- [Research Engineer, Retrieval & Search, Applied Engineering](https://hotfix.jobs/jobs/16d93174-4349-4f36-82d4-a5b06b69b2f5) - OpenAI - San Francisco, CA - $293k – $585k/yr
- [Distributed Training Engineer, Sora](https://hotfix.jobs/jobs/fe5c7041-cc15-401a-8921-5e929f6adf4d) - OpenAI - San Francisco, CA - $293k – $490k/yr
- [Agent Post-Training, API & Power Users](https://hotfix.jobs/jobs/fe6ae516-bafa-4cc6-af78-f20aa90a9eb3) - OpenAI - San Francisco, CA - $295k – $445k/yr
- [Software Engineer, RL Training Infra](https://hotfix.jobs/jobs/5c9ca321-9012-4a35-9c66-fbd05cddb52e) - OpenAI - San Francisco, CA - $295k – $445k/yr
- [Software Engineer, ML Systems & Training Architecture](https://hotfix.jobs/jobs/09ff26d4-974c-471b-9d70-82c93f3c6aab) - OpenAI - San Francisco, CA - $295k – $380k/yr

**Apply:** https://hotfix.jobs/jobs/c4fa7032-a3f8-424e-96b9-93f5b749c237
**Canonical:** https://hotfix.jobs/jobs/c4fa7032-a3f8-424e-96b9-93f5b749c237