# Inference Technical Lead, On-Device Transformers

**Company:** [OpenAI](https://hotfix.jobs/companies/openai)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Salary:** $445k – $445k/yr
**Skills:** Gpus, Npus, Transformers, CUDA, Inference Engines, Kernel Development, Runtime Systems, Ml Pipelines, Accelerators, Kv-Cache
**Posted:** 2026-03-13

> Technical lead evaluating hardware platforms and co-designing transformer models for on-device deployment. Leads team building low-level inference stack, optimizing for latency, memory, and power constraints. Requires deep experience with accelerators, transformers, and performance-critical ML software.

## Job Description

### In this role, you will:
- Evaluate and select silicon platforms (GPUs, NPUs, and specialized accelerators) for on-device and edge deployment of OpenAI models.
- Work closely with research teams to co-design model architectures that meet real-world deployment constraints such as latency, memory, power, and bandwidth.
- Analyze and model system performance, identifying tradeoffs between model design, memory hierarchy, compute throughput, and hardware capabilities.
- Partner with hardware vendors and internal infrastructure teams to bring up new accelerators and ensure efficient execution of transformer workloads.
- Build and lead a team of engineers responsible for implementing the low-level inference stack, including kernel development and runtime systems.
- Run through the necessary walls to take nascent research capabilities and turn them into capabilities we can build on top of.

### You might thrive in this role if you:
- Have experience evaluating or deploying workloads on **GPUs**, **NPUs**, or other specialized accelerators.
- Understand the performance characteristics of **transformer models**, including attention, KV-cache behavior, and memory bandwidth requirements.
- Have designed or optimized high-performance compute systems, such as **inference engines**, distributed runtimes, or hardware-aware ML pipelines.
- Have experience building or leading teams working on low-level performance-critical software such as **CUDA kernels**, compilers, or ML runtimes.
- Have already spent time in the weeds teaching models to speak and perceive.

## Similar roles

- [Software Engineer, Research Infrastructure](https://hotfix.jobs/jobs/aba979c0-7492-4fbb-bbee-841c3aa67496) - Anthropic - San Francisco, CA - $405k – $625k/yr
- [Research Engineer, Chip Design RL](https://hotfix.jobs/jobs/0585d64e-ac0f-4a02-9863-58040a53cde8) - Anthropic - San Francisco, CA - $500k – $850k/yr
- [Research Engineer, Code RL](https://hotfix.jobs/jobs/54cd95f6-3c45-4cb3-a4e1-8e7b6497e0b5) - Anthropic - San Francisco, CA - $500k – $850k/yr
- [Inference Technical Lead, Sora](https://hotfix.jobs/jobs/47f5466e-e1d5-4015-a345-795ca5001ef9) - OpenAI - San Francisco, CA - $380k – $380k/yr
- [Research Engineer, Life Sciences](https://hotfix.jobs/jobs/3da68010-679e-43e2-b57f-64fadcdb6a2f) - Anthropic - San Francisco, CA - $350k – $500k/yr

**Apply:** https://hotfix.jobs/jobs/7deef6b4-f80c-4e0c-a02d-f70c37d0f8ab
**Canonical:** https://hotfix.jobs/jobs/7deef6b4-f80c-4e0c-a02d-f70c37d0f8ab