# Software Engineer, Frontier Systems

**Company:** [OpenAI](https://hotfix.jobs/companies/openai)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $250k – $445k/yr
**Experience:** 7+ years
**Skills:** Python, SQL, Promql, Kubernetes, Linux, Pcie, InfiniBand, Roce, GPU, Distributed Systems
**Posted:** 2026-05-11

> Builds infrastructure to monitor, detect, remediate, and verify hardware health across global GPU/CPU clusters at hyperscale. Owns node lifecycle workflows and partners with teams to ensure compute reliability for AI training and inference. Requires 7+ years experience with Python, distributed systems, and operational tooling.

## Job Description

## Responsibilities
- Define and maintain health signals across GPUs, CPUs, networking, and platform infrastructure.
- Build and evolve health checks that detect, remediate, and verify failures at scale.
- Ensure critical health checks execute with minimal latency to maximize workload uptime.
- Investigate hardware failures and system-level issues across large-scale compute environments.
- Own node lifecycle workflows including drain, quarantine, repair, RMA, and return-to-service processes.
- Build automation and tooling that enables global cluster management with minimal manual intervention.
- Partner with workload, reliability, and provider teams to integrate health signals into training and inference systems.

## Requirements
- 7+ years of industry experience in software or infrastructure engineering.
- Strong proficiency with **Python** and shell scripting.
- Experience building large-scale distributed systems or infrastructure platforms.
- Comfort digging into noisy operational data using **SQL**, **PromQL**, or similar tooling.
- Experience building reproducible analyses and operational tooling.
- Strong systems debugging and operational instincts with an ownership mindset.

## Nice-to-Haves
- Experience with low-level hardware systems and Linux tooling (e.g. **PCIe**, **InfiniBand**, **RoCE**, networking, power management, kernel performance tuning, FW/SW debugging).
- Experience operating or debugging large-scale GPU or accelerator clusters.
- Expertise in network operations, observability, or systems telemetry.
- Experience with automated remediation systems or fleet lifecycle management.
- Experience improving reliability, utilization, or workload uptime in distributed compute environments.

## Similar roles

- [Lead Voice Infrastructure Engineer](https://hotfix.jobs/jobs/c286cddc-05a4-4145-afc8-132cb5c2000b) - Stuut - San Francisco, CA - $250k – $290k/yr
- [Senior Software Engineer, Infrastructure](https://hotfix.jobs/jobs/def51027-c615-4950-acba-e46662dd6b9a) - Decagon - San Francisco, CA - $250k – $330k/yr
- [Data Center Controls Network Engineer](https://hotfix.jobs/jobs/ab647475-c575-4a37-ac28-7f4c5ef3e8ec) - OpenAI - San Francisco, CA - $257k – $327k/yr
- [Senior Software Engineer, Infra/Systems](https://hotfix.jobs/jobs/17f71b1c-4c11-4ef1-bdea-b4ff123df84d) - Convex - San Francisco, CA - From $240k/yr
- [Site Reliability Engineer](https://hotfix.jobs/jobs/ab64f10b-7c19-4c49-b827-0e3559c514c9) - Forward Networks - Santa Clara, CA - $230k – $250k/yr

**Apply:** https://hotfix.jobs/jobs/5a732c74-cdbe-4fc9-9ba3-9be015a80279
**Canonical:** https://hotfix.jobs/jobs/5a732c74-cdbe-4fc9-9ba3-9be015a80279