Software Engineer, Fleet Hardware Health
Develops automation and monitoring tools to ensure reliability and uptime of OpenAI's large-scale compute fleet, including server provisioning, health monitoring, and performance optimization. Requires strong Linux, networking, and programming skills in Python or Go, with experience in large-scale server environments.
About the job
Responsibilities
- Build and maintain automation systems for provisioning and managing server fleets.
- Develop tools to monitor server health, performance, and lifecycle events.
- Collaborate with clusters, networking, and infrastructure teams.
- Partner with external operators to ensure a high level of quality.
- Identify and fix performance bottlenecks and inefficiencies.
- Continuously improve automation to reduce manual work.
Requirements
- Experience managing large-scale server environments.
- A balance of strengths in building and operationalizing.
- Proficiency in Python, Go, or similar languages.
- Strong Linux, networking, and server hardware knowledge.
- Comfort digging into noisy data with SQL, PromQL, and Pandas or any other tool.
Bonus Skills
- Experience with low level details of hardware components, protocols, and associated Linux tooling (e.g., PCIe, Infiniband, networking, power management, kernel perf tuning).
- Knowledge of hardware management protocols (e.g., IPMI, Redfish).
- High-performance computing (HPC) or distributed systems experience.
- Prior experience developing, managing, or designing hardware.
- Familiarity with monitoring tools (e.g., Prometheus, Grafana).
Skills
Python, Go, Linux, SQL, Promql, pandas, Prometheus, Grafana, Kubernetes, Ipmi, Redfish, Pcie, InfiniBand
Similar jobs
Hardware Engineering jobs
OpenAI
San Francisco, CA
Leads day-to-day operations for a large-scale consumer device test lab, managing device fleets, test rigs, infrastructure, provisioning, maintenance, and logistics. The role partners with engineering and QA teams to maintain reliable environments for validation and release testing.
OpenAI
San Francisco, CA
Designs, commissions, and operates the electrical infrastructure for robotic actuator dynamometers and test cells, including power systems, motor drives, instrumentation, DAQ, and safety circuits. The role requires hands-on experience with electromechanical test equipment, power electronics, measurement integrity, and electrical documentation.
OpenAI
San Francisco, CA
Drives PCBA manufacturing strategy, process development, production readiness, and quality for next-generation AI hardware from prototype through high-volume production. The role requires deep PCBA experience and cross-functional execution with engineering teams, suppliers, and manufacturing partners.
Fluidstack
Austin, TXDesign, deploy, and standardize SCADA architecture and telemetry for real-time monitoring and control of power, cooling, and mechanical systems at gigawatt-scale AI data centers. Requires experience commissioning industrial control systems, integrating protocols, building trusted HMIs/alarms, and ensuring data quality for operations.
OpenAI
San Francisco, CA
Owns signal-integrity and power-integrity architecture, modeling, optimization, and validation for advanced AI ASIC packages. The role requires at least five years of high-speed interconnect and package electrical-design experience, plus expertise with electromagnetic and SI/PI simulation tools.