# Staff Software Engineer, GPU Infrastructure Lifecycle Management

**Company:** [Together AI](https://hotfix.jobs/companies/together-ai)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $240k – $280k/yr
**Experience:** 7+ years
**Skills:** Go, Python, Rust, Temporal, cadence, Kubernetes, Kafka, nats, SQS, CUDA, nccl, InfiniBand, redfish, ipmi
**Posted:** 2026-07-16

> Build and own software state machines and control planes that automate the full lifecycle of GPU infrastructure from bare metal provisioning to running AI inference clusters. Requires strong software engineering experience with orchestration, reconciliation loops, and event-driven systems.

## Job Description

## Responsibilities
- Build the provisioning state machine: design and implement the software that models the full lifecycle of a physical host from discovery, inference bring-up to GPU driver/CUDA stack, health validation, and decommission/RMA — as explicit, versioned states and transitions.
- Build the self-service API: design declarative APIs and a control plane so the inference team can request, scale, and tear down inference clusters with one API call — no ticket, no human in the loop.
- Automate self-healing: detect degraded or failed nodes, drain them safely, trigger repair or replacement, and reintroduce healthy capacity into the pool automatically.
- Own reliability of the pipeline: idempotency, retries, rollback, and drift detection so the provisioning system is as dependable as any other production service.
- Partner with the inference/ML platform team: understand the cluster shapes they need — topology, interconnect, scheduling constraints — and encode them as first-class abstractions in the platform.
- Engineer it like software: strong typing, automated tests, code review, versioning, and CI/CD for infrastructure code — this is a product, not a collection of Ansible playbooks.

## Requirements
- Strong software engineering background in Go, Python, Rust, or similar.
- Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent to run long-lived, manifest-driven workflows that survive failures and resume mid-execution.
- Experience building software control planes or orchestration systems that model state and reconcile it over time (e.g., Kubernetes controllers/operators, custom reconciliation loops, workflow engines).
- Experience with event-driven systems — designing and building software around message queues, event streams, or pub/sub (e.g., Kafka, NATS, SQS).
- A product mindset. You’ve built internal platforms or APIs consumed by other engineering teams and care about the developer experience of what you ship.

## Nice-to-Haves
- Exposure to bare-metal provisioning (PXE/iPXE, Redfish/IPMI, BMC) and/or networking fundamentals (VLANs, BGP, fabric design), or GPU/accelerator infrastructure.
- Experience with GPU cluster software stacks (NCCL, CUDA, InfiniBand/RoCE).
- Prior work at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization.
- Systems programming in Rust or Go.

## Compensation
- US base salary range: $240,000 - $280,000 + equity + benefits.

## Similar roles

- [Staff Platform Engineer, Service Infrastructure](https://hotfix.jobs/jobs/2d9f12dc-a433-495b-b10a-54a1b46c9185) - Together AI - San Francisco, CA - $240k – $280k/yr
- [Staff Software Engineer, AI Developer Tooling](https://hotfix.jobs/jobs/27334e3b-c398-4047-a6b2-c579aae27c3f) - Sentry - San Francisco, CA - $240k – $320k/yr
- [Staff Software Engineer, Site Reliability Engineer (SRE)](https://hotfix.jobs/jobs/c1a45fb8-cc27-49ce-9b8c-df80d4d908d8) - Harvey - San Francisco, CA - $238k – $290k/yr
- [Senior Staff Software Engineer, Managed Orchestration](https://hotfix.jobs/jobs/13165fc9-83b3-4e03-af13-4661f44f9bce) - Crusoe - San Francisco, CA - $238k – $288k/yr
- [Staff Software Engineer - Forge](https://hotfix.jobs/jobs/1aa46952-edc5-4cbc-9c3f-b39f8144851e) - Snowflake - Bellevue, WA - $236k – $339k/yr

**Apply:** https://hotfix.jobs/jobs/23bd79e0-99af-4b10-8ed4-ae8b1f90ed31
**Canonical:** https://hotfix.jobs/jobs/23bd79e0-99af-4b10-8ed4-ae8b1f90ed31