Staff Software Engineer, Kubernetes Platform
Senior-level engineer to own and scale Anthropic's massive Kubernetes control plane and scheduler for training frontier AI models across hundreds of thousands of nodes. Requires deep Kubernetes internals experience and 12+ years building production distributed systems.
About the job
Key Responsibilities
- Own, operate, and extend the Kubernetes scheduler for Anthropic's accelerator fleets, including custom scheduling plugins and policies for gang scheduling, topology awareness, and preemption
- Scale the Kubernetes control plane (apiserver, etcd, controller-manager) to support clusters far beyond typical limits, and find the next bottleneck before it finds us
- Design, build, and operate core cluster services such as service discovery that every workload in the fleet depends on
- Build and maintain custom controllers, operators, and CRDs
- Partner with research, training, and inference to understand workload shapes and turn their requirements into platform capabilities
- Collaborate with cloud providers on required features and escalations
- Participate in on-call, lead incident response, and design processes (postmortems, runbooks, SLOs) that help the team avoid repeating failures
Minimum Qualifications
- Significant software engineering experience building and operating production distributed systems
- Proficiency in at least one systems-appropriate language (e.g., Go, Python, Rust, or C++)
- Deep, hands-on Kubernetes experience (well beyond "user of") into scheduler, controllers, apiserver, or operating large multi-tenant clusters
- Demonstrated ability to debug complex issues across the stack, from API behavior down to node and network-level root causes
- A track record of designing for reliability, correctness, and clear failure semantics in systems other engineers depend on
- Strong written and verbal communication; comfort building consensus with internal stakeholders
Preferred Qualifications
- Experience with Kubernetes internals or contributions: kube-scheduler / scheduling framework, apiserver, etcd, client-go, controller-runtime, or similar
- Experience building or operating cluster schedulers or batch systems (e.g., Kueue, Volcano, Slurm, or in-house equivalents)
- Background scaling control planes or coordination systems (etcd, ZooKeeper, Consul, or large DNS/service-mesh deployments)
- Familiarity with ML infrastructure: GPUs, TPUs, or Trainium; gang scheduling; topology-aware placement; collective networking such as NCCL
- Experience with GCP and/or AWS, including GKE/EKS internals and Infrastructure as Code
- Low-level systems experience such as Linux kernel tuning, cgroups, or eBPF
- 12+ years of relevant industry experience, including time leading large, ambiguous infrastructure projects
Compensation & Benefits
- Annual Salary: $405,000—$485,000 USD
- Competitive compensation and benefits, optional equity donation matching, generous vacation and parental leave, flexible working hours
Skills
Kubernetes, Go, Python, Rust, C++, Etcd, GCP, AWS, GKE, EKS, Linux, Ebpf, Nccl, Gpus, Tpus
Similar jobs
DevOps / SRE jobs
Anthropic
San Francisco, CABuild and operate portable infrastructure that enables Claude to run reliably across multiple cloud providers and accelerator platforms. The role requires 8+ years of distributed-systems experience, multi-cloud architecture expertise, production programming, Kubernetes, and Infrastructure as Code proficiency.
Anthropic
London, United Kingdom
Build and operate foundational observability infrastructure spanning telemetry pipelines, profiling, tracing, and diagnostic tooling across large-scale compute clusters. The role requires deep systems-level experience and 10+ years of relevant industry experience.
Anthropic
London, United Kingdom
Leads reliability engineering for critical AI serving systems, spanning SLOs, observability, high availability, and incident response. Requires strong distributed-systems or infrastructure experience, with model-serving, accelerator, networking, and resilience-testing expertise valued.
Anthropic
San Francisco, CAStaff-level site reliability engineer responsible for safely deploying and operating safeguards infrastructure across model releases and cloud platforms. The role emphasizes production change management, high-stakes incident response, and automating manual launch and validation processes.
Headway
San Francisco, CAOwn the cloud platform, deployment architecture, container infrastructure, networking, autoscaling, cost controls, and Python runtime health for a high-scale healthcare technology platform. The role requires 8+ years in infrastructure, platform, or SRE work, deep AWS expertise, Terraform experience, and Staff-level cross-team influence.