# Senior Software Engineer, AI Infrastructure

**Company:** [Ai2](https://hotfix.jobs/companies/ai2)
**Location:** Seattle, WA
**Role:** DevOps / SRE
**Salary:** $126k – $189k/yr
**Experience:** 8+ years
**Skills:** Go, Python, Linux, Docker, Distributed Systems, Kubernetes, Slurm, Nccl, InfiniBand, SRE
**Posted:** 2026-06-08

> Senior engineer building and operating large-scale HPC infrastructure for AI model training. Owns job scheduling, automation, and performance optimization across GPU clusters.

## Job Description

## Responsibilities
- Independently design and deliver critical systems spanning the full stack—from the Beaker job scheduler to the execution runtime
- Build innovative tooling and software-defined infrastructure to accelerate researcher velocity and automate cluster health management
- Conduct root-cause analysis on complex distributed system failures and implement optimizations for distributed workloads
- Provide input into the roadmap for managing large-scale HPC systems, including deployment of compute, networking, and storage
- Review code/design docs, mentor team members, and drive process improvements
- Communicate and collaborate with internal research staff to share system designs and support implementation

## Requirements
- 8+ years of professional experience developing business-critical software and operating large-scale compute infrastructure
- Proficiency in Go and/or Python
- Bachelor’s degree in related field (advanced degree may substitute for experience)
- Expert-level knowledge of Linux internals and container runtimes (Docker)
- Proven track record designing, debugging, and optimizing high-scale distributed systems and databases
- Exceptional writing skills and ability to drive consensus across researchers and engineers
- Principled approach to engineering and excitement for non-profit research environment

## Nice-to-Haves
- Experience with workload schedulers (Kubernetes, Slurm) and high-performance networking (NCCL, InfiniBand)
- Prior experience training or fine-tuning frontier AI models
- Deep systems administration or SRE background in HPC context
- Contributions to open-source infrastructure or orchestration projects
- Familiarity with on-prem storage systems (WEKA, Ceph)

## Similar roles

- [Senior Software Engineer, Platform](https://hotfix.jobs/jobs/aed20003-ef0c-4e32-b129-e831666b8640) - Ai2 - Seattle, WA - $126k – $189k/yr
- [Senior Software Engineer-Internal Tools](https://hotfix.jobs/jobs/a61df95a-ffa0-4861-abbe-c4cf9cbb4fe3) - Forterra - Arlington, VA - $125k – $140k/yr
- [Site Reliability Engineer](https://hotfix.jobs/jobs/262d56f0-f417-4699-823a-7aa0a613d563) - MongoDB - Boston, MA - $127k – $249k/yr
- [Sr. Software Engineer I](https://hotfix.jobs/jobs/b0e37099-faa1-47c8-bf3c-71af3c72da75) - Invoca - Remote - $127k – $191k/yr
- [Senior Site Reliability Engineer](https://hotfix.jobs/jobs/5d2f6ce8-4fb0-41bf-bb5d-d11904d8ffa9) - MongoDB - Remote - $127k – $249k/yr

**Apply:** https://hotfix.jobs/jobs/bb813d79-dfd2-4649-91b8-417f1d8c33fc
**Canonical:** https://hotfix.jobs/jobs/bb813d79-dfd2-4649-91b8-417f1d8c33fc