# Senior Production Engineer, Operational Excellence

**Company:** [Crusoe](https://hotfix.jobs/companies/crusoe)
**Location:** San Francisco, CA, Sunnyvale, CA
**Role:** DevOps / SRE
**Salary:** $172k – $209k/yr
**Experience:** 5+ years
**Skills:** Prometheus, Grafana, Alertmanager, OpenTelemetry, Kubernetes, Linux, Terraform, Ansible, Python, Go, AWS, GCP, SRE, Gpu Workloads, Hpc
**Posted:** 2026-05-05

> Senior Production Engineer ensures reliability, scalability, and performance of GPU cloud infrastructure powering AI workloads. Drives observability, incident response, automation, and operational improvements in large-scale distributed systems.

## Job Description

## Responsibilities
- Collaborate with cross-functional teams to define and evolve availability metrics for Crusoe’s cloud platform, including establishing, measuring, and improving SLIs and SLOs
- Participate in production incident response, diagnosing and resolving service disruptions while contributing to post-incident reviews and root cause analysis
- Build, operate, and improve observability across Crusoe’s infrastructure using tools such as **Prometheus**, **Grafana**, **Alertmanager**, and **OpenTelemetry**
- Identify reliability risks, performance bottlenecks, and early indicators of potential production issues across distributed systems
- Develop automation and tooling that reduces operational toil, improves recovery times, and enables self-healing infrastructure
- Partner with compute, networking, storage, and platform teams to strengthen service resilience and disaster recovery capabilities
- Contribute to improving operational processes, knowledge sharing, and reliability best practices across the engineering organization

## Requirements
- 5+ years of experience in Production Engineering, **SRE**, or large-scale infrastructure operations
- Experience supporting **GPU workloads**, **HPC environments**, or latency/throughput-sensitive distributed systems
- Strong knowledge of **Linux/Unix** systems, including debugging complex issues across kernel and user space
- Previous experience in Infrastructure roles building or managing compute, storage or networking platforms
- Understanding of modern cloud infrastructure fundamentals including **Kubernetes**, distributed systems, virtualization, and cloud platforms (**AWS/GCP**)
- Familiarity with incident management practices and reliability frameworks (**SRE**, **ITIL**, or similar)
- Experience with monitoring and observability tools such as **Prometheus** and **Grafana**
- Familiarity with infrastructure-as-code and configuration management tools such as **Terraform** or **Ansible**
- Scripting or programming experience with languages such as **Go**, **Python**, **C**, or **C++**
- Strong communication skills and the ability to collaborate across engineering teams
- Ability to remain calm and effective while troubleshooting complex issues in high-impact production environments

## Nice-to-Haves
- Experience working with **Kubernetes** or container orchestration platforms at scale
- Exposure to change management processes, operational readiness reviews, or structured root cause analysis
- Experience designing self-healing systems, automated remediation, or event-driven operational tooling
- Interest in scaling **AI** or **HPC** infrastructure and solving reliability challenges in GPU-heavy environments

## Compensation
- Base salary range: $172,000 – $209,000 + Bonus
- Restricted Stock Units included

## Similar roles

- [Senior Virtualization Validation Engineer](https://hotfix.jobs/jobs/d600ecc9-bdcc-4f15-a5e4-c12e28ae1e79) - Crusoe - San Francisco, CA - $173k – $210k/yr
- [Sr. Site Reliability Engineer](https://hotfix.jobs/jobs/2c837b9b-f230-4e2a-aadb-21d12039b4d2) - Illumio - Sunnyvale, CA - $170k – $196k/yr
- [Software Engineer, Compute Infrastructure](https://hotfix.jobs/jobs/55f94e84-3561-47ca-abbf-933b6bad3ee2) - Render - Remote - $170k – $290k/yr
- [Senior Software Engineer, Developer Productivity Cloud Infrastructure](https://hotfix.jobs/jobs/f213ddfe-a997-4ee7-af19-46182e0c937a) - Skydio - San Mateo, CA - $170k – $240k/yr
- [Senior Software Engineer - Observability and Reliability](https://hotfix.jobs/jobs/1c60bbb9-ebb3-437e-bd25-590308c243d4) - Sigma - San Francisco, CA - $170k – $240k/yr

**Apply:** https://hotfix.jobs/jobs/17dad742-cf70-4122-940b-da85dad20386
**Canonical:** https://hotfix.jobs/jobs/17dad742-cf70-4122-940b-da85dad20386