# Software Engineer - Reliability

**Company:** [Luma AI](https://hotfix.jobs/companies/lumalabs-ai)
**Location:** Remote
**Role:** DevOps / SRE
**Salary:** $170k – $360k/yr
**Experience:** 8+ years
**Skills:** Linux, AWS, Oci, Kubernetes, Python, Go, Bash, InfiniBand, Rdma, Nvidia, Amd, Dcgm, Rocm, SOC 2
**Posted:** 2025-12-06

> Builds, maintains, and scales multi-cloud GPU infrastructure for AI training/inference, focusing on reliability, performance tuning, automation, and security in a fast-paced startup. Requires 8+ years SRE experience with deep Linux, cloud, and high-performance networking expertise.

## Job Description

## What You’ll Do
- Architect for Reliability & Scale: Participate in critical re-architecture sessions to redesign our systems for higher efficiency and scale.
- Own Multi-Cloud GPU Clusters: Take end-to-end ownership of our production clusters for training and inference across AWS and OCI, ensuring high availability and peak performance.
- Drive Security & Compliance: Assist in achieving and maintaining security certifications (SOC 2 Type 1 & 2, ISO standards) by implementing robust infrastructure security practices.
- Deep Linux Performance Tuning: Use your mastery of Linux systems to troubleshoot and optimize performance at the OS and kernel level.
- Build Robust Automation: Write high-quality tools and automation in **Python**, **Go**, or **Bash** to manage, monitor, and heal our infrastructure.
- Debug Complex Hardware/Software Failures: Serve as the final escalation point for the most challenging GPU, networking (**InfiniBand**/**RDMA**), and system-level issues.

## Who You Are
- 8+ years of experience as an **SRE**, production engineer, or infrastructure engineer in a fast-paced, large-scale environment.
- Deep **Linux** mastery: hands-on expertise in **Linux**, containerized systems, and debugging low-level system performance.
- **Cloud** infrastructure expert: strong experience with providers like **AWS** or **OCI**.
- Tenacious troubleshooter for hardware/software intersections.
- Security-minded with knowledge of **SOC 2** and **ISO** compliance.
- Expert in high-performance networking: **InfiniBand**, **RDMA**, or **RoCE**.

## What Sets You Apart (Bonus Points)
- Deep expertise with **GPU** tooling for **NVIDIA** and **AMD** GPUs like **DCGM** or **ROCm**.
- Experience managing large-scale **GPU** clusters for **AI/ML** workloads.
- Familiarity with job management systems based on **Kubernetes** or orchestration frameworks like **Ray**.

## Similar roles

- [Sr. Site Reliability Engineer](https://hotfix.jobs/jobs/2c837b9b-f230-4e2a-aadb-21d12039b4d2) - Illumio - Sunnyvale, CA - $170k – $196k/yr
- [Software Engineer, Compute Infrastructure](https://hotfix.jobs/jobs/55f94e84-3561-47ca-abbf-933b6bad3ee2) - Render - Remote - $170k – $290k/yr
- [Senior Software Engineer, Developer Productivity Cloud Infrastructure](https://hotfix.jobs/jobs/f213ddfe-a997-4ee7-af19-46182e0c937a) - Skydio - San Mateo, CA - $170k – $240k/yr
- [Senior Software Engineer - Observability and Reliability](https://hotfix.jobs/jobs/1c60bbb9-ebb3-437e-bd25-590308c243d4) - Sigma - San Francisco, CA - $170k – $240k/yr
- [Senior Software Engineer, Infrastructure](https://hotfix.jobs/jobs/4648241c-cc1b-4176-9930-563bc53a2b4c) - Skydio - San Mateo, CA - $170k – $258k/yr

**Apply:** https://hotfix.jobs/jobs/14144da0-fdc9-44df-8359-7cb565cb4c26
**Canonical:** https://hotfix.jobs/jobs/14144da0-fdc9-44df-8359-7cb565cb4c26