Software Engineer, Frontier Systems

Responsibilities

• Define and maintain health signals across GPUs, CPUs, networking, and platform infrastructure.
• Build and evolve health checks that detect, remediate, and verify failures at scale.
• Ensure critical health checks execute with minimal latency to maximize workload uptime.
• Investigate hardware failures and system-level issues across large-scale compute environments.
• Own node lifecycle workflows including drain, quarantine, repair, RMA, and return-to-service processes.
• Build automation and tooling that enables global cluster management with minimal manual intervention.
• Partner with workload, reliability, and provider teams to integrate health signals into training and inference systems.

Requirements

• 7+ years of industry experience in software or infrastructure engineering.
• Strong proficiency with Python and shell scripting.
• Experience building large-scale distributed systems or infrastructure platforms.
• Comfort digging into noisy operational data using SQL, PromQL, or similar tooling.
• Experience building reproducible analyses and operational tooling.
• Strong systems debugging and operational instincts with an ownership mindset.

Nice-to-Haves

• Experience with low-level hardware systems and Linux tooling (e.g. PCIe, InfiniBand, RoCE, networking, power management, kernel performance tuning, FW/SW debugging).
• Experience operating or debugging large-scale GPU or accelerator clusters.
• Expertise in network operations, observability, or systems telemetry.
• Experience with automated remediation systems or fleet lifecycle management.
• Experience improving reliability, utilization, or workload uptime in distributed compute environments.