Software Engineer - Reliability

What You’ll Do

• Architect for Reliability & Scale: Participate in critical re-architecture sessions to redesign our systems for higher efficiency and scale.
• Own Multi-Cloud GPU Clusters: Take end-to-end ownership of our production clusters for training and inference across AWS and OCI, ensuring high availability and peak performance.
• Drive Security & Compliance: Assist in achieving and maintaining security certifications (SOC 2 Type 1 & 2, ISO standards) by implementing robust infrastructure security practices.
• Deep Linux Performance Tuning: Use your mastery of Linux systems to troubleshoot and optimize performance at the OS and kernel level.
• Build Robust Automation: Write high-quality tools and automation in Python, Go, or Bash to manage, monitor, and heal our infrastructure.
• Debug Complex Hardware/Software Failures: Serve as the final escalation point for the most challenging GPU, networking (InfiniBand/RDMA), and system-level issues.

Who You Are

• 8+ years of experience as an SRE, production engineer, or infrastructure engineer in a fast-paced, large-scale environment.
• Deep Linux mastery: hands-on expertise in Linux, containerized systems, and debugging low-level system performance.
• Cloud infrastructure expert: strong experience with providers like AWS or OCI.
• Tenacious troubleshooter for hardware/software intersections.
• Security-minded with knowledge of SOC 2 and ISO compliance.
• Expert in high-performance networking: InfiniBand, RDMA, or RoCE.

What Sets You Apart (Bonus Points)

• Deep expertise with GPU tooling for NVIDIA and AMD GPUs like DCGM or ROCm.
• Experience managing large-scale GPU clusters for AI/ML workloads.
• Familiarity with job management systems based on Kubernetes or orchestration frameworks like Ray.