Principal Engineer, AI Inference Reliability

Responsibilities

• Define and drive reliability strategy, including establishing SLOs and aligning engineering teams.
• Design and implement reliability mechanisms for fault detection, graceful degradation, failover, throttling, and recovery across multiple regions and data centers.
• Lead large-scale incident management, including postmortems, root-cause analysis, and prevention loops.
• Architect for reliability and observability by influencing system design for redundancy, durability, and debuggability.
• Develop internal tooling and frameworks for chaos testing, load simulation, and distributed fault injection.
• Collaborate with software, infrastructure, and hardware teams to embed reliability across the inference service.
• Build dashboards and alerts to monitor service health and communicate actionable reliability metrics.
• Mentor engineers and establish best practices for designing, testing, and operating reliable large-scale systems.

Requirements

• Bachelor's or master's degree in computer science or a related field.
• 7+ years of experience in backend, infrastructure, or reliability engineering for large-scale distributed systems.
• Strong programming skills in at least one backend programming language, such as Python, C++, Go, or Rust.
• Deep experience with reliability principles, including SLO/SLI/SLA design, incident response, and postmortem culture.
• Excellent communication and cross-functional leadership skills.

Nice-to-haves

• Experience building large-scale AI infrastructure systems.

Benefits

• Work on a breakthrough AI platform beyond the constraints of GPU-based systems.
• Publish and open-source cutting-edge AI research.
• Work on one of the world's fastest AI supercomputers.
• Enjoy job stability with startup vitality.
• Participate in a non-corporate work culture that respects individual beliefs.