Staff Software Engineer, AI Reliability Engineering

Responsibilities

• Develop Service Level Objectives for large language model serving systems, balancing availability, latency, and development velocity.
• Design and implement monitoring and observability systems across the token path.
• Help design and implement highly available serving infrastructure across multiple regions and cloud providers.
• Lead incident response for critical AI services, driving rapid recovery, thorough incident reviews, and systematic improvements.
• Support the reliability of safeguard model serving.

Requirements

• Strong background in distributed systems, infrastructure, or reliability engineering.
• Reliability-minded software engineering or Site Reliability Engineering experience.
• Ability to troubleshoot unfamiliar systems during incidents and drive resolution.
• Holistic understanding of system composition and integration points.
• Strong communication, collaboration, relationship-building, and ownership skills.

Nice-to-haves

• Experience as an SRE, Production Engineer, or in a similar reliability-focused role on large-scale systems.
• Experience operating large-scale model serving or training infrastructure with more than 1,000 GPUs.
• Experience with ML hardware accelerators, including GPUs, TPUs, or Trainium.
• Knowledge of RDMA and InfiniBand.
• Expertise with AI-specific observability tools and frameworks.
• Experience with chaos engineering and systematic resilience testing.
• Contributions to open-source infrastructure or ML tooling.

Compensation and Benefits

• Annual salary: £325,000–£390,000 GBP.
• Benefits include competitive compensation and benefits, optional equity donation matching, generous vacation and parental leave, flexible working hours, and office collaboration support.
• Bachelor’s degree or an equivalent combination of education, training, and experience is required.
• Field of study should be relevant to the role through coursework, training, or professional experience.