Principal Site Reliability Engineer

Key Responsibilities

• Define and implement a robust strategy for delivering and running software reliably and at scale across multiple datacenters and cloud-based solutions.
• Architect self-service platforms and internal tooling that let product teams, external customers, and cluster operators safely trigger and observe critical workflows with minimal handoffs.
• Define and evolve reliability practices for inference workloads, including SLOs and SLIs for latency, throughput, and accuracy stability; error budgets; blameless postmortems; chaos testing; and capacity forecasting across multi-datacenter and on-prem environments.
• Mentor senior SREs, support critical incident escalations, and use production pain points to prioritize the highest-leverage automation work.
• Measure and drive impact through clear metrics, including toil reduction, deployment velocity, SLO compliance, MTTR, and adoption of self-service workflows.

Required Experience & Skills

• 15+ years in SRE, infrastructure engineering, or platform engineering, with a record of setting technical direction and delivering reliability improvements at large scale in FAANG, hyperscaler, frontier AI, or similarly demanding production environments.
• Deep experience with large-scale compute fleets, internal control planes, schedulers, orchestration systems, capacity management, and reliability automation.
• Experience defining and driving cross-team architecture for production control planes, capacity orchestration, fleet management, or self-service infrastructure platforms with clear operational ownership.
• Strong judgment in converging fragmented workflows, tools, and teams into coherent architectures that improve reliability, efficiency, and operational leverage.
• Ability to lead complex, ambiguous technical programs end to end; influence senior cross-functional stakeholders; mentor senior engineers; and communicate technical strategy clearly.
• Hands-on experience with production observability, incident response, and SLO-based reliability management across metrics, logs, traces, alerting, dashboards, and operational review loops.

Nice-to-Haves

• Experience with Bazel or other large-scale build systems in production.
• Background in AI/ML inference systems, including model serving runtimes, disaggregated inference, GPU orchestration, latency and accuracy SLOs, or drift monitoring.
• Prior work on predictive autoscaling, chaos engineering, or cost-aware capacity management for compute-intensive workloads.