Staff+ Software Engineer, Capacity Engineering

Key Responsibilities

• Build the planning and allocation stack for capacity allocation, including cross-region and cross-provider placement, guardrails, queueing, and occupancy KPIs.
• Drive efficiency programs such as stranding and rightsizing, unused capacity recovery, and job-level utilization across training, inference, and eval workloads. Establish per-config baselines and collaborate with system-owning teams to improve utilization.
• Own attribution and forecasting: reconcile billing across multiple providers against telemetry and internal systems, attribute spend to workloads, and convert demand signals and research roadmaps into compute plans.
• Build the data platform: pipelines ingesting occupancy, utilization, and cost data from a diversifying fleet into BigQuery, ensuring completeness, latency SLOs, and gap detection.
• Operate Kubernetes-native systems at scale, including collection agents, workload labeling, and taint/reservation/scheduling behaviors.
• Treat outputs as products: gather requirements, define schema contracts, design for diverse consumers (research engineers to CFO), and maintain on-call and SLOs.

What You Bring

• Strong track record building and operating production systems (hands-on engineering role with DevOps flavor).
• Production-quality Python and SQL (pipeline code in Python; BigQuery SQL including table-valued functions and views; idiomatic, tested, maintainable).
• Deep experience with at least one major cloud provider (AWS, Google Cloud, or Azure) and its operations.
• Experience with observability tooling including Prometheus, PromQL, and Grafana (writing recording rules and building relied-upon monitoring).
• Ability to gather own requirements and work across organizational boundaries in ambiguous environments with limited direction.

Preferred Qualifications

• Experience with capacity planning, resource management, or cost attribution at a hyperscaler or in large-scale ML environments (product engineering and developer experience counts).
• Scheduling/packing efficiency or profiling-driven optimization of large distributed workloads.
• Multi-cloud data ingestion, especially normalizing billing exports, reservation APIs, commitments, and vendor telemetry.
• Total cost of ownership and forecasting, including decomposing infrastructure growth drivers.
• Accelerator infrastructure familiarity (GPU metrics like DCGM, TPU utilization, Trainium metrics, or ML training/inference at hardware level).
• Building internal data products with self-service access, schema contracts, APIs, documentation, and discoverability.
• Storage efficiency, retention, and lifecycle at exabyte scale.

Compensation

Annual Salary: $320000–$485000 USD

Minimum education: Bachelor’s degree or equivalent.