Engineering Manager, Inference Infrastructure

Responsibilities

• Own the technical roadmap for coordinating the inference fleet, including traffic routing, capacity placement, cache placement, demand responsiveness, and control-plane/inference-engine synchronization.
• Partner with product, inference engine, performance, and capacity teams to identify throughput, latency, utilization, and cost improvements and deliver measurable results.
• Establish quantitative modeling practices for evaluating system changes and expected impact.
• Set technical strategy for control-plane evolution across heterogeneous hardware, multiple cloud providers, and serving surfaces.
• Lead operational practices including on-call rotations, incident response, postmortems, and deployment safety.
• Coordinate across API, inference engine, capacity planning, and cloud deployment teams.
• Develop, retain, and hire high-caliber engineers; coach engineers and shape team structure as scope grows.
• Unblock critical initiatives and synthesize design decisions when needed.

Requirements

• Experience managing engineering teams responsible for critical-path production infrastructure at scale.
• Deep systems expertise in areas such as load balancing, scheduling, cluster orchestration, autoscaling, cache-coherent distributed state, or high-performance networking.
• Experience shipping performance or efficiency improvements in large-scale systems and quantifying latency, cost, and other impacts.
• Experience operating production infrastructure with on-call, incident response, capacity events, and deployment discipline.
• Results-oriented approach and comfort balancing throughput, latency, cost, stability, timelines, and feature velocity.
• Strong cross-functional collaboration and relationship-building skills.
• Curiosity about machine-learning systems and transformer inference.
• Bachelor’s degree or equivalent combination of education, training, and experience in a relevant field.

Nice-to-haves

• 5+ years of engineering management experience.
• Experience with LLM inference serving, including KV caching, continuous batching, request scheduling, or prefill/decode disaggregation.
• Experience with cluster schedulers, autoscalers, load balancers, service meshes, or fleet control planes at scale, including Kubernetes internals or equivalent systems.
• Experience operating workloads across multiple clouds or partner platforms.
• Familiarity with heterogeneous accelerator fleets and hardware-dependent workload placement and rollout sequencing.
• Experience leading teams at supercomputing or hyperscaler infrastructure scale.
• Experience leading multiple teams or groups through rapid growth, hiring, onboarding, and team restructuring.

Compensation

• Annual salary: $405,000–$625,000 USD