# Tech Lead Manager, Inference

**Company:** [Luma AI](https://hotfix.jobs/companies/lumalabs-ai)
**Location:** Redwood City, CA
**Role:** ML Engineering
**Experience:** 8+ years
**Skills:** Distributed Systems, ML Infrastructure, Model Serving, Inference Platforms, vLLM, Sglang, Tensorrt-Llm, Python, PyTorch, Kubernetes, Continuous Batching, Kv Cache Management, Quantization, Speculative Decoding, Parallelism Strategies
**Posted:** 2026-07-07

> Tech Lead Manager to hands-on lead the inference platform team at Luma AI. Own the full serving stack for multimodal models across thousands of GPUs, spending 50%+ time as IC on architecture, optimization, and debugging while growing the team and setting technical direction.

## Job Description

## What You’ll Do
- Spend at least half your time hands-on in the serving stack: architect and build core platform components, own the hardest design decisions, and debug the toughest production incidents yourself.
- Lead, grow, and develop the inference engineering team: own hiring, coaching, and career growth, and build the team’s operational culture — on-call, incident response, capacity planning, and postmortems.
- Set the technical roadmap for the serving platform: model serving engines, request routing and scheduling, autoscaling, caching, observability, and deployment pipelines.
- Own the platform’s SLOs and economics: latency and availability targets, GPU utilization, and cost per generation across every model we serve.
- Partner closely with research to ship new model architectures into production on day zero, and to integrate serving into online RL and evaluation loops.
- Manage and optimize inference workloads across heterogeneous fleets — multiple clusters, clouds, and GPU vendors — including capacity planning and hardware bring-up.
- Build sophisticated scheduling and queueing systems that optimally leverage expensive GPU resources against live traffic patterns, cluster availability, and user priority.

## Representative Projects
- Design intelligent routing and scheduling that optimizes request distribution across thousands of GPUs in multiple regions and clouds.
- Stand up disaggregated prefill/decode serving with tiered KV-cache reuse across GPU memory, DRAM, NVMe, and network storage.
- Autoscale and hot-swap models across the fleet to dynamically match GPU supply with live demand across production, research, and experimental workloads.
- Take a new multimodal architecture from research checkpoint to a production deployment serving millions of users, including quantization, speculative decoding, and precision/regression validation across hardware platforms.
- Build end-to-end tracing that follows any inference request through its full lifetime — queueing, routing, prefill, decode, and delivery.
- Integrate the inference stack into an online reinforcement learning pipeline where serving throughput directly gates training progress.

## Background
- 8+ years of engineering experience in large-scale distributed systems or ML infrastructure, with several years building and operating model-serving or inference platforms in production.
- Experience running inference platforms at scale — you have operated fleets on the order of thousands of GPUs across multiple clusters or clouds, and you understand what breaks at that scale.
- Technical leadership experience, including managing or leading engineers through periods of rapid growth — and a genuine desire to keep at least half your time in hands-on technical work rather than move into pure management.
- Deep, practical expertise in LLM and foundation-model serving engines (vLLM, SGLang, TensorRT-LLM, or equivalent) — ideally you’ve modified engine internals, debugged edge cases under load, and contributed improvements back.
- Strong command of the serving-performance toolkit: continuous batching, KV-cache management, quantization, speculative decoding, and parallelism strategies (TP/EP/pipeline).
- Strong Python and PyTorch; experience operating services on Kubernetes at scale.
- Experience with queues, scheduling, traffic control, and fleet management at scale.

## Bonus Points
- Experience serving diffusion, video, or other multimodal generative models (not just text), and with FFmpeg/multimedia processing.
- Experience with modern networking stacks — RDMA (RoCE, InfiniBand), NVLink — including KV-cache transfer and multi-node serving topologies.
- Experience across heterogeneous accelerator platforms (NVIDIA, AMD, TPU, Trainium) and the porting/validation work that comes with them.
- Contributions to open-source serving infrastructure (vLLM, SGLang, Ray, Kubernetes ecosystem).
- Systems-language depth (Rust, C++, CUDA/HIP) for kernel- and runtime-level optimization.

## Similar roles

- [Senior Machine Learning Engineer, Agent Oversight](https://hotfix.jobs/jobs/1f504a69-044f-47ab-872b-120a1706cd88) - Scale AI - San Francisco, CA - $216k – $270k/yr
- [Senior Software Engineer, Agent Oversight](https://hotfix.jobs/jobs/808ce719-6b16-4b6e-8887-8c8f8a780b75) - Scale AI - San Francisco, CA - $216k – $270k/yr
- [Senior Software Engineer, AI Infrastructure](https://hotfix.jobs/jobs/dd1c3420-6c0a-47ed-a1a7-de401bdf506c) - Ambient.ai - Redwood City, CA - $168k – $205k/yr
- [Senior Software Engineer](https://hotfix.jobs/jobs/423c60aa-ca4c-400d-babe-a50099433cbb) - SpotOn - Chicago, IL - $160k – $190k/yr
- [Senior AI Engineer, Enterprise](https://hotfix.jobs/jobs/87a30fd1-3fe6-47f3-a0a1-da444f3c00f4) - Snowflake - Menlo Park, CA - $156k – $224k/yr

**Apply:** https://hotfix.jobs/jobs/eee42cb9-d342-4be6-9c9b-b1bdc4c639d7
**Canonical:** https://hotfix.jobs/jobs/eee42cb9-d342-4be6-9c9b-b1bdc4c639d7