Skip to content
Luma AILuma AIPalo Alto, CA

Research Scientist / Engineer – Training Infrastructure

Builds and optimizes distributed training infrastructure for large-scale multimodal AI models across thousands of GPUs. Requires deep expertise in PyTorch, CUDA, parallelization techniques, and GPU clusters.

188k – 395k/yr
HybridML Engineering

About the role

Responsibilities

  • Design, implement, and optimize efficient distributed training systems for models with thousands of GPUs
  • Research and implement advanced parallelization techniques (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel)
  • Build monitoring, visualization, and debugging tools for large-scale training runs
  • Optimize training stability, convergence, and resource utilization across massive clusters

Experience

  • Extensive experience with distributed PyTorch training and parallelisms in foundation model training
  • Deep understanding of GPU clusters, networking, and storage systems
  • Familiarity with communication libraries (NCCL, MPI) and distributed system optimization

(Preferred)

  • Strong Linux systems administration and scripting capabilities
  • Experience managing training runs across >100 GPUs
  • Experience with containerization, orchestration, and cloud infrastructure

Compensation

Base pay range: $187,500 – $395,000 per year

Skills

PyTorchCUDADistributed SystemsFsdpTensor ParallelPipeline ParallelExpert ParallelNcclMpiKubernetes

Similar roles

ML Engineering jobs
Luma AI

Software Engineer, Inference

Luma AIPalo Alto, CA

Develops and optimizes inference engines for multimodal AI models, integrating new architectures, building scheduling systems, and managing large-scale GPU deployments. Requires strong Python, model serving frameworks like PyTorch/vLLM, and Kubernetes expertise.

188k – 395k/yr
HybridML Engineering
Luma AI

Software Engineer, ML Platform

Luma AIPalo Alto, CA

Builds foundational ML platform infrastructure including model serving pipelines, GPU scheduling systems, and CI/CD for large-scale multimodal AI models. Requires 5+ years in distributed systems with expertise in Python, Kubernetes, and AWS.

188k – 395k/yr
Hybrid5+ YOEML Engineering
Chime

Software Engineer, Machine Learning Platform

ChimeSan Francisco, CA

Build and operate Chime's ML platform on AWS, including distributed training systems, feature stores, data pipelines, and CI/CD tooling. Partner with ML teams to improve reliability, observability, and developer experience for production models.

187k – 259k/yr
Hybrid5+ YOEML Engineering
EigenLayer

Agentic AI Engineer

EigenLayerSeattle, WA +1

Builds production-ready agentic AI systems including runtimes, orchestration, reliability, observability, and integrations with LLMs/APIs. Requires strong backend experience, shipped agent/LLM systems, and production reliability expertise.

187k – 253k/yr
RemoteML Engineering
Zoox

Software Engineer - Behavior Capabilities

ZooxFoster City, CA

As a Software Engineer on the Behavior Capabilities team, you will develop and implement algorithmic advancements to expand the robot's driving abilities in complex scenarios, focusing on improving trip progress and vehicle uptime.

189k – 270k/yr
Hybrid4+ YOEML Engineering