# Research Scientist / Engineer – Training Infrastructure

**Company:** [Luma AI](https://hotfix.jobs/companies/lumalabs-ai)
**Location:** Palo Alto, CA
**Role:** ML Engineering
**Salary:** $188k – $395k/yr
**Skills:** PyTorch, CUDA, Distributed Systems, Fsdp, Tensor Parallel, Pipeline Parallel, Expert Parallel, Nccl, Mpi, Kubernetes
**Posted:** 2025-03-12

> Builds and optimizes distributed training infrastructure for large-scale multimodal AI models across thousands of GPUs. Requires deep expertise in PyTorch, CUDA, parallelization techniques, and GPU clusters.

## Job Description

## Responsibilities
- Design, implement, and optimize efficient distributed training systems for models with thousands of GPUs
- Research and implement advanced parallelization techniques (FSDP, Tensor Parallel, Pipeline Parallel, Expert Parallel)
- Build monitoring, visualization, and debugging tools for large-scale training runs
- Optimize training stability, convergence, and resource utilization across massive clusters

## Experience
- Extensive experience with distributed PyTorch training and parallelisms in foundation model training
- Deep understanding of GPU clusters, networking, and storage systems
- Familiarity with communication libraries (NCCL, MPI) and distributed system optimization

**(Preferred)**
- Strong Linux systems administration and scripting capabilities
- Experience managing training runs across >100 GPUs
- Experience with containerization, orchestration, and cloud infrastructure

## Compensation
Base pay range: $187,500 – $395,000 per year

## Similar roles

- [Software Engineer, Inference](https://hotfix.jobs/jobs/c31a9085-9d3a-4515-b43b-42516d45d8f1) - Luma AI - Palo Alto, CA - $188k – $395k/yr
- [Software Engineer, ML Platform](https://hotfix.jobs/jobs/a9f2a8c2-33a8-4d0d-91c4-60f6a5135e7d) - Luma AI - Palo Alto, CA - $188k – $395k/yr
- [Software Engineer, Machine Learning Platform](https://hotfix.jobs/jobs/8227a48c-d3ba-4a85-afb5-a934accf63b3) - Chime - San Francisco, CA - $187k – $259k/yr
- [Agentic AI Engineer](https://hotfix.jobs/jobs/d74d66cb-3089-4707-b593-9f9e90c11b05) - EigenLayer - Remote - $187k – $253k/yr
- [Software Engineer - Behavior Capabilities](https://hotfix.jobs/jobs/07184ed0-bd8d-46fe-92cd-a18258309fb4) - Zoox - Foster City, CA - $189k – $270k/yr

**Apply:** https://hotfix.jobs/jobs/df564675-5da1-475c-a6b4-e81cc62da4eb
**Canonical:** https://hotfix.jobs/jobs/df564675-5da1-475c-a6b4-e81cc62da4eb