# Member of Technical Staff — Pretraining Infra

**Company:** [Nuance Labs](https://hotfix.jobs/companies/nuance-labs)
**Location:** Seattle, WA
**Role:** ML Engineering
**Salary:** $300k – $400k/yr
**Experience:** 7+ years
**Skills:** Pytorch Fsdp, Deepspeed, Megatron, Nccl, Data Parallelism, Tensor Parallelism, Pipeline Parallelism, Sequence Parallelism, Mixed Precision, Activation Checkpointing
**Posted:** 2026-06-05

> Own and scale the distributed training infrastructure for large-scale omni model pretraining across GPU clusters, covering job orchestration, parallelism, GPU communication, data loading, and performance optimization.

## Job Description

## What You’ll Own
- Own the distributed training stack for omni model pretraining, from 0→1 system design to 1→10 scaling across large GPU clusters.
- Build and operate the core training runtime: job orchestration, distributed execution, checkpointing, recovery, monitoring, and debugging for long-running training jobs.
- Optimize large-scale training performance across parallelism strategy, GPU communication, memory usage, data throughput, MFU, step time, and end-to-end training efficiency.
- Build infrastructure for omni training workloads: high-throughput audio/video/text data loading, temporal alignment, variable sequence handling, multimodal synchronization, and memory-efficient training.
- Evolve the platform as model architectures, training recipes, data mixtures, sequence lengths, hardware constraints, and research directions change.

## What We’re Looking For
- Hands-on experience running large-scale distributed training jobs across large GPU clusters; experience at hundreds of GPUs minimum, 1,000+ GPUs a strong plus.
- Deep understanding of distributed training mechanics: data/tensor/pipeline/sequence parallelism, gradient communication, collectives, mixed precision, activation checkpointing, optimizer state, memory pressure, and framework-level tradeoffs.
- Strong understanding of GPU communication and performance debugging: NCCL, all-reduce/all-gather/reduce-scatter, communication-computation overlap, topology, synchronization, stragglers, low MFU, OOMs, checkpoint bottlenecks, and data starvation.
- Practical experience with at least one major large-scale training stack such as Megatron, PyTorch FSDP, DeepSpeed, or equivalent internal infrastructure.
- Understanding of omni or multimodal training challenges, especially audio/video/language data, long temporal context, variable sequence lengths, modality-specific bottlenecks, and high-throughput dataloading.
- Strong software engineering fundamentals, curiosity, and adaptability to new model architectures, training frameworks, hardware constraints, and research ideas.

## Bonus Points
- Prior 0→1 experience building large-scale training infrastructure or deeply modifying core training frameworks, runtimes, checkpointing, or debugging systems.
- Experience training large omni or multimodal models involving audio, video, text, or long-context temporal data.
- Experience with adjacent infrastructure areas such as RL/post-training, data infrastructure, synthetic data generation, evaluation, or serving.
- Publications or substantial open-source contributions in ML systems, distributed systems, HPC, GPU performance, or training infrastructure.

## Compensation
$300,000 – $400,000 base salary, plus meaningful equity.

## Similar roles

- [Senior Member of Research Staff, Optimization](https://hotfix.jobs/jobs/c76416b4-65e4-4c23-84ba-272548563c1e) - The Voleon Group - Berkeley, CA - $300k – $325k/yr
- [Member of Technical Staff — RL Research](https://hotfix.jobs/jobs/c2906806-7087-47d8-a868-41139e89d602) - Nuance Labs - Seattle, WA - $300k – $400k/yr
- [Staff Machine Learning Operations Engineer](https://hotfix.jobs/jobs/f8a44aef-b610-4ca2-a7a5-8b118454fb1d) - Garner Health - New York, NY - $298k – $351k/yr
- [Staff + Senior Software Engineer, Cloud Inference Launch Engineering](https://hotfix.jobs/jobs/18b08a6a-5359-4836-b72e-c641fa718b04) - Anthropic - San Francisco, CA - $320k – $485k/yr
- [Staff Software Engineer, Inference](https://hotfix.jobs/jobs/711c31e7-ec43-46d4-871a-d9074979e024) - Anthropic - San Francisco, CA - $320k – $485k/yr

**Apply:** https://hotfix.jobs/jobs/2fef7d3e-f562-4f7e-a823-66d87234fc89
**Canonical:** https://hotfix.jobs/jobs/2fef7d3e-f562-4f7e-a823-66d87234fc89