# Member of Technical Staff

**Company:** [Black Forest Labs](https://hotfix.jobs/companies/black-forest-labs)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Salary:** $180k – $290k/yr
**Experience:** 7+ years
**Skills:** PyTorch, CUDA, Triton, Cutlass, Fsdp, Nccl, Nsight Systems, Nsight Compute, Torch Profiler, Gpu Kernels, Low-Precision Training, Distributed Training, Profiling, Quantization
**Posted:** 2026-06-29

> Research Engineer embedded in large-scale multimodal model training at Black Forest Labs (creators of FLUX). Own performance, stability, and low-precision optimizations for production training runs; profile, debug distributed systems, implement GPU kernels, and partner with researchers to ship frontier generative models.

## Job Description

## What You’ll Work On

- Improve the performance, reliability, and numerical stability of production training runs for large multimodal generative models
- Profile full training steps across model code, attention, kernels, data loading, encoders, communication, optimizer steps, checkpointing, and memory pressure
- Implement and validate GPU-level optimizations: fused kernels, attention paths, low-precision matmuls, quantization kernels, CUDA/Triton/CuTe/CUTLASS experiments, and no-compile alternatives where they make sense
- Push lower-precision training forward, including FP8 / MXFP8 / FP4-style paths, weight and activation quantization, accumulation choices, convergence risk, and quality tradeoffs against baseline training runs
- Work with researchers to translate architecture changes into efficient training implementations, and help distinguish real model-quality progress from changes that only look good in a microbenchmark
- Debug distributed training failures: NaNs, loss spikes, silent numerical drift, memory leaks, stragglers, bad nodes, NCCL issues, and throughput cliffs
- Build benchmarking and profiling harnesses that make performance claims trustworthy across hardware, shapes, sequence lengths, and training configurations
- Help the training team move quickly when an urgent bottleneck appears, while turning repeated failures into better abstractions and tools

## What We’re Looking For

- Experience working deeply on large-scale training systems, ideally as part of a training group working closely with researchers
- Strong PyTorch fluency, including comfort reading and modifying low-level training code rather than only using high-level APIs
- Experience with distributed training concepts such as FSDP, tensor/model/context/sequence parallelism, activation checkpointing, NCCL, and overlapping compute and communication
- Hands-on experience improving training throughput, memory footprint, or stability in real training runs
- Experience profiling GPU workloads with tools like Nsight Systems, Nsight Compute, torch profiler, trace viewers, or custom telemetry
- Practical GPU performance judgment: you may use modern coding agents and tools as much as you want, but you need the understanding to verify correctness, numerical behavior, and performance, and to own the result
- Understanding of low-precision training and quantization tradeoffs: FP8, MXFP8, FP4/NVFP4-style formats, scaling, accumulation, numerical validation, and convergence risk
- Good research judgment: you can partner with researchers on ablations, understand what the measurements do and do not prove, and keep optimization work tied to model-quality outcomes
- Comfortable operating in ambiguity: sometimes the task is a clean implementation, sometimes it is a production fire, and sometimes it is figuring out which of three plausible explanations is actually true

## Nice-to-Haves

- Have supported or co-owned training for a frontier foundation model that shipped or reached a major release
- Have written or substantially improved forward/backward GPU kernels, or have shown you can make progress on kernel-level work with strong measurement and validation discipline
- Have worked on attention performance, variable sequence length training, non-standard attention patterns
- Have experience on Hopper or Blackwell-class GPUs
- Have worked on low-precision training
- Have experience with diffusion, flow matching, DiT, and multimodal generative model training; if your deepest background is autoregressive or LLM training systems, you are excited to learn the diffusion and multimodal modeling stack quickly
- Can move naturally between profiler traces, kernel code, distributed systems failures, and research discussions

## Similar roles

- [Member of Technical Staff](https://hotfix.jobs/jobs/dd7c8bc4-741a-45d5-91a8-e6b1c4ad67e0) - xAI - Palo Alto, CA - $180k – $440k/yr
- [Member of Technical Staff, Applied Research](https://hotfix.jobs/jobs/d19460df-bf56-4f87-aa95-5e7bdf40798f) - LlamaIndex - San Francisco, CA - $180k – $250k/yr
- [Member of Technical Staff, Inference](https://hotfix.jobs/jobs/70280e8d-9e99-4be1-a488-b13edc2d4080) - xAI - Palo Alto, CA - $180k – $440k/yr
- [Member of Technical Staff - X Search](https://hotfix.jobs/jobs/2480c779-d772-4053-b432-64b3e48d4cd3) - xAI - Palo Alto, CA - $180k – $440k/yr
- [Member of Technical Staff - Post-Training and RL](https://hotfix.jobs/jobs/ed7e3533-4b73-4672-b792-579d9520ae65) - xAI - Palo Alto, CA - $180k – $600k/yr

**Apply:** https://hotfix.jobs/jobs/544a5b36-3a56-45fa-89fe-0873120ece22
**Canonical:** https://hotfix.jobs/jobs/544a5b36-3a56-45fa-89fe-0873120ece22