Research Engineer, Large-Scale Training

Responsibilities

• Design, implement, and optimize core components of Together's large-scale training infrastructure.
• Integrate new model architectures, validate training correctness and convergence, and optimize performance for production fine-tuning workloads.
• Profile distributed training workloads to identify and eliminate bottlenecks across compute, memory, and communication.
• Design and execute experiments to validate performance hypotheses and benchmark new approaches against state-of-the-art methods.
• Partner closely with Research Scientists to productionize novel training methods and contribute to publications and open-source releases.
• Rapidly enable support for newly released open-source foundation models on the Together platform.
• Build and maintain experimental infrastructure that accelerates research while ensuring production-quality reliability and scalability.

Requirements

• Demonstrated ability to independently take ambiguous performance or infrastructure problems from investigation through deployment.
• Strong programming skills in Python and PyTorch, with an emphasis on writing efficient, maintainable code.
• Hands-on experience training or fine-tuning large neural networks in multi-GPU or multi-node environments.
• Solid understanding of ML systems fundamentals, including GPU architecture, mixed-precision training, and distributed training paradigms such as data, tensor, pipeline, or expert parallelism.
• Strong communication skills and the ability to collaborate effectively with both researchers and engineers.
• Passion for staying current with advances in AI research and applying them to real-world systems.
• Excitement about translating cutting-edge research into production systems that deliver customer impact.

Nice to Have

• Experience writing optimized NVIDIA GPU kernels using CUDA or Triton, or implementing communication collectives with technologies such as NCCL or NVSHMEM.
• Experience with large-scale training frameworks such as FSDP, DeepSpeed, Megatron-LM, or custom distributed training systems.
• Experience optimizing distributed training for compute efficiency, memory efficiency, or scalability.
• Experience running and managing large-scale GPU experiments, including scheduling, monitoring, and fault tolerance.
• Contributions to widely used open-source ML or ML systems projects.
• Experience building or operating ML products or managed services used by external customers.

Compensation

US base salary range for this full-time position is $200,000 - $290,000.