Post-Training Research Engineer

Responsibilities

• Build in-house tooling to support post-training of custom models, including reinforcement learning, supervised finetuning, and in-house research techniques.
• Train a wide spectrum of model architectures with various techniques efficiently and at scale.
• Work across the stack: systems-level concepts like Kubernetes, cgroups, storage systems, and networking topologies; PyTorch distributed tensor computation; GPU kernels.

Requirements

• Deep understanding of modern ML techniques and tools for training transformers.
• Advanced experience in a tensor/array computation library like PyTorch, TensorFlow, Jax, or similar.
• Detailed understanding of transformer training parallelism strategies like data parallelism, sharded data parallelism, tensor parallelism, pipeline parallelism, context parallelism.
• Experience and knowledge to profile and improve the performance of a distributed GPU program in PyTorch or similar.
• Ability to perform roofline analysis on a transformer training setup.
• Willingness to dive into messy problems, work with researchers, derive specifications, and execute.
• Familiarity with HPC and distributed computing platforms like Slurm, Ray, Kubernetes, Dask.
• Familiarity with cluster networking technology like Infiniband, RoCE, GPUDirect.
• Solid fundamentals in operating systems concepts like processes, files, kernel drivers, containerisation, and networking protocols.
• Sense of creativity and willingness to ask difficult questions about approach, assumptions, and tooling choices.

Benefits

• Competitive compensation, including meaningful equity.
• 100% coverage of medical, dental, and vision insurance for employee and dependents.
• Generous PTO policy including company wide Winter Break.
• Paid parental leave.
• Company-facilitated 401(k).
• Exposure to a variety of ML startups.