Member of Technical Staff - Multimodal Understanding
Develops large-scale distributed systems and pipelines for multimodal AI pre-training, post-training, and inference across image, video, audio, and text. Requires expert Python proficiency, experience with JAX/PyTorch/XLA, and scaling multimodal ML systems.
About the job
Responsibilities
- Design, build, and optimize large-scale distributed systems for multimodal pre-training, post-training, inference, data processing, and tokenization at web/petabyte scale.
- Develop high-throughput pipelines for data acquisition, preprocessing, filtering, generation, decoding, loading, crawling, visualization, and management (images, videos, audio + text).
- Advance multimodal capabilities including spatial-temporal compression, cross-modal alignment, world modeling, reasoning, emergent abilities, audio/image/video understanding & generation, real-time video processing, and noisy data handling.
- Drive data quality and studies: curation (human/synthetic), filtering techniques, analysis, and scalable pipelines to support trillion-parameter models.
- Create evaluation frameworks, internal benchmarks, reward models, and metrics that capture real-world usage, failure modes, interactive dynamics, and human-AI synergy.
- Innovate on algorithms, modeling approaches, hardware/software/algorithm co-design, and scaling paradigms for state-of-the-art performance.
- Build research tooling, user-friendly interfaces, prototypes/demos, full-stack applications, and enable rapid iteration based on feedback.
- Work across the stack (pre-training → SFT/RL/post-training) to enable reasoning, tool calling, agentic behaviors, orchestration, and seamless real-time interactions.
Basic Qualifications
- Hands-on experience with multimodal pre-training, post-training, or fine-tuning (vision, audio, video, or cross-modal).
- Expert-level proficiency in Python (core language), with strong experience in at least one of: JAX / PyTorch / XLA.
- Proven track record building or optimizing large-scale distributed ML systems (training/inference optimization, GPU utilization, multi-GPU/TPU setups, hardware co-design).
- Deep experience designing and running data pipelines at scale: curation, filtering, generation, quality studies, especially for noisy/real-world multimodal data.
- Strong fundamentals in evaluation design, benchmarks, reward modeling, or RL techniques (particularly for interactive/agentic behaviors).
- Proactive self-starter who thrives in high-intensity environments and is passionate about pushing multimodal AI frontiers.
- Willingness to own end-to-end initiatives and do whatever it takes to deliver breakthrough user experiences.
Preferred Skills and Experience
- Experience leading major improvements in model capabilities through better data, modeling, algorithms, or scaling.
- Familiarity with state-of-the-art in multimodal LLMs, scaling laws, tokenizers, compression techniques, reasoning, or agentic systems.
- Proficiency in Rust and/or C++ for performance-critical components.
- Hands-on work with large-scale orchestration tools such as Spark, Ray, or Kubernetes.
- Background building full-stack tooling: performant interfaces, real-time research demos/apps, or end-to-end product ownership.
- Passion for end-to-end user experience in interactive, real-time multimodal AI systems.
Compensation and Benefits
- $180,000 - $440,000 USD base salary
- Equity, comprehensive medical, vision, and dental coverage, access to a 401(k) retirement plan, short & long-term disability insurance, life insurance, and various other discounts and perks.
Skills
Python, JAX, PyTorch, Xla, Rust, C++, Spark, Ray, Kubernetes, Rl, Distributed Systems
Similar jobs
ML Engineering jobs
Mirage
New York, NY
Build and deploy agentic systems that power AI-driven creative video workflows. The role requires 5+ years of experience, production ML or agentic pipeline development, context engineering, and expertise in evaluation and agent infrastructure.
Mirage
New York, NY
Build and advance agentic machine-learning systems for multimodal creative tasks, with a focus on video understanding, reasoning, control, and tool use. The role requires strong production ML or agent-pipeline experience and deep knowledge of modern LLM techniques.
Taste Labs
San Francisco, CA
Build evaluation methods, RL environments, agent tooling, and scalable infrastructure that make subjective qualities such as design and taste measurable for frontier AI models. The role requires experience with evaluations, RL environments, ML or post-training, plus strong backend engineering skills.
Mirage
New York, NY
Build and scale generative video and multimodal models, optimizing training and inference for efficiency, throughput, and ultra-low latency. The role requires deep learning systems expertise, strong PyTorch/CUDA experience, and the ability to move research models into production.
Earnin
Mountain View, CA
Build production-grade AI agents, evaluation infrastructure, and developer tooling that make AI-assisted engineering faster, safer, and reusable across teams. The role requires software engineering experience, platform or internal developer-product experience, and hands-on expertise with LLM integration and orchestration.