# Staff Software Engineer, AI Runtime

**Company:** [Databricks](https://hotfix.jobs/companies/databricks)
**Location:** Mountain View, CA, San Francisco, CA
**Role:** ML Engineering
**Salary:** $190k – $265k/yr
**Experience:** 10+ years
**Skills:** PyTorch, Fsdp, Deepspeed, Megatron, Gpu Scheduling, Distributed Training, Checkpointing, Nvlink, InfiniBand, Roce, Collective Communication, Multi-Node Orchestration, High-Performance Computing, Ml Systems
**Posted:** 2026-06-08

> Staff Software Engineer building and scaling Databricks' managed large-scale GPU training platform (AIR). Focus on distributed training performance, scheduling, fault tolerance, and developer experience for thousands of accelerators.

## Job Description

## Impact

- Drive the architecture and evolution of AIR's managed GPU training platform, delivering scalable, high-throughput, and resilient training across fleets that span thousands of accelerators.
- Solve the hardest problems in large-scale training, including multi-node orchestration, distributed parallelism strategies, GPU scheduling and dynamic routing, high-throughput data loading, and checkpoint and restore for very long-running jobs.
- Push GPU efficiency and training performance, raising utilization (such as model FLOPs utilization and end-to-end throughput) and lowering cost per training run across diverse model architectures and hardware generations.
- Build the resilience and observability foundations that keep multi-node jobs healthy, detecting and recovering from hardware and software failures with minimal disruption to customers.
- Partner with product, research, and platform teams to shape the APIs, CLI, and developer experience that make it easy to launch, monitor, and debug production training jobs.
- Lead end-to-end engineering efforts, from design through production rollout, holding a high bar for performance, correctness, and reliability.
- Make direct, high-impact contributions to the core systems behind AIR, and help bring up support for the latest accelerators and new regions as the fleet grows.
- Champion engineering excellence, mentor other engineers through design reviews and technical discussions, and help shape Databricks' long-term technical direction in AI training infrastructure.

## Requirements

- 10+ years of experience building and operating large-scale distributed systems, with significant depth in GPU training infrastructure, high-performance computing, or ML systems.
- Hands-on experience with distributed training frameworks (such as PyTorch, FSDP, DeepSpeed, or Megatron) and the parallelism strategies (data, tensor, pipeline, and sequence parallelism) used to train large models.
- Strong understanding of training resilience patterns, including checkpointing, failure detection, and automatic recovery for long-running, multi-node jobs.
- Solid grasp of GPU performance fundamentals, including accelerator architecture, high-speed interconnects (such as NVLink and InfiniBand or RoCE), collective communication, and the bottlenecks that govern training throughput and utilization.
- Experience building and operating managed, multi-tenant platform products in the cloud, with clear SLAs and SLOs for availability, performance, and reliability.
- Strong foundation in algorithms, data structures, and system design as applied to performance-sensitive, large-scale distributed systems.
- Proven ability to deliver technically complex, high-impact initiatives that create clear customer or business value.
- Strong communication skills and the ability to collaborate across product, research, and infrastructure teams in a fast-moving environment.
- Strategic, product-oriented mindset with the ability to align technical execution to a long-term vision, and a passion for mentoring engineers and fostering technical excellence.
- BS in Computer Science or a related field (MS or PhD preferred).

## Similar roles

- [Staff Machine Learning Engineer](https://hotfix.jobs/jobs/d1943f91-7f1d-4c37-840b-24ef6a11d96b) - Databricks - San Francisco, CA - $190k – $285k/yr
- [Staff Machine Learning Engineer, Computer Vision](https://hotfix.jobs/jobs/2969c5c0-8479-43bb-b9b6-f8df93b528d7) - Pinterest - San Francisco, CA - $189k – $390k/yr
- [Staff Machine Learning Engineer](https://hotfix.jobs/jobs/3c08f024-1b6c-4c43-86de-db177792a225) - Pinterest - San Francisco, CA - $189k – $390k/yr
- [Staff Software Engineer - GenAI Performance and Kernel](https://hotfix.jobs/jobs/7d20b86b-17a6-4dbe-8bc0-89703822b0fc) - Databricks - San Francisco, CA - $191k – $233k/yr
- [Staff Software Engineer - GenAI inference](https://hotfix.jobs/jobs/c24ce5c3-c403-4ff3-90c0-c110271f0098) - Databricks - San Francisco, CA - $191k – $233k/yr

**Apply:** https://hotfix.jobs/jobs/2ea7944c-45ee-4f48-8c85-a9ae03a53b15
**Canonical:** https://hotfix.jobs/jobs/2ea7944c-45ee-4f48-8c85-a9ae03a53b15