# Senior Software Engineer, AI Runtime

**Company:** [Databricks](https://hotfix.jobs/companies/databricks)
**Location:** Mountain View, CA, San Francisco, CA
**Role:** ML Engineering
**Salary:** $160k – $225k/yr
**Experience:** 5+ years
**Skills:** PyTorch, Fsdp, Deepspeed, Megatron, Gpu Scheduling, Distributed Training, Checkpointing, Nvlink, InfiniBand, Roce, Collective Communication, Multi-Node Orchestration, High-Performance Computing, Ml Systems, System Design
**Posted:** 2026-06-08

> Senior Software Engineer building and scaling Databricks' managed GPU training platform (AI Runtime) for large-scale distributed AI model training. Requires 5+ years in distributed systems and hands-on experience with GPU training frameworks.

## Job Description

## Responsibilities
- Drive the architecture and evolution of AIR's managed GPU training platform, delivering scalable, high-throughput, and resilient training across fleets that span thousands of accelerators.
- Solve the hardest problems in large-scale training, including multi-node orchestration, distributed parallelism strategies, GPU scheduling and dynamic routing, high-throughput data loading, and checkpoint and restore for very long-running jobs.
- Push GPU efficiency and training performance, raising utilization (such as model FLOPs utilization and end-to-end throughput) and lowering cost per training run across diverse model architectures and hardware generations.
- Build the resilience and observability foundations that keep multi-node jobs healthy, detecting and recovering from hardware and software failures with minimal disruption to customers.
- Partner with product, research, and platform teams to shape the APIs, CLI, and developer experience that make it easy to launch, monitor, and debug production training jobs.
- Lead end-to-end engineering efforts, from design through production rollout, holding a high bar for performance, correctness, and reliability.
- Make direct, high-impact contributions to the core systems behind AIR, and help bring up support for the latest accelerators and new regions as the fleet grows.
- Champion engineering excellence, mentor other engineers through design reviews and technical discussions, and contribute to Databricks' technical direction in AI training infrastructure.

## Requirements
- 5+ years of experience building and operating large-scale distributed systems, with experience in GPU training infrastructure, high-performance computing, or ML systems.
- Experience with distributed training frameworks (such as PyTorch, FSDP, DeepSpeed, or Megatron) and the parallelism strategies (data, tensor, pipeline, and sequence parallelism) used to train large models.
- Strong understanding of training resilience patterns, including checkpointing, failure detection, and automatic recovery for long-running, multi-node jobs.
- Solid grasp of GPU performance fundamentals, including accelerator architecture, high-speed interconnects (such as NVLink and InfiniBand or RoCE), collective communication, and the bottlenecks that govern training throughput and utilization.
- Experience building and operating managed, multi-tenant platform products in the cloud, with clear SLAs and SLOs for availability, performance, and reliability.
- Strong foundation in algorithms, data structures, and system design as applied to performance-sensitive, large-scale distributed systems.
- Proven ability to deliver technically complex, high-impact initiatives that create clear customer or business value.
- Strong communication skills and the ability to collaborate across product, research, and infrastructure teams in a fast-moving environment.
- Customer-focused mindset with the ability to align implementation details with product goals, and a passion for mentoring engineers and fostering technical excellence.
- BS in Computer Science or a related field (MS or PhD preferred).

## Nice-to-Haves
- MS or PhD in Computer Science or a related field.

## Similar roles

- [Senior Software Engineer](https://hotfix.jobs/jobs/423c60aa-ca4c-400d-babe-a50099433cbb) - SpotOn - Chicago, IL - $160k – $190k/yr
- [Senior Engineer, Applied AI](https://hotfix.jobs/jobs/fde322de-5f40-47c3-81b1-e7c042bd1df2) - Honor - Remote - $160k – $180k/yr
- [Senior AI Engineer](https://hotfix.jobs/jobs/2716f0ca-948f-41b7-83c2-06362589b8a7) - Octus - New York, NY - $160k – $170k/yr
- [Senior Forward-Deployed Engineer, Federal](https://hotfix.jobs/jobs/d963f808-e129-4719-ac6b-6aaa2103050f) - Deepgram - Washington, DC - $160k – $200k/yr
- [Senior Full Stack AI Engineer (Rapid Prototyping & Analytics )](https://hotfix.jobs/jobs/13f303ca-d8b8-4357-a5a2-7de425d06903) - Prompt Health - Remote - $160k – $220k/yr

**Apply:** https://hotfix.jobs/jobs/f56a1580-de92-4d64-bf6f-dcedbf8b8082
**Canonical:** https://hotfix.jobs/jobs/f56a1580-de92-4d64-bf6f-dcedbf8b8082