# Staff AI Runtime Engineer

**Company:** [FlexAI](https://hotfix.jobs/companies/flexai)
**Location:** Santa Clara, CA
**Role:** ML Engineering
**Experience:** 8+ years
**Skills:** PyTorch, TensorFlow, JAX, Python, C++, Kubernetes, Ray, Torchelastic, Distributed Training, Multi-Gpu
**Posted:** 2026-03-24

> Designs, develops, and optimizes core runtime infrastructure for distributed AI training and inference using PyTorch-based stack. Requires 8+ years in systems engineering, deep learning runtimes, Python/C++, and multi-node GPU workloads.

## Job Description

## What You'll Do

**Lead Runtime Design & Development:**
- Own the core runtime architecture supporting AI training and inference at scale.
- Design resilient and elastic runtime features (e.g. dynamic node scaling, job recovery) within our custom PyTorch stack.
- Optimize distributed training reliability, orchestration, and job-level fault tolerance.

**Drive Performance at Scale:**
- Profile and enhance low-level system performance across training and inference pipelines.
- Improve packaging, deployment, and integration of customer models in production environments.
- Ensure consistent throughput, latency, and reliability metrics across multi-node, multi-GPU setups.

**Build Internal Tooling & Frameworks:**
- Design and maintain libraries and services that support model lifecycle: training, checkpointing, fault recovery, packaging, and deployment.
- Implement observability hooks, diagnostics, and resilience mechanisms for deep learning workloads.
- Champion best practices in CI/CD, testing, and software quality across the AI Runtime stack.

**Collaborate & Mentor:**
- Work cross-functionally with Research, Infrastructure, and Product teams to align runtime development with customer and platform needs.
- Guide technical discussions, mentor junior engineers, and help scale the AI Runtime team’s capabilities.

## What You’ll Need to Be Successful
- 8+ years of experience in systems/software engineering, with deep exposure to AI runtime, distributed systems, or compiler/runtime interaction.
- Experience in delivering PaaS services.
- Proven experience optimizing and scaling deep learning runtimes (e.g. PyTorch, TensorFlow, JAX) for large-scale training and/or inference.
- Strong programming skills in **Python** and **C++** (Go or Rust is a plus).
- Familiarity with distributed training frameworks, low-level performance tuning, and resource orchestration.
- Experience working with multi-GPU, multi-node, or cloud-native AI workloads.
- Solid understanding of containerized workloads, job scheduling, and failure recovery in production environments.

## Nice to Have
- Contributions to PyTorch internals or open-source DL infrastructure projects.
- Familiarity with LLM training pipelines, checkpointing, or elastic training orchestration.
- Experience with **Kubernetes**, **Ray**, **TorchElastic**, or custom AI job orchestrators.
- Background in systems research, compilers, or runtime architecture for HPC or ML.
- Startup previous experience

## Similar roles

- [Member of Technical Staff](https://hotfix.jobs/jobs/dd7c8bc4-741a-45d5-91a8-e6b1c4ad67e0) - xAI - Palo Alto, CA - $180k – $440k/yr
- [Staff Software Engineer, AI & App Experience](https://hotfix.jobs/jobs/78128221-8199-41cd-9c63-502d3f61d603) - Chime - New York, NY - $223k – $308k/yr
- [Staff + Senior Software Engineer, Cloud Inference Launch Engineering](https://hotfix.jobs/jobs/18b08a6a-5359-4836-b72e-c641fa718b04) - Anthropic - San Francisco, CA - $320k – $485k/yr
- [Senior Staff Software Engineer, Legal Automation](https://hotfix.jobs/jobs/7870d3e1-ebec-4f7b-9ae5-c62926a06d6e) - Coinbase - Remote - $254k – $299k/yr
- [Staff Machine Learning Engineer, Computer Vision](https://hotfix.jobs/jobs/2969c5c0-8479-43bb-b9b6-f8df93b528d7) - Pinterest - San Francisco, CA - $189k – $390k/yr

**Apply:** https://hotfix.jobs/jobs/249fa653-da1e-4cf1-90af-dd45b5b12d2d
**Canonical:** https://hotfix.jobs/jobs/249fa653-da1e-4cf1-90af-dd45b5b12d2d