# AI Infrastructure Engineer, Model Serving Platform

**Company:** [Scale AI](https://hotfix.jobs/companies/scale-ai)
**Location:** San Francisco, CA, New York, NY, Seattle, WA
**Role:** DevOps / SRE
**Salary:** $179k – $224k/yr
**Experience:** 4+ years
**Skills:** Python, Go, Rust, C++, Kubernetes, Docker, AWS, GCP, Terraform, vLLM
**Posted:** 2026-02-12

> Designs and builds scalable platforms for serving LLMs, focusing on fault-tolerant systems, model integration, and observability. Requires 4+ years in backend systems, strong programming skills, and LLM serving expertise.

## Job Description

## You will:
- Build and maintain fault-tolerant, high-performance systems for serving LLMs workloads at scale.
- Build an internal platform to empower LLM capability discovery.
- Collaborate with researchers and engineers to integrate and optimize models for production and research use cases.
- Conduct architecture and design reviews to uphold best practices in system design and scalability.
- Develop monitoring and observability solutions to ensure system health and performance.
- Lead projects end-to-end, from requirements gathering to implementation, in a cross-functional environment.

## Ideally you'd have:
- 4+ years of experience building large-scale, high-performance backend systems.
- Strong programming skills in one or more languages (e.g., **Python**, **Go**, **Rust**, **C++**).
- Experience with LLM serving and routing fundamentals (e.g. rate limiting, token streaming, load balancing, budgets, etc.)
- Experience with LLM capabilities and concepts such as reasoning, tool calling, prompt templates, etc.
- Experience with containers and orchestration tools (e.g., **Docker**, **Kubernetes**).
- Familiarity with cloud infrastructure (**AWS**, **GCP**) and infrastructure as code (e.g., **Terraform**).
- Proven ability to solve complex problems and work independently in fast-moving environments.

## Nice to haves:
- Experience with modern LLM serving frameworks such as **vLLM**, **SGLang**, **TensorRT-LLM**, or **text-generation-inference**.

## Similar roles

- [Platform Engineer](https://hotfix.jobs/jobs/14231c8e-abad-4a41-94e3-d41fb2ede6f5) - Govsignals - New York, NY - $180k – $250k/yr
- [Network Engineer, Design & Engineering](https://hotfix.jobs/jobs/48ff3694-5232-4d66-b43c-a091b96c8902) - Fluidstack - New York, NY - $180k – $300k/yr
- [Developer Productivity Engineer](https://hotfix.jobs/jobs/9a60db26-80d5-4b2a-b613-56ae58d49794) - Hightouch - Remote - $180k – $320k/yr
- [Data Center Network Engineer](https://hotfix.jobs/jobs/473ec43a-a5df-4433-99bd-4053f3e6f6bc) - Baseten - San Francisco, CA - $180k – $360k/yr
- [Infrastructure Engineer (Observability)](https://hotfix.jobs/jobs/098c81d9-0827-4baa-a70b-444cc5b04f37) - Lightning AI - Remote - $180k – $200k/yr

**Apply:** https://hotfix.jobs/jobs/0ad9c0be-8704-4076-a1bd-761c6a1f4e63
**Canonical:** https://hotfix.jobs/jobs/0ad9c0be-8704-4076-a1bd-761c6a1f4e63