# Staff Software Engineer, Model Infrastructure

**Company:** [Harvey](https://hotfix.jobs/companies/harvey)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Salary:** $231k – $340k/yr
**Experience:** 7+ years
**Skills:** Go, Java, Python, Rust, C++, Distributed Systems, Cloud Infrastructure, Kubernetes, Model Routing, Llm Serving, Observability, Networking, Kafka, Spark, Flink
**Posted:** 2026-09-06

> Leads the design and operation of reliable, scalable model infrastructure powering AI inference across multiple providers. Requires 7+ years of distributed-systems engineering experience, strong programming skills, and expertise in production reliability and cloud infrastructure.

## Job Description

## Responsibilities
- Lead the design and implementation of the Model Infrastructure platform.
- Build highly available, low-latency, operationally excellent systems for AI inference.
- Design and improve the Unified Model Controller and Model Selector for degradation detection and policy-based traffic routing.
- Develop model provisioning, capacity management, failover, and traffic-engineering systems across multiple AI providers.
- Integrate model providers and maintain provider APIs and SDKs.
- Improve observability with health dashboards, alerting, token-usage analytics, cost reporting, and end-to-end telemetry.
- Support model launches, experimentation, and monitoring of production AI workloads.
- Drive capacity planning, utilization optimization, infrastructure efficiency, and cost visibility.
- Collaborate on infrastructure for model evaluation, training, deployment, and agent platforms.
- Lead cross-functional technical initiatives and mentor engineers.

## Requirements
- 7+ years of software engineering experience building large-scale distributed systems.
- Experience designing and operating highly available production services.
- Strong programming skills in Go, Java, Python, Rust, or C++.
- Deep understanding of distributed systems, cloud infrastructure, networking, and observability.
- Experience leading technical projects across multiple engineering teams.
- Ability to balance long-term architecture with pragmatic execution.
- Strong communication and collaboration skills.

## Nice to Have
- Experience with AI infrastructure, LLM serving, or machine-learning platforms.
- Experience with model routing, inference gateways, or policy-based serving systems.
- Experience with OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, or open-source LLMs.
- Experience with Kubernetes, cloud infrastructure, and service-mesh technologies.
- Experience with large-scale observability and SRE best practices.
- Experience with Kafka, Spark, Flink, Airflow, or Iceberg.
- Familiarity with GPU infrastructure or model-training platforms.

## Compensation
- $231,000–$340,000 USD annually.

## Similar jobs

- [Staff Software Engineer, Autonomy Capabilities](https://hotfix.jobs/jobs/02d97f5d-f4e7-445e-8842-756b55d1843c) - Shield AI - San Mateo, CA - $234k – $351k/yr
- [Staff Software Engineer - Snowflake Feature Store](https://hotfix.jobs/jobs/66983557-05aa-4583-a35d-38df9881675d) - Snowflake - Bellevue, WA - $236k – $339k/yr
- [Staff Engineer, Autonomy Capabilities – Maritime](https://hotfix.jobs/jobs/6584cef4-dd17-40e1-854e-fb8a2411eb50) - Shield AI - Washington, DC - $221k – $331k/yr
- [Senior Staff Software Engineer, Agentic Search](https://hotfix.jobs/jobs/5beacffe-b81b-4377-8a7b-fb5c956d03ee) - Ironclad - San Francisco, CA - $220k – $270k/yr
- [Staff Software Engineer - AI Agent Evaluations](https://hotfix.jobs/jobs/d3ace257-7e34-447e-83ab-6d583c58936a) - Idme - Mountain View, CA - $218k – $271k/yr

**Apply:** https://hotfix.jobs/jobs/e7669ca0-c832-485f-9b10-dafc4bf42dc3
**Canonical:** https://hotfix.jobs/jobs/e7669ca0-c832-485f-9b10-dafc4bf42dc3