# Senior Software Engineer, Model Infrastructure

**Company:** [Harvey](https://hotfix.jobs/companies/harvey)
**Location:** San Francisco, CA
**Role:** Backend Engineering
**Salary:** $193k – $290k/yr
**Experience:** 5+ years
**Skills:** Go, Java, Python, Rust, C++, Distributed Systems, Cloud Infrastructure, Networking, Observability, Kubernetes, Service Mesh, Kafka, Spark, Flink, Airflow
**Posted:** 2026-09-06

> Leads development of Harvey’s model infrastructure platform, including reliable AI inference, model routing, provider integrations, observability, and capacity management. Requires distributed-systems experience, strong programming skills, and technical leadership across engineering teams.

## Job Description

## Responsibilities
- Lead the design and implementation of Harvey’s Model Infrastructure platform.
- Build highly available, low-latency, operationally excellent systems for AI inference.
- Design and improve the Unified Model Controller and Model Selector to detect model degradation and route traffic based on reliability, latency, quality, compliance, and cost.
- Develop model provisioning, capacity management, failover, and traffic-engineering systems across multiple AI providers.
- Integrate model providers and maintain provider APIs and SDKs.
- Build observability capabilities including health dashboards, alerting, token-usage analytics, cost reporting, and end-to-end telemetry.
- Partner with Product Engineering on model launches, experimentation, and production monitoring.
- Drive capacity planning, utilization optimization, and cost visibility.
- Collaborate with AI Research on infrastructure for model evaluation, training, and deployment.
- Lead cross-functional technical initiatives and mentor engineers.

## Requirements
- 4+ years of software engineering experience building large-scale distributed systems.
- Experience designing and operating highly available production services.
- Strong programming skills in Go, Java, Python, Rust, or C++.
- Deep understanding of distributed systems, cloud infrastructure, networking, and observability.
- Experience leading technical projects across multiple engineering teams.
- Ability to balance long-term architecture with pragmatic execution.
- Strong communication and collaboration skills.
- Passion for building foundational platforms for other engineering teams.

## Nice to Have
- Experience with AI infrastructure, LLM serving, or machine learning platforms.
- Experience with model routing, inference gateways, or policy-based serving systems.
- Experience with OpenAI, Anthropic, Azure OpenAI, Fireworks, Baseten, or open-source LLMs.
- Experience with Kubernetes, cloud infrastructure, and service mesh technologies.
- Experience with large-scale observability and SRE practices.
- Experience with Kafka, Spark, Flink, Airflow, or Iceberg.
- Familiarity with GPU infrastructure or model training platforms.

## Compensation
- $193,400–$290,000 USD

## Similar jobs

- [Senior Software Engineer, Video Streaming](https://hotfix.jobs/jobs/325e70d9-cd4d-43ee-8363-6ea4c3668fb2) - Nuro - Mountain View, CA - $194k – $291k/yr
- [Senior Software Engineer, Networking & Real-Time Systems](https://hotfix.jobs/jobs/f50e9f5a-e607-4d00-9b63-05cb8ad54f78) - Nuro - Mountain View, CA - $194k – $291k/yr
- [Tech Lead Software Engineer, Fleet Connectivity](https://hotfix.jobs/jobs/f654dc49-4a65-45b1-9008-5f604769d4fd) - Nuro - Mountain View, CA - $194k – $291k/yr
- [Senior Software Engineer, Test Core](https://hotfix.jobs/jobs/f33409a0-62e5-4cca-9887-eb5fbd75e55f) - Vanta - Remote - $195k – $229k/yr
- [Senior Software Engineer, Core Platform](https://hotfix.jobs/jobs/9a340daf-30c5-4179-bfee-744833373b5a) - Pave - San Francisco, CA - $196k – $265k/yr

**Apply:** https://hotfix.jobs/jobs/39cb3280-96b4-422b-a0de-56b7d6808902
**Canonical:** https://hotfix.jobs/jobs/39cb3280-96b4-422b-a0de-56b7d6808902