# Software Engineer, ML Serving

**Company:** [Unusual](https://hotfix.jobs/companies/unusual)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Experience:** 5+ years
**Skills:** Ml Serving, Nvidia Triton, vLLM, Sglang, Tensor Parallel, Pipeline Parallel, Linux, Docker, Kubernetes, Terraform, gRPC, Webrtc, AWS, GCP
**Posted:** 2026-06-24

> Build and scale real-time TTS serving infrastructure for voice AI models, from GPU inference engines to production APIs. Requires hands-on experience with multinode ML serving frameworks, distributed inference, and cloud/SRE practices.

## Job Description

## What You'll Own

- Architecture and implementation of Rime's TTS serving infrastructure, from GPU-backed inference engines to the API surface.
- Model optimization from a single-node to disaggregated fleet serving.
- Compatibility with different NVIDIA hardwares from Hopper to Blackwell and beyond for on-prem and cloud deployments.
- Continuous integration and deployment workflows for the model serving pipeline.
- Site reliability: on-call rotation, monitoring, alerting, and observability across the serving stack.
- Resource provision, cost management across our GPU fleet.

## What We're Looking For

- Hands-on experience with real-time multinode ML serving infrastructure — ML serving framework experience: NVIDIA Dynamo/Triton, vLLM, SGLang, or equivalent.
- Experience with distributed or disaggregated model serving (Tensor Parallel, Pipeline Parallel, or equivalent).
- Strong cloud infrastructure fundamentals: Linux internals, networking, containerization (Docker, Kubernetes).
- IaC experience — Terraform, Packer, or comparable.
- On-call is part of the job. You treat production reliability as a shared responsibility.

## Nice to Have

- Experience with multinode training (DDP, FSDP, etc.).
- Experience with gRPC or other bidirectional binary streaming protocols.
- Experience with audio streaming and related technologies (WebRTC, WebSockets, etc.).
- Experience with a multilingual monorepo where you pick the best language out of merit more than personal experience.
- Experience with multi-cloud infrastructures (AWS, GCP, OCI, etc.).
- Comfort with configuration management tooling (Ansible, Chef, Puppet, or similar).
- SRE, DevOps, or platform engineering background at a startup.
- Experience at an early-stage company.

## Similar roles

- [CoDesign & NextGen Performance Engineer](https://hotfix.jobs/jobs/2cb728b7-a9ea-44b0-b16a-47364021f7be) - Cerebras Systems - Sunnyvale, CA
- [Research Engineer, Privacy](https://hotfix.jobs/jobs/3220c5d0-0e5f-4869-b133-1ed126b4208a) - OpenAI - San Francisco, CA - $380k – $445k/yr
- [Research Engineer](https://hotfix.jobs/jobs/b4d18018-e64d-4828-a8b4-303225328e17) - Console - San Francisco, CA - $200k – $350k/yr
- [Software Engineer, AI Platform](https://hotfix.jobs/jobs/7ddad000-3e1c-47cb-a45b-31d8e72abc4f) - Notion - San Francisco, CA - $180k – $201k/yr
- [Machine Learning Engineer](https://hotfix.jobs/jobs/97713038-58cb-46c5-a9c7-0e1554e1ba4f) - Liftoff - Remote - $215k – $275k/yr

**Apply:** https://hotfix.jobs/jobs/a766b5cb-314d-4dd8-8fd4-afa254e360dc
**Canonical:** https://hotfix.jobs/jobs/a766b5cb-314d-4dd8-8fd4-afa254e360dc