Skip to content
UnusualUnusualSan Francisco, CA

Software Engineer, ML Serving

Build and scale real-time TTS serving infrastructure for voice AI models, from GPU inference engines to production APIs. Requires hands-on experience with multinode ML serving frameworks, distributed inference, and cloud/SRE practices.

Salary not listed
On-site5+ YOEML Engineering

About the role

What You'll Own

  • Architecture and implementation of Rime's TTS serving infrastructure, from GPU-backed inference engines to the API surface.
  • Model optimization from a single-node to disaggregated fleet serving.
  • Compatibility with different NVIDIA hardwares from Hopper to Blackwell and beyond for on-prem and cloud deployments.
  • Continuous integration and deployment workflows for the model serving pipeline.
  • Site reliability: on-call rotation, monitoring, alerting, and observability across the serving stack.
  • Resource provision, cost management across our GPU fleet.

What We're Looking For

  • Hands-on experience with real-time multinode ML serving infrastructure — ML serving framework experience: NVIDIA Dynamo/Triton, vLLM, SGLang, or equivalent.
  • Experience with distributed or disaggregated model serving (Tensor Parallel, Pipeline Parallel, or equivalent).
  • Strong cloud infrastructure fundamentals: Linux internals, networking, containerization (Docker, Kubernetes).
  • IaC experience — Terraform, Packer, or comparable.
  • On-call is part of the job. You treat production reliability as a shared responsibility.

Nice to Have

  • Experience with multinode training (DDP, FSDP, etc.).
  • Experience with gRPC or other bidirectional binary streaming protocols.
  • Experience with audio streaming and related technologies (WebRTC, WebSockets, etc.).
  • Experience with a multilingual monorepo where you pick the best language out of merit more than personal experience.
  • Experience with multi-cloud infrastructures (AWS, GCP, OCI, etc.).
  • Comfort with configuration management tooling (Ansible, Chef, Puppet, or similar).
  • SRE, DevOps, or platform engineering background at a startup.
  • Experience at an early-stage company.

Skills

Ml ServingNvidia TritonvLLMSglangTensor ParallelPipeline ParallelLinuxDockerKubernetesTerraformgRPCWebrtcAWSGCP

Similar roles

ML Engineering jobs
Cerebras Systems

CoDesign & NextGen Performance Engineer

Cerebras SystemsSunnyvale, CA

Characterize, analyze, and optimize performance of state-of-the-art AI models on Cerebras' wafer-scale hardware. Build performance models, optimize kernels and compilers, debug runtime behavior, and develop visualization tools to influence next-gen AI architecture.

Salary not listed
On-site3+ YOEML Engineering
OpenAI

Research Engineer, Privacy

OpenAISan Francisco, CA

Research Engineer on OpenAI's Privacy team designing and prototyping privacy-preserving ML algorithms like differential privacy and federated learning at scale. Requires hands-on PETs experience, fluency in PyTorch/JAX, and a track record implementing or publishing novel privacy work.

380k – 445k/yr
HybridML Engineering
Console

Research Engineer

ConsoleSan Francisco, CA

Research Engineer building self-improving AI agent systems at Console. Develop eval/optimization loops, fine-tune specialist models, and improve agent reasoning over enterprise context using production data to drive measurable gains in quality, latency, and reliability.

200k – 350k/yr
On-siteML Engineering
Notion

Software Engineer, AI Platform

NotionSan Francisco, CA +1

Build and scale the shared AI platform foundations at Notion, enabling fast and safe shipping of AI products. Requires experience with LLM/ML platforms, strong ownership, and comfort across backend, infrastructure, and product code.

180k – 201k/yr
Hybrid5+ YOEML Engineering
Liftoff

Machine Learning Engineer

LiftoffCalifornia

Machine Learning Engineer building statistical models, optimization systems, and experiments for mobile ad tech economics on the Revenue Engine team. Requires PhD in CS/ML/Economics and industry experience applying ML or economics at scale.

215k – 275k/yr
RemoteML Engineering