# Senior Backend Engineer, Inference Platform

**Company:** [Together AI](https://hotfix.jobs/companies/together-ai)
**Location:** San Francisco, CA
**Role:** Backend Engineering
**Salary:** $160k – $250k/yr
**Experience:** 5+ years
**Skills:** Rust, Go, Python, TypeScript, Kubernetes, Sglang, vLLM, Nvidia Dynamo, CUDA, Distributed Systems
**Posted:** 2026-01-05

> Build and optimize the core inference platform powering frontier AI models at global scale, focusing on low-latency routing, auto-scaling, and GPU resource management. Requires 5+ years in large-scale distributed systems and expert programming skills.

## Job Description

## Responsibilities

- Build and optimize global and local request routing, ensuring low-latency load balancing across data centers and model engine pods.
- Develop auto-scaling systems to dynamically allocate resources and meet strict SLOs across dozens of data centers.
- Design systems for multi-tenant traffic shaping, tuning both resource allocation and request handling — including smart rate limiting and regulation — to ensure fairness and consistent experience across all users.
- Engineer trade-offs between latency and throughput to serve diverse workloads efficiently.
- Optimize prefix caching to reduce model compute and speed up responses.
- Collaborate with ML researchers to bring new model architectures into production at scale.
- Continuously profile and analyze system-level performance to identify bottlenecks and implement optimizations.

## Requirements

- 5+ years of demonstrated experience building large-scale, fault-tolerant, distributed systems and API microservices.
- Strong background in designing, analyzing, and improving efficiency, scalability, and stability of complex systems.
- Excellent understanding of low-level OS concepts: multi-threading, memory management, networking, and storage performance.
- Expert-level programming in one or more of: Rust, Go, Python, or TypeScript.
- Knowledge of modern LLMs and generative models and how they are served in production is a plus.
- Experience working with the open source ecosystem around inference is highly valuable; familiarity with SGLang, vLLM, or NVIDIA Dynamo will be especially handy.
- Experience with Kubernetes or container orchestration is a strong plus.
- Familiarity with GPU software stacks (CUDA, Triton, NCCL) and HPC technologies (InfiniBand, NVLink, MPI) is a plus.
- Bachelor’s or Master’s degree in Computer Science, Computer Engineering, or related field, or equivalent practical experience.

## Compensation

US base salary range: $160,000 - $250,000 + equity + benefits. Salary determined by location, level, role, experience, skills, and job-related knowledge.

## Similar roles

- [Senior Backend Engineer](https://hotfix.jobs/jobs/4b7e2c83-fc89-4cec-bff5-78a5c1cfc2d7) - Aurelian - Seattle, WA - $160k – $220k/yr
- [Senior Software Engineer, Platform Infrastructure](https://hotfix.jobs/jobs/8763e3a9-7407-4f92-ad6d-7419cc31b95d) - Aptos - Palo Alto, CA - $160k – $260k/yr
- [Senior Software Engineer - Endpoint Services Portal](https://hotfix.jobs/jobs/32a4fddd-c6c1-4932-b757-02fa73f358df) - Huntress - Remote - $160k – $190k/yr
- [Sr Software Engineer, Edge](https://hotfix.jobs/jobs/0abc1162-dbc5-491c-b4f4-164fef4ab9b7) - Cribl - Remote - $160k – $210k/yr
- [Senior Software Engineer, Platform](https://hotfix.jobs/jobs/d822ed13-8628-41f6-97e2-6918d8c1c600) - SnapLogic - San Mateo, CA - $160k – $180k/yr

**Apply:** https://hotfix.jobs/jobs/0673feac-070c-4281-bd1d-6de838d97bf9
**Canonical:** https://hotfix.jobs/jobs/0673feac-070c-4281-bd1d-6de838d97bf9