Senior Backend Engineer, Inference Platform

Responsibilities

• Build and optimize global and local request routing, ensuring low-latency load balancing across data centers and model engine pods.
• Develop auto-scaling systems to dynamically allocate resources and meet strict SLOs across dozens of data centers.
• Design systems for multi-tenant traffic shaping, tuning both resource allocation and request handling — including smart rate limiting and regulation — to ensure fairness and consistent experience across all users.
• Engineer trade-offs between latency and throughput to serve diverse workloads efficiently.
• Optimize prefix caching to reduce model compute and speed up responses.
• Collaborate with ML researchers to bring new model architectures into production at scale.
• Continuously profile and analyze system-level performance to identify bottlenecks and implement optimizations.

Requirements

• 5+ years of demonstrated experience building large-scale, fault-tolerant, distributed systems and API microservices.
• Strong background in designing, analyzing, and improving efficiency, scalability, and stability of complex systems.
• Excellent understanding of low-level OS concepts: multi-threading, memory management, networking, and storage performance.
• Expert-level programming in one or more of: Rust, Go, Python, or TypeScript.
• Knowledge of modern LLMs and generative models and how they are served in production is a plus.
• Experience working with the open source ecosystem around inference is highly valuable; familiarity with SGLang, vLLM, or NVIDIA Dynamo will be especially handy.
• Experience with Kubernetes or container orchestration is a strong plus.
• Familiarity with GPU software stacks (CUDA, Triton, NCCL) and HPC technologies (InfiniBand, NVLink, MPI) is a plus.
• Bachelor’s or Master’s degree in Computer Science, Computer Engineering, or related field, or equivalent practical experience.

Compensation

US base salary range: $160,000 - $250,000 + equity + benefits. Salary determined by location, level, role, experience, skills, and job-related knowledge.