Senior Backend Engineer, Inference Platform
Build and optimize the core inference platform powering frontier AI models at global scale, focusing on low-latency routing, auto-scaling, and GPU resource management. Requires 5+ years in large-scale distributed systems and expert programming skills.
About the job
Responsibilities
- Build and optimize global and local request routing, ensuring low-latency load balancing across data centers and model engine pods.
- Develop auto-scaling systems to dynamically allocate resources and meet strict SLOs across dozens of data centers.
- Design systems for multi-tenant traffic shaping, tuning both resource allocation and request handling — including smart rate limiting and regulation — to ensure fairness and consistent experience across all users.
- Engineer trade-offs between latency and throughput to serve diverse workloads efficiently.
- Optimize prefix caching to reduce model compute and speed up responses.
- Collaborate with ML researchers to bring new model architectures into production at scale.
- Continuously profile and analyze system-level performance to identify bottlenecks and implement optimizations.
Requirements
- 5+ years of demonstrated experience building large-scale, fault-tolerant, distributed systems and API microservices.
- Strong background in designing, analyzing, and improving efficiency, scalability, and stability of complex systems.
- Excellent understanding of low-level OS concepts: multi-threading, memory management, networking, and storage performance.
- Expert-level programming in one or more of: Rust, Go, Python, or TypeScript.
- Knowledge of modern LLMs and generative models and how they are served in production is a plus.
- Experience working with the open source ecosystem around inference is highly valuable; familiarity with SGLang, vLLM, or NVIDIA Dynamo will be especially handy.
- Experience with Kubernetes or container orchestration is a strong plus.
- Familiarity with GPU software stacks (CUDA, Triton, NCCL) and HPC technologies (InfiniBand, NVLink, MPI) is a plus.
- Bachelor’s or Master’s degree in Computer Science, Computer Engineering, or related field, or equivalent practical experience.
Compensation
US base salary range: $160,000 - $250,000 + equity + benefits. Salary determined by location, level, role, experience, skills, and job-related knowledge.
Skills
Rust, Go, Python, TypeScript, Kubernetes, Sglang, vLLM, Nvidia Dynamo, CUDA, Distributed Systems
Similar jobs
Backend Engineering jobs
Lithic
United States
Build and operate the foundational API platform powering Lithic’s fintech products, with ownership across reliability, authorization, audit logging, and API evolution. Requires 4+ years of backend experience, strong API design skills, and senior-level ownership in a remote, async environment.
Granica
Mountain View, CA
Build and optimize distributed compute infrastructure for enterprise-scale analytics and AI workloads, improving query performance, reliability, scheduling, and compute costs. Requires senior-level distributed systems experience and production expertise with Spark or comparable query and data-processing engines.
Opto
New York, NYBuild and own backend software for a private-markets investment platform, contributing across the full development lifecycle. The role requires 6+ years of software development experience, strong Python, REST API, SQL, and distributed-systems expertise, plus familiarity with AWS and AI/LLM applications.
Hinge Health
San Francisco, CA
Senior backend engineer designing and operating reliable billing and financial systems, APIs, data models, and distributed workflows. The role requires 5+ years of professional software development experience, strong backend expertise, and collaboration across Product, Finance, Operations, and Data.
Launchdarkly
United States
Designs and operates scalable backend services for an experimentation platform, including data ingestion, metric computation, and results processing. Requires 6+ years of software engineering experience, strong Go or comparable backend-language skills, and experience with distributed, cloud-based, data-intensive systems.