Software Engineer - Model APIs
Builds and optimizes Model APIs for high-performance AI model serving, focusing on inference capabilities, low-latency backend services, and GPU optimizations. Requires 3+ years in distributed systems with strong performance debugging skills.
About the job
Responsibilities
- Design, build, and operate the Model APIs surface with focus on advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling and multi-modal serving
- Profile and optimize TensorRT-LLM kernels, analyze CUDA kernel performance, implement custom CUDA operators, tune memory allocation patterns for maximum throughput and optimize communication patterns across multi-GPU setups
- Productionize performance improvements across runtimes with deep understanding of their internals: speculative decoding implementations, guided generation for structured outputs, custom scheduling and routing algorithms for high-performance serving
- Build comprehensive benchmarking frameworks that measure real-world performance across different model architectures, batch sizes, sequence lengths, and hardware configurations
- Productionize performance improvements across runtimes (e.g. TensorRT, TensorRT-LLM): speculative decoding, quantization, batching, and KV-cache reuse
- Instrument deep observability (metrics, traces, logs) and build repeatable benchmarks to measure speed, reliability, and quality
- Implement platform fundamentals: API versioning, validation, usage metering, quotas, and authentication
- Collaborate closely with other teams to deliver robust, developer-friendly model serving experiences
Requirements
- 3+ years experience building and operating distributed systems or large-scale APIs
- Proven track record of owning low-latency, reliable backend services (rate-limiting, auth, quotas, metering, migrations)
- Infra instincts with performance sensibilities: profiling, tracing, capacity planning, and SLO management
- Comfortable debugging complex systems, from runtime internals to GPU execution traces
- Strong written communication; able to produce clear design docs and collaborate across functions
Nice to Have
- Experience with LLM runtimes (vLLM, SGLang, TensorRT-LLM) or contributions to open-source inference engines (vLLM, TensorRT-LLM, SGLang, TGI)
- Knowledge of Kubernetes, service meshes, API gateways, or distributed scheduling
- Background in developer-facing infrastructure or open-source APIs
Skills
Tensorrt-Llm, CUDA, Kubernetes, vLLM, Sglang, Tgi, Distributed Systems, Api Gateways, Service Meshes, Speculative Decoding, Quantization, Kv-Cache
Similar jobs
Backend Engineering jobs
Kalshi
New York, NY
Build and maintain high-performance backend trading infrastructure, including order-management systems and event-driven APIs. The role requires at least three years of backend engineering experience, concurrent programming expertise, relational database proficiency, AWS familiarity, and Go, Rust, or C++ skills.
Tessera Labs
San Jose, CA
Build and scale backend APIs, microservices, data pipelines, and enterprise integrations powering an AI automation platform. The role requires 3–5 years of backend experience, strong Python skills, and familiarity with databases, cloud infrastructure, security, and distributed systems.
Mirage
New York, NY
Build and own backend services, APIs, data pipelines, and infrastructure powering an AI-native video platform. The role requires 5+ years of industry experience and strong expertise in distributed systems, production scaling, and generative AI integration.
Sesame
San Francisco, CABuild and operate scalable backend systems for voice-agent products, spanning distributed services, data infrastructure, real-time audio, and ML-enabled workflows. The role requires expert programming ability, strong database and reliability engineering skills, Kubernetes experience, and production experience with systems at scale.
Zoox
Foster City, CA
Build foundational, high-performance C++ systems that capture, store, retrieve, and transmit data from autonomous vehicle fleets. The role requires a bachelor’s degree, 3+ years of software development experience, and expertise in concurrency, networking, or inter-process communication.