# Software Engineer - Model APIs

**Company:** [Baseten](https://hotfix.jobs/companies/baseten)
**Location:** San Francisco, CA, New York, NY
**Role:** Backend Engineering
**Salary:** $180k – $360k/yr
**Experience:** 3+ years
**Skills:** Tensorrt-Llm, CUDA, Kubernetes, vLLM, Sglang, Tgi, Distributed Systems, Api Gateways, Service Meshes, Speculative Decoding, Quantization, Kv-Cache
**Posted:** 2025-10-11

> Builds and optimizes Model APIs for high-performance AI model serving, focusing on inference capabilities, low-latency backend services, and GPU optimizations. Requires 3+ years in distributed systems with strong performance debugging skills.

## Job Description

## Responsibilities
- Design, build, and operate the Model APIs surface with focus on advanced inference capabilities: structured outputs (JSON mode, grammar-constrained generation), tool/function calling and multi-modal serving
- Profile and optimize TensorRT-LLM kernels, analyze CUDA kernel performance, implement custom CUDA operators, tune memory allocation patterns for maximum throughput and optimize communication patterns across multi-GPU setups
- Productionize performance improvements across runtimes with deep understanding of their internals: speculative decoding implementations, guided generation for structured outputs, custom scheduling and routing algorithms for high-performance serving
- Build comprehensive benchmarking frameworks that measure real-world performance across different model architectures, batch sizes, sequence lengths, and hardware configurations
- Productionize performance improvements across runtimes (e.g. TensorRT, TensorRT-LLM): speculative decoding, quantization, batching, and KV-cache reuse
- Instrument deep observability (metrics, traces, logs) and build repeatable benchmarks to measure speed, reliability, and quality
- Implement platform fundamentals: API versioning, validation, usage metering, quotas, and authentication
- Collaborate closely with other teams to deliver robust, developer-friendly model serving experiences

## Requirements
- 3+ years experience building and operating distributed systems or large-scale APIs
- Proven track record of owning low-latency, reliable backend services (rate-limiting, auth, quotas, metering, migrations)
- Infra instincts with performance sensibilities: profiling, tracing, capacity planning, and SLO management
- Comfortable debugging complex systems, from runtime internals to GPU execution traces
- Strong written communication; able to produce clear design docs and collaborate across functions

## Nice to Have
- Experience with LLM runtimes (vLLM, SGLang, TensorRT-LLM) or contributions to open-source inference engines (vLLM, TensorRT-LLM, SGLang, TGI)
- Knowledge of Kubernetes, service meshes, API gateways, or distributed scheduling
- Background in developer-facing infrastructure or open-source APIs

## Similar roles

- [Backend Engineer](https://hotfix.jobs/jobs/8c0cdac6-4075-42b1-a874-6e856a68d2ba) - Fathom - AI Notetaker - Remote - $180k – $240k/yr
- [Software Engineer, Knowledge Systems](https://hotfix.jobs/jobs/afb75fbb-e379-420c-99a7-1eb6f7155291) - Exa - San Francisco, CA - $180k – $350k/yr
- [Software Engineer, Control Plane](https://hotfix.jobs/jobs/1f8d4743-d927-4daf-b738-f3122fe354d7) - Hightouch - Remote - $180k – $260k/yr
- [Software Engineer, Streaming Systems](https://hotfix.jobs/jobs/42426ad1-4ce5-451e-bd46-474088d85183) - Hightouch - Remote - $180k – $320k/yr
- [Software Engineer, Customer Studio Backend](https://hotfix.jobs/jobs/c1a85145-b4d5-46dd-9b35-e98f04a3d626) - Hightouch - Remote - $180k – $320k/yr

**Apply:** https://hotfix.jobs/jobs/c6f89d74-c16e-4b9a-aec4-61ef644a51c3
**Canonical:** https://hotfix.jobs/jobs/c6f89d74-c16e-4b9a-aec4-61ef644a51c3