# Software Engineer - Model Performance

**Company:** [Baseten](https://hotfix.jobs/companies/baseten)
**Location:** San Francisco, CA, New York, NY
**Role:** ML Engineering
**Salary:** $180k – $360k/yr
**Skills:** PyTorch, TensorRT, Tensorrt-Llm, CUDA, Python, C++, vLLM, Sglang, Docker, Kubernetes
**Posted:** 2024-03-28

> Software Engineer optimizes ML model inference performance using techniques like quantization and speculative decoding. Requires backend experience with PyTorch, TensorRT, CUDA, and deep GPU knowledge for LLMs.

## Job Description

## Responsibilities
- Implement, refine, and productionize cutting-edge techniques (quantization, speculative decoding, kv cache reuse, chunked prefill and LoRA) for ML model inference and infrastructure.
- Deep dive into underlying codebases of **TensorRT**, **PyTorch**, **TensorRT-LLM**, **vllm**, **sglang**, **CUDA**, and other libraries to debug ML performance issues.
- Apply and scale optimization techniques across a wide range of ML models, particularly large language models.
- Collaborate with a diverse team to design and implement innovative solutions.
- Own projects from idea to production.

## Requirements
- Bachelor's, Master's, or Ph.D. degree in Computer Science, Engineering, Mathematics, or related field.
- Experience with one or more general-purpose programming languages, such as **Python** or **C++**.
- Familiarity with LLM optimization techniques (e.g., quantization, speculative decoding, continuous batching).
- Strong familiarity with ML libraries, especially **PyTorch**, **TensorRT**, or **TensorRT-LLM**.
- Demonstrated interest and experience in LLMs.
- Deep understanding of GPU architecture.

## Bonus
- Proficiency in enhancing the performance of software systems, particularly in the context of large language models (LLMs).
- Experience with **CUDA** or similar technologies.
- Deep understanding of software engineering principles and a proven track record of developing and deploying AI/ML inference solutions.
- Experience with **Docker** and **Kubernetes**.

## Similar roles

- [Software Engineer, AI Platform](https://hotfix.jobs/jobs/7ddad000-3e1c-47cb-a45b-31d8e72abc4f) - Notion - San Francisco, CA - $180k – $201k/yr
- [Research Engineer, Generalist](https://hotfix.jobs/jobs/1308107a-d911-4d6a-924d-e87d4654d356) - Exa - San Francisco, CA - $180k – $350k/yr
- [Software Engineer - BIS](https://hotfix.jobs/jobs/e9bc193b-be22-4ba7-8730-9d1ea41c31ef) - Baseten - San Francisco, CA - $180k – $360k/yr
- [AI Engineer (Core)](https://hotfix.jobs/jobs/36b94901-fb9d-47d4-a46b-51ef8679ea67) - Build - San Francisco, CA - $180k – $250k/yr
- [Research Engineer](https://hotfix.jobs/jobs/54931695-a06a-4671-b30e-cacd5daf8c92) - Lightning AI - Remote - $180k – $250k/yr

**Apply:** https://hotfix.jobs/jobs/bf6e68af-aa6d-46f9-9e4d-3996a59cd161
**Canonical:** https://hotfix.jobs/jobs/bf6e68af-aa6d-46f9-9e4d-3996a59cd161