# Software Engineer, LLM Infrastructure

**Company:** [Fireworks AI](https://hotfix.jobs/companies/fireworks-ai)
**Location:** San Mateo, CA
**Role:** ML Engineering
**Salary:** $175k – $220k/yr
**Experience:** 5+ years
**Skills:** Python, Go, PyTorch, MLflow, Kubernetes, vLLM, sglang, trt-llm
**Posted:** 2025-11-05

> Build and maintain scalable backend infrastructure for Fireworks AI's generative AI platform, including LLM CI/CD pipelines, control planes, and model serving systems. Requires 5+ years software engineering experience focused on ML/infrastructure, strong Python/Go skills, and familiarity with PyTorch, Kubernetes, and LLM concepts.

## Job Description

## Key Responsibilities
- Contribute to the design and development of scalable backend infrastructure that supports distributed training, inference, and data pipelines.
- Build and maintain core backend services such as LLM CI/CD pipeline, control plane, and model serving systems.
- Support performance optimization, cost efficiency, and reliability improvements across compute, storage, and networking layers.
- Building frameworks and safeguards to ensure Fireworks AI has the best model quality in the industry.
- Collaborate with performance, training, and product teams to translate research and product needs into infrastructure solutions.
- Participate in code reviews, technical discussions, and continuous integration and deployment processes.

## Minimum Qualifications
- Bachelor’s degree in Computer Science, Engineering, or a related technical field (or equivalent practical experience).
- 3 years of experience in software engineering, with a focus on infrastructure or machine learning systems.
- Strong programming skills in Python, Go, or a similar language.
- Proven experience in ML infrastructure and tooling (e.g., PyTorch, MLflow, Vertex AI, SageMaker, Kubernetes, etc.).
- Basic understanding of LLM knowledge (e.g., context length, disaggregated prefill, KV cache memory estimation, etc).

## Preferred Qualifications
- 5+ years of experience in software engineering, with a focus on infrastructure or machine learning systems.
- Experience with open source inference engine like vLLM, Sglang, or TRT-LLM.
- Contributions to open-source infrastructure or ML projects.
- Experience in building large scale ML/MLOps infrastructure.

## Similar roles

- [Research Engineer, Agentic Systems](https://hotfix.jobs/jobs/278f7939-10a0-4e06-aea5-532256d85fbb) - Mirage - New York, NY - $175k – $275k/yr
- [Software Engineer](https://hotfix.jobs/jobs/bea83f70-e958-4a95-8de7-9846f50f6729) - xAI - Palo Alto, CA - $175k – $275k/yr
- [Software Engineer, ML Products](https://hotfix.jobs/jobs/9fbada47-05ec-4c0c-84dc-31c53d8a0541) - Mirage - New York, NY - $175k – $275k/yr
- [Robotic Software Engineer, Perception](https://hotfix.jobs/jobs/9cb99e86-43d1-4920-8b0f-6352de40420b) - Applied Intuition - Sunnyvale, CA - $175k – $250k/yr
- [Research Engineer](https://hotfix.jobs/jobs/3ac0b434-bec7-4f49-93a3-787ad4af3951) - Hedra - San Francisco, CA - $175k – $275k/yr

**Apply:** https://hotfix.jobs/jobs/0e1de019-2a15-41c7-b558-9612b33f4feb
**Canonical:** https://hotfix.jobs/jobs/0e1de019-2a15-41c7-b558-9612b33f4feb