Software Engineer, LLM Infrastructure

Key Responsibilities

• Contribute to the design and development of scalable backend infrastructure that supports distributed training, inference, and data pipelines.
• Build and maintain core backend services such as LLM CI/CD pipeline, control plane, and model serving systems.
• Support performance optimization, cost efficiency, and reliability improvements across compute, storage, and networking layers.
• Building frameworks and safeguards to ensure Fireworks AI has the best model quality in the industry.
• Collaborate with performance, training, and product teams to translate research and product needs into infrastructure solutions.
• Participate in code reviews, technical discussions, and continuous integration and deployment processes.

Minimum Qualifications

• Bachelor’s degree in Computer Science, Engineering, or a related technical field (or equivalent practical experience).
• 3 years of experience in software engineering, with a focus on infrastructure or machine learning systems.
• Strong programming skills in Python, Go, or a similar language.
• Proven experience in ML infrastructure and tooling (e.g., PyTorch, MLflow, Vertex AI, SageMaker, Kubernetes, etc.).
• Basic understanding of LLM knowledge (e.g., context length, disaggregated prefill, KV cache memory estimation, etc).

Preferred Qualifications

• 5+ years of experience in software engineering, with a focus on infrastructure or machine learning systems.
• Experience with open source inference engine like vLLM, Sglang, or TRT-LLM.
• Contributions to open-source infrastructure or ML projects.
• Experience in building large scale ML/MLOps infrastructure.