# Distributed LLM Inference Engineer

**Company:** [Anyscale](https://hotfix.jobs/companies/anyscale)
**Location:** San Francisco, CA, Palo Alto, CA, California
**Role:** ML Engineering
**Salary:** $170k – $247k/yr
**Skills:** PyTorch, Ray, vLLM, Tensorrt-Llm, Distributed Systems, Ml Inference, CUDA, Triton, Tvm, Mlir, TensorFlow
**Posted:** 2026-05-27

> Build and optimize distributed LLM inference systems at scale using Ray, integrating with engines like vLLM to deliver high-throughput, low-latency batch and online inference solutions.

## Job Description

## Responsibilities
- Iterate quickly with product teams to ship end-to-end solutions for batch and online inference at high scale for Ray users and Anyscale customers
- Work across the stack integrating Ray Data and LLM engines to provide optimizations for low-cost, large-scale ML inference
- Integrate with open-source software like vLLM, work with the community to adopt techniques in Anyscale solutions, and contribute improvements to open source
- Follow state-of-the-art developments in open source and research, implementing and extending best practices

## Requirements
- Familiarity with running ML inference at large scale with high throughput and low latency
- Familiarity with deep learning and deep learning frameworks (e.g., PyTorch)
- Solid understanding of distributed systems and ML inference challenges

## Nice-to-Haves
- ML Systems knowledge
- Experience using Ray
- Work with community on LLM engines like vLLM, TensorRT-LLM
- Contributions to deep learning frameworks (PyTorch, TensorFlow)
- Contributions to deep learning compilers (Triton, TVM, MLIR)
- Prior experience working on GPUs / CUDA

## Compensation & Benefits
- Market-based compensation approach
- Equity (stock options)
- Healthcare plans with 99% premiums covered for employees and dependents
- 401k Retirement Plan
- Education & Wellbeing Stipend
- Paid Parental Leave
- Fertility Benefits
- Paid Time Off
- Commute reimbursement
- 100% of in-office meals covered

## Similar roles

- [Machine Learning Engineer](https://hotfix.jobs/jobs/a3359f08-ea84-4500-b684-678e1b789991) - Airbnb - San Francisco, CA - $170k – $180k/yr
- [AI Platform Engineer](https://hotfix.jobs/jobs/e5a08a02-c346-4778-8598-c6f2b3ff1189) - Notable - San Mateo, CA - $170k – $205k/yr
- [Autonomy Engineer - ML & DL Infrastructure](https://hotfix.jobs/jobs/64e727de-a200-4cdd-81c1-9ed94150308b) - Skydio - San Mateo, CA - $170k – $278k/yr
- [Research Engineer](https://hotfix.jobs/jobs/b91e0912-d75f-418f-97a6-1b0a51108f23) - Forus - New York, NY - $170k – $300k/yr
- [AI Engineer, Quality (Evals)](https://hotfix.jobs/jobs/54a01151-d1f8-4597-be26-69ae94139fb2) - Fieldguide - Remote - $170k – $220k/yr

**Apply:** https://hotfix.jobs/jobs/d33066df-7725-4fc7-9306-c2a1eacc4ef1
**Canonical:** https://hotfix.jobs/jobs/d33066df-7725-4fc7-9306-c2a1eacc4ef1