# Engineering Manager, Model Inference

**Company:** [Abridge](https://hotfix.jobs/companies/abridge)
**Location:** San Francisco, CA
**Role:** Engineering Management
**Salary:** $220k – $270k/yr
**Experience:** 5+ years
**Skills:** PyTorch, TensorRT, vLLM, TensorFlow, Llm Architecture, Batching, Quantization, Kernel Fusion, Flashattention, Gpu Performance Analysis, Tensor Parallelism, Pipeline Parallelism, Expert Parallelism, Kubernetes, GCP
**Posted:** 2026-05-20

> Engineering Manager leading the Model Inference team, responsible for architecting and scaling low-latency, high-throughput LLM serving infrastructure and growing a team of AI inference engineers.

## Job Description

## What You’ll Do
- Lead and grow a high-performing team of AI inference engineers focused on building and scaling infrastructure for Abridge’s products and APIs
- Own the technical direction of our inference systems—making key decisions around batching, throughput, latency, and GPU utilization
- Architect and scale inference infrastructure for reliability, efficiency, and observability; lead incident response
- Benchmark and eliminate bottlenecks throughout the inference stack
- Partner with ML Research teams on model optimization, quantization, and deployment
- Develop APIs for AI inference used by both internal teams and external customers
- Recruit, mentor, and develop engineering talent; establish team processes, engineering standards, and operational excellence
- Work closely with the GenAI Platform, Data, and Product teams to plan and execute projects that directly impact clinicians and patients

## What You’ll Bring
- 5+ years of engineering experience with 1+ years in a technical leadership or management role
- Deep, hands-on experience with ML systems and inference frameworks (e.g., PyTorch, TensorRT, vLLM, TensorFlow)
- Strong understanding of LLM architecture (e.g. Multi-Head Attention, Multi/Grouped-Query Attention, and common transformer components)
- Experience with inference optimizations (e.g. batching, quantization, kernel fusion, FlashAttention)
- Familiarity with GPU characteristics, roofline models, and performance analysis
- Experience deploying reliable, distributed, real-time systems at scale
- Experience with parallelism strategies: tensor parallelism, pipeline parallelism, expert parallelism
- Skilled at hiring and mentorship, with a demonstrated track record of helping engineers grow their skills and careers
- Strong technical communication and cross-functional collaboration skills
- Comfortable giving constructive feedback on technical designs and code reviews
- Has thrived in a fast-growing startup and knows how to operate with urgency and focus

## Added Bonus
- Background in training infrastructure and RL workloads
- Skilled in building secure, compliant systems on major cloud platforms (GCP preferred, AWS experience welcome)
- Experience with Kubernetes and container orchestration at scale
- Published work or contributions to inference optimization research

## Similar roles

- [Engineering Manager, Billing Platform](https://hotfix.jobs/jobs/37b04730-f405-435b-aa44-059c3a03c7a8) - Sentry - San Francisco, CA - $220k – $280k/yr
- [Verification and Validation Manager - Autonomy Trucking](https://hotfix.jobs/jobs/ca810fb3-0ce1-46c7-a294-5e5a235d544c) - Applied Intuition - Sunnyvale, CA - $220k – $320k/yr
- [Engineering Manager, Developer Infrastructure](https://hotfix.jobs/jobs/0e6d7993-8795-4760-ace5-d6eec393cba6) - Sentry - San Francisco, CA - $220k – $300k/yr
- [Engineering Manager - DevEx](https://hotfix.jobs/jobs/fec7ba70-ee31-4c09-902d-45dfb1b2dad9) - Phantom - Remote - $220k – $250k/yr
- [Engineering Manager](https://hotfix.jobs/jobs/2d8a0208-116a-4882-9762-3552a5127b4e) - Adaptive - New York, NY - $220k – $270k/yr

**Apply:** https://hotfix.jobs/jobs/f4ba9c4f-a699-45d6-8d4f-08bffa47a291
**Canonical:** https://hotfix.jobs/jobs/f4ba9c4f-a699-45d6-8d4f-08bffa47a291