Senior Software Engineer, Model Serving
Designs and builds scalable infrastructure for high-throughput, low-latency AI/ML model serving on CPU/GPU. Requires 5+ years in distributed systems, inference expertise, and strong system design skills.
About the job
The impact you will have:
- Design and implement core systems and APIs that power Databricks Model Serving, ensuring scalability, reliability, and operational excellence.
- Drive architectural decisions and trade-offs to optimize performance, throughput, autoscaling, and operational efficiency for CPU and GPU serving workloads.
- Contribute directly to key components across the serving infrastructure — from model container builds and deployment workflows to runtime systems like routing, caching, observability, and intelligent autoscaling — ensuring smooth and efficient operations at scale.
- Collaborate cross-functionally with product, platform, and research teams to translate customer needs into reliable and performant systems.
- Lead technical initiatives that improve latency, availability, and cost-effectiveness across both customer-facing and foundational serving layers.
- Establish best practices for code quality, testing, and operational readiness, and mentor other engineers through design reviews and technical guidance.
What we look for:
- 5+ years of experience building and operating large-scale distributed systems.
- Experience in model serving, inference systems, or related infrastructure (e.g., routing, scheduling, autoscaling, and observability).
- Strong foundation in algorithms, data structures, and system design as applied to large-scale, low-latency serving systems.
- Proven ability to deliver technically complex, high-impact initiatives that create measurable customer or business value.
- Experience building architecture for large-scale, performance-sensitive CPU/GPU inference systems.
- Strong communication skills and ability to collaborate across teams in fast-moving environments.
- Customer-focused mindset with the ability to align implementation details with product goals.
- Passion for mentoring, growing engineers, and fostering technical excellence.
Skills
Distributed Systems, Model Serving, Inference Systems, System Design, Kubernetes, Autoscaling, Observability, Routing, Caching, GPU
Similar jobs
ML Engineering jobs
Upstart
United States
Build and operate backend infrastructure for machine learning model training, serving, feature management, and marketplace simulation. The role requires 6+ years of software engineering experience, distributed systems expertise, and experience with production ML platforms.
Lightning AI
New York, NYBuild and optimize large language model training and post-training pipelines, improving model quality, distributed performance, evaluation, and production readiness. The role requires deep PyTorch and transformer experience, strong distributed-systems and software-engineering skills, and expertise in modern LLM optimization techniques.
Parallel Systems
Los Angeles, CA
Develop and deploy real-time perception and sensor-fusion software for autonomous battery-electric rail vehicles. The role requires strong robotics, geometry-based computer vision, C/C++ and Rust experience, plus hands-on work with multimodal sensors and production systems.
Beacon Biosignals
United States
The Senior Algorithm Engineer leads development and production deployment of machine and deep learning algorithms for biosignal and medical-device applications. The role requires 5+ years of industry experience, DSP and statistics expertise, PyTorch proficiency, and familiarity with regulated health or similar domains.
LangChain
New York, NY
Leads hands-on development and deployment of production AI agents and workflows, sets technical standards, and mentors engineers. Requires 4+ years of software engineering experience, production LLM application experience, and strong Python or TypeScript skills.