Staff + Senior Software Engineer, Inference Deployment

Key Responsibilities

• Design, build, and maintain the distributed systems that serve Claude to millions of users worldwide.
• Develop resilient, flexible systems that adapt in real time to real world events.
• Develop intelligent request routing, load balancing, and traffic management systems across thousands of accelerators.
• Maximize compute efficiency across the fleet by autoscaling and orchestrating production, research, and experimental workloads.
• Build and operate production-grade deployment pipelines for releasing new models to users.
• Provide high-performance inference infrastructure that enables researchers to develop next-generation models.
• Integrate new AI accelerator platforms and support inference for new model architectures.

Minimum Qualifications

• Significant software engineering experience, particularly with distributed systems.
• Results-oriented, with a bias towards flexibility and impact.
• Willingness to pick up slack, even if it goes outside your job description.
• Desire to learn more about machine learning systems and infrastructure.
• Thrive in environments where technical excellence directly drives both business results and research breakthroughs.
• Care about the societal impacts of your work.

Preferred Qualifications

• Experience with high-performance, large-scale distributed systems.
• Experience implementing and deploying machine learning systems at scale.
• Experience with load balancing, request routing, or traffic management systems.
• Familiarity with LLM inference optimization, batching, and caching strategies.
• Experience with Kubernetes and cloud infrastructure (AWS, GCP, Azure).
• Proficiency in Python or Rust.

Representative Projects

• Designing intelligent routing algorithms that optimize request distribution across many accelerators in different environments.
• Autoscaling our compute fleet to dynamically match supply with demand across production, research, and experimental workloads.
• Building production-grade deployment pipelines for releasing new models to millions of users reliably.
• Contributing to new inference features.
• Supporting inference for new model architectures.
• Analyzing observability data to tune performance based on real-world production workloads.
• Managing multi-region deployments and geographic routing for global customers.

Education

• Bachelor’s degree or an equivalent combination of education, training, and/or experience in a field relevant to the role.